Data Platform Lead

Hiring: Data Platform Lead at Santa Clara, CA 5 Days Onsite

Mandatory Skills: Databricks | Data Engineering and Analysis | Cloud Security | Governance | Encryption | Multi-Tenant Platforms

Function: Data Platforms / Advanced Analytics

Role Type: Technical Lead / Solution Lead

Primary Platform: Databricks Lakehouse on cloud

Scope: EPIC Data Platform, dedicated tenant and multi-tenant capabilities

Role Purpose

The EPIC Data Platform Lead will own the technical direction and implementation leadership for a secure, governed, scalable cloud data platform built on Databricks. The role combines hands-on data engineering and analytical problem solving with architecture leadership across tenant isolation, data governance, identity and access, encryption, observability, production readiness, and platform operations. The lead will translate business and engineering requirements into implementable platform capabilities for internal, customer-dedicated, and controlled multi-tenant use cases.

Expected Outcomes:

Trusted data products: Curated, traceable, analytics-ready data with clear ownership and quality controls.

Secure tenant boundaries: Validated isolation across workspace, catalog, storage, identity, network, jobs, APIs, and exports

Production-grade operations: Observable, supportable, cost-aware services with automated deployment and evidence-based controls.

Key Responsibilities

Platform Architecture and Technical Leadership: Define target architecture, engineering standards, roadmaps, decision records, reusable patterns, and non-functional requirements for EPIC Databricks environments. Lead design reviews and make trade-offs across performance, security, operability, scalability, and cost.

Databricks Implementation: Lead workspace, Unity Catalog, Delta Lake, pipeline, workflow, SQL warehouse, compute policy, external location, storage credential, and deployment-pattern implementation. Establish maintainable medallion-layer processing and production engineering practices.

Data Engineering and Analysis: Design and review batch, streaming, and event-driven ingestion; transformation and source-to-target logic; reconciliation; data profiling; exploratory analysis; root-cause analysis; and analytical data products. Use data to validate latency, completeness, linking, accuracy, and business-rule outcomes.

Security by Design: Partner with cybersecurity, IAM, cloud, network, and application teams to implement least privilege, SSO/federation, service principals, secrets management, private connectivity, controlled egress, hardening, vulnerability remediation, and auditable access.

Governance and Data Protection: Implement data classification, taxonomy, ownership, metadata, lineage, retention, access reviews, fine-grained permissions, row filters, column masks, controlled sharing, DLP-aligned controls, and evidence-driven compliance.

Encryption and Key Management: Design and implement encryption in transit and at rest, customer-managed keys and BYOK patterns where required, KMS/HSM integration, key scope and separation, rotation, revocation, monitoring, recovery, and control validation.

Dedicated and Multi-Tenant Delivery: Define tenant onboarding, registry, provisioning, configuration, isolation, routing, metering, offboarding, and migration patterns. Prevent unauthorized cross-tenant access and validate isolation through automated negative testing and periodic control reviews.

Observability and Operations: Implement end-to-end logging, auditability, lineage, data-quality monitoring, health dashboards, alerting, SIEM integration, incident response, runbooks, service-level measures, capacity planning, and cost showback.

Delivery Leadership: Own backlog quality, milestones, dependencies, risk mitigation, release readiness, production cutover, operational handoff, and stakeholder communication. Mentor engineers and coordinate delivery across data, cloud, security, governance, QA, infrastructure, and application teams.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or a related field, or equivalent practical experience.
• Strong experience leading the design and implementation of enterprise cloud data platforms, with substantial hands-on Databricks experience.
• Strong working knowledge of Apache Spark, Delta Lake, Databricks Workflows, Unity Catalog, SQL, and Python. Scala experience is beneficial.
• Demonstrated ability to perform complex data analysis, profiling, reconciliation, debugging, performance analysis, and root-cause investigation using large datasets.
• Experience implementing production-grade batch, streaming, micro-batch, event, and file-based ingestion patterns, including schema evolution, replay, backfill, and idempotent processing.
• Experience with cloud-native data services, object storage, IAM, private networking, key management, logging, monitoring, infrastructure as code, and CI/CD. AWS experience is preferred; Azure or Google Cloud Platform experience is also relevant.
• Strong knowledge of data security and governance concepts, including least privilege, identity federation, service principals, classification, lineage, retention, masking, audit logging, DLP, controlled data sharing, and regulated or customer-sensitive data handling.
• Experience designing or operating dedicated-tenant, multi-tenant, or customer-isolated platforms, including tenant lifecycle, logical and physical isolation, resource governance, and cross-tenant security testing.
• Experience implementing encryption in transit and at rest, cloud KMS or HSM integrations, customer-managed keys, BYOK, key rotation, separation of duties, and cryptographic control evidence.
• Strong architecture, technical documentation, stakeholder management, and engineering leadership skills, with the ability to convert ambiguous requirements into executable designs and delivery plans.

Preferred Qualifications

• Experience with Databricks on AWS, including S3, KMS, PrivateLink, VPC endpoints, IAM roles, CloudTrail, CloudWatch, and enterprise network controls.
• Experience with Kafka or equivalent event-streaming platforms and cloud edge or integration services.
• Experience with Databricks Asset Bundles, Terraform, Git-based workflows, automated testing, release pipelines, policy as code, and environment promotion.
• Experience building data-quality frameworks, lineage, observability, operational dashboards, and cost or usage reporting by tenant.
• Experience with Delta Sharing, APIs, BI tools, MLflow, AI/ML workloads, or governed data-product consumption patterns.
• Knowledge of Zero Trust, NIST CSF, CIS Controls, security architecture reviews, threat modeling, penetration testing, exception management, and audit evidence practices.
• Experience in semiconductor manufacturing, R&D, lab, metrology, equipment telemetry, or OT-integrated data environments is an advantage.
• Relevant Databricks, cloud architecture, data engineering, security, or governance certifications are desirable.

Leadership Behaviors

• Customer and Data Protection Mindset: Treats tenant isolation, confidentiality, and evidence as design requirements, not afterthoughts.
• Outcome Ownership: Drives work from architecture and backlog through production validation and operational handoff.
• Analytical Rigor: Challenges assumptions, quantifies gaps, validates results, and communicates findings with traceable evidence.
• Pragmatic Architecture: Balances enterprise standards with delivery speed, maintainability, scalability, and supportability.
• Cross-functional Influence: Aligns engineering, security, governance, infrastructure, product, and business stakeholders without losing technical depth.
• Engineering Excellence: Promotes reusable components, automation, documentation, testing, peer review, and continuous improvement.

Measures of Success

• Architectural decisions, security controls, and governance requirements are translated into an executable and traceable engineering backlog.
• Databricks pipelines and analytical data products meet agreed quality, latency, reliability, security, and support readiness criteria.
• Dedicated and multi-tenant environments demonstrate validated isolation across data, identity, compute, storage, network, and operational boundaries.
• Encryption, key-management, access-governance, and audit controls are implemented with reviewable evidence and defined operational ownership.
• Platform releases include automated testing, observability, rollback, runbooks, production support, and stakeholder-ready status reporting.
• The engineering team becomes more effective through technical guidance, reusable patterns, stronger documentation, and disciplined delivery practices.