Architect and implement centralized data platform on Databricks.
Establish governance patterns using Unity Catalog.
Optimize for cost and performance at scale.
Enable Data Engineers to build confidently on the platform.
Lead the architecture for migrating multi-terabyte datasets from legacy systems to Databricks.
Design Unity Catalog structures enabling secure data separation between product lines.
Build infrastructure that scales efficiently—through intelligent caching, query optimization, and compute management strategies.
Establish monitoring, alerting, and data quality validation to ensure platform reliability.
Requirements
Databricks Expertise (Required)
Unity Catalog: Production experience with multi-catalog governance, metastore design, and lineage tracking.
Data Structuring: Experience designing and building unified schemas across multiple disparate product lines.
Delta Lake: Expert-level experience with Z-ordering, compaction, liquid clustering, and performance tuning at multi-TB scale.
Delta Live Tables: Strong hands-on experience building declarative ETL pipelines, including change data capture and expectations/constraints.
Databricks Workflows: Experience with job orchestration, scheduling, and operational monitoring.
Business Intelligence: Experience enabling company-wide analytics and reporting with modern business intelligence tools and maintaining source of truth data and metrics.
PySpark & Databricks SQL: Strong proficiency for code review, performance tuning, and query optimization.
Core Platform Engineering: 5-8 years in data engineering or data platform roles, with 3+ years hands-on Databricks experience.
Track record leading at least one significant platform build or migration project.
AWS experience (S3, IAM, VPC) with ability to collaborate on infrastructure decisions.