Principal Data Platform Engineer

Role Overview

• Architect and implement centralized data platform on Databricks.
• Establish governance patterns using Unity Catalog.
• Optimize for cost and performance at scale.
• Enable Data Engineers to build confidently on the platform.
• Lead the architecture for migrating multi-terabyte datasets from legacy systems to Databricks.
• Design Unity Catalog structures enabling secure data separation between product lines.
• Build infrastructure that scales efficiently—through intelligent caching, query optimization, and compute management strategies.
• Establish monitoring, alerting, and data quality validation to ensure platform reliability.

Requirements

• Databricks Expertise (Required)
• Unity Catalog: Production experience with multi-catalog governance, metastore design, and lineage tracking.
• Data Structuring: Experience designing and building unified schemas across multiple disparate product lines.
• Delta Lake: Expert-level experience with Z-ordering, compaction, liquid clustering, and performance tuning at multi-TB scale.
• Delta Live Tables: Strong hands-on experience building declarative ETL pipelines, including change data capture and expectations/constraints.
• Databricks Workflows: Experience with job orchestration, scheduling, and operational monitoring.
• Business Intelligence: Experience enabling company-wide analytics and reporting with modern business intelligence tools and maintaining source of truth data and metrics.
• PySpark & Databricks SQL: Strong proficiency for code review, performance tuning, and query optimization.
• Core Platform Engineering: 5-8 years in data engineering or data platform roles, with 3+ years hands-on Databricks experience.
• Track record leading at least one significant platform build or migration project.
• AWS experience (S3, IAM, VPC) with ability to collaborate on infrastructure decisions.
• Infrastructure-as-code experience (Terraform preferred).
• Technical Leadership: Demonstrated ability architecting data platforms from first principles and defending technical decisions.
• Strong written and verbal communication—document architecture decisions and present to both technical and business stakeholders.
• Preferred But Not Required: Experience with financial data, accounting systems (NetSuite), or enterprise ERP platforms.
• Background building platforms that serve AI/ML workloads (experience preparing data for downstream ML consumption, RAG and retrieval, and LLMs).
• Understand advanced intelligence concepts such as relationship surfacing with knowledge graphs.
• Familiarity with data governance frameworks and compliance requirements for regulated industries.

Tech Stack

• AWS
• ERP
• ETL
• PySpark
• SQL
• Terraform
• Unity

Benefits

• A collaborative team culture with opportunities for career development.
• Ample opportunities to be recognized, build valuable skills, and grow your career.
• Generous vacation policy, including paid parental leave.
• Comprehensive health plans with FSA and HSA options.
• 401(k) retirement plan.
• Life and disability insurance coverage.
• Supplemental benefits like a dependent care savings plan, pet insurance, will preparation, and an employee assistance program.