Lead Data Engineer

Protective helps protect customers against life’s uncertainties and provide protection and peace of mind. The company is seeking a Lead Data Engineer to set the technical direction for a delivery pod building data products on its Databricks lakehouse on Azure, with responsibility for design, modeling, data contracts, quality, operations, and technical leadership.

Responsibilities:

• Lead the design of data products end to end: what gets ingested, how it is cleansed and conformed, how it is modeled, and what the Gold layer looks like to the people querying it
• Own dimensional design — grain, natural and surrogate keys, Type 2 history, facts, bridges, and conformed dimensions shared across the pod’s products
• Set the pod’s position on where logic belongs: what is cleaned in Silver, what is business logic in Gold, and what is a consumer’s own concern
• Keep models as simple as the questions require, and push back on designs that will not hold
• Partner with ML engineering where the pod’s Gold layer is the training or feature source for a model, so those datasets are contracted, versioned, and reproducible like any other consumer-facing product
• Own the pod’s ODCS data contracts as real interfaces: named owners, named consumers, enforceable quality rules, freshness and update expectations, and an explicit breaking-change policy
• Make the compatibility call on every proposed contract change, and drive consumer notification when a change is genuinely breaking
• Represent the pod’s contracts in cross-domain conversations, where one pod’s Gold layer is another team’s dependency
• Set and hold the pod’s engineering standards for Python, SQL, dbt, testing, model structure, naming, and repository conventions — consistent with the platform’s paved paths and Azure DevOps CI gates rather than in competition with them
• Lead code review
• Be the reviewer who catches the modeling mistake, the missing test, and the change that will break a consumer, and who explains why so the pod learns it
• Ensure quality rules are enforced in tests and asset checks rather than asserted in documentation, and that pipeline health is observable without someone going to look — freshness, volume, latency, and cost instrumented, and alerting set against the SLAs and SLOs the pod’s contracts commit to
• Own the pod’s operational posture for its own pipelines: failure diagnosis, data-issue triage, backfills, cost and performance tuning, on-call coverage and escalation, root-cause analysis, and runbooks someone other than the author can execute
• Keep the pod’s delivery inside the control expectations of a regulated carrier: change management through pull request and pipeline, segregation of duties between authoring and deploying, least-privilege access, and audit evidence that falls out of CI/CD rather than being assembled for an auditor
• Develop reusable frameworks, templates, and patterns that raise the pod’s consistency and delivery speed
• Partner with the Product Owner and Scrum Master on decomposition and refinement: turn use cases and features into estimable, grounded stories with testable acceptance criteria and an identified target layer and repository
• Hold the Definition of Ready before the pod commits and the Definition of Done before the pod calls something finished — merged and approved code, passing CI and coverage gates, and evidence that the outcome is real
• Identify unknowns that need a spike rather than an estimate, and say so during planning rather than mid-sprint
• Grow the engineers on the pod through design review, pairing, and code review rather than by taking the hard work yourself
• Bring new engineers up to productive speed on the platform’s conventions and tooling, and reduce single points of knowledge — no data product that only one person understands
• Work with the platform team on capability gaps: when the pod needs something the platform does not yet offer, raise it as a demand signal rather than building a private workaround
• Partner with the DataOps/MLOps Lead on the shared CI/CD, orchestration, and observability standards — adopt and strengthen the paved path rather than forking it, and be the pod’s voice on what it is still missing
• Work with data architecture and governance on solution shape, Unity Catalog placement, and access requirements

Requirements:

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field; equivalent practical experience considered
• 6+ years building and operating production data pipelines and consumer-facing data models, from source ingestion through to published data products
• Strong hands-on Python and SQL, with the credibility to make design calls and the willingness to still write and review code
• Hands-on experience with Databricks or a comparable Spark-based lakehouse, including Delta Lake, MERGE, incremental processing, and performance tuning
• Deep dimensional modeling experience — grain, keys, slowly changing dimensions, facts and dimensions, conformed dimensions
• Demonstrated technical leadership: setting standards, leading design, and raising other engineers' work, whether or not the role carried a lead title
• Experience owning data that other teams depend on, including handling breaking changes and production data incidents
• Experience with orchestration (Dagster, Databricks Workflows, Airflow, or similar), Git-based collaborative development, code review, and CI/CD — Azure DevOps or comparable
• Experience setting observability and SLA/SLO expectations for data other teams depend on, and running the incident and communication path when they are missed
• Ability to explain trade-offs clearly to engineers, product owners, and business stakeholders, and to say no to a design that will not hold
• Databricks certification (Data Engineer Professional or equivalent demonstrated depth)
• Unity Catalog at multi-team scale: catalogs, schemas, external locations, permissions, and lineage
• dbt at scale on Databricks, and Python-based modeling frameworks over Delta Lake
• Dagster and Dagster Cloud, including assets, asset checks, and branch deployments
• Practical experience with data contracts, ODCS, or data-mesh style data product ownership
• Experience with a declarative Python ingestion framework such as dlt (dltHub) or comparable
• Data quality and observability tooling such as Great Expectations, Monte Carlo, or similar
• Familiarity with MLOps practice — MLflow, model registries, and model serving — sufficient to design data products that ML systems can depend on
• Azure and Azure DevOps
• Financial services, insurance, or another regulated industry, including data access, lineage, and audit expectations
• Experience introducing AI-assisted development into a team's normal workflow in a disciplined way