Dir Data Engineering (Databrick/ADO/Heathcare)

Dice is a recruiting platform that is seeking a Director of Data Engineering & Platform to lead its enterprise data engineering organization. The role oversees approximately 10 data engineers, Databricks platform operations, Azure DevOps delivery processes, Agile/Scrum execution, scalable data pipelines, platform reliability, and security and compliance for healthcare data.

Responsibilities:

• Lead, develop, and manage a team of approximately 10 data engineers responsible for building and operating the Company’s enterprise data platform
• Establish a strong engineering culture focused on quality, accountability, scalability, automation, and predictable delivery
• Set engineering standards for data pipelines, transformations, testing, deployment, monitoring, documentation, and production support
• Provide technical guidance and mentorship to engineers while developing future technical and team leaders
• Maintain sufficient technical depth to participate in design reviews, challenge engineering approaches, and help resolve complex technical issues
• Establish clear performance expectations and continually improve the productivity and capabilities of the engineering organization
• Serve as the primary engineering leader for the Company’s Databricks ecosystem
• Lead engineering and platform execution across multiple Databricks environments and workspaces
• Establish consistent standards for development, testing, promotion, security, monitoring, and production operations across environments
• Drive platform performance, scalability, reliability, observability, and cost optimization
• Partner with Data Architecture on the implementation and evolution of the Company’s lakehouse architecture
• Establish reusable frameworks and engineering patterns leveraging Databricks, Spark, Delta Lake, Python, and SQL
• Support adoption and optimization of capabilities including Unity Catalog, Delta Lake, Databricks Workflows, MLflow, and broader Databricks tooling
• Own the engineering team's Agile/Scrum delivery process
• Serve as a strong Scrum leader, driving sprint planning, backlog refinement, daily stand-ups, retrospectives, capacity planning, and delivery commitments
• Establish clear sprint goals, engineering priorities, and measurable delivery expectations
• Partner with Data Products to translate product priorities and business requirements into executable engineering work
• Manage engineering capacity across platform initiatives, new development, technical debt, production support, and strategic projects
• Identify and remove blockers that impact engineering velocity
• Create visibility into engineering progress, dependencies, risks, and delivery commitments
• Drive continuous improvement through sprint metrics, retrospectives, and engineering performance data
• Own and mature the engineering team's use of Azure DevOps (ADO) for planning, execution, and delivery management
• Establish standards for Epics, Features, User Stories, Tasks, Bugs, and engineering backlogs
• Ensure engineering work is properly defined, estimated, prioritized, assigned, and tracked
• Build visibility into sprint velocity, capacity, backlog health, delivery risks, and dependencies
• Leverage ADO to improve transparency and communication across Engineering, Data Products, Data Architecture, and executive leadership
• Partner with engineering teams on CI/CD and automated deployment practices
• Lead the design, development, and operation of scalable pipelines ingesting and transforming clinical, financial, operational, and enterprise data
• Establish modern engineering standards for ETL/ELT, batch and near-real-time processing, orchestration, testing, observability, and code promotion
• Ensure pipelines meet established standards for performance, reliability, data quality, and availability
• Build reusable engineering frameworks that accelerate the integration of new data sources
• Ensure engineering capabilities can scale to support the Company’s continued organizational growth and acquisition activity
• Establish production support and incident-management practices for critical data pipelines and platform services
• Partner closely with the Director of Data Products to translate business and clinical priorities into an executable engineering roadmap
• Partner closely with the Director of Data Architecture to implement the Company’s enterprise data architecture, governance, modeling, security, and data quality standards
• Provide the VP with clear visibility into engineering capacity, platform health, delivery performance, technical risks, resource needs, and major dependencies
• Establish clear SLAs and operational standards for critical data pipelines and platform services
• Implement monitoring, alerting, observability, and incident-response practices
• Drive automated testing and data validation throughout the engineering lifecycle
• Establish CI/CD and disciplined code-promotion processes across Databricks environments
• Ensure appropriate source control, peer review, documentation, and release-management practices
• Continuously identify opportunities to improve platform performance and engineering productivity
• Balance new feature delivery with technical debt, platform maintenance, and reliability
• Partner with Data Architecture, Security, and Governance teams to ensure engineering solutions meet enterprise security and data governance standards
• Ensure appropriate access controls, auditability, encryption, and secure development practices across Databricks environments
• Ensure data engineering practices support HIPAA and applicable healthcare regulatory requirements
• Implement data lineage, quality, classification, and governance standards within engineering workflows

Requirements:

• 8+ years of progressive data engineering experience , including significant experience building enterprise-scale data platforms
• 3+ years of engineering leadership experience , ideally managing teams of 8–15+ engineers
• Deep hands-on Databricks experience is required , including experience operating complex production environments
• Demonstrated experience working across multiple Databricks environments/workspaces
• Strong expertise with Python, SQL, PySpark/Spark, and Delta Lake
• Strong experience building production-grade ETL/ELT pipelines and modern lakehouse architectures
• Experience with Databricks Workflows and/or orchestration technologies such as Azure Data Factory or Airflow
• Strong understanding of CI/CD, automated testing, observability, source control, and modern engineering practices
• Heavy Azure DevOps (ADO) experience , including backlog management, sprint planning, work-item management, capacity planning, and delivery reporting
• Demonstrated experience leading Agile/Scrum engineering teams and driving disciplined sprint execution
• Strong experience managing technical teams while remaining meaningfully engaged with architecture and engineering decisions
• Experience operating in Azure or another major cloud environment
• Deep, recent, production-level Databricks experience
• Proven success leading and developing a data engineering team of comparable scale
• Experience owning engineering execution across multiple environments and complex enterprise data ecosystems
• Heavy hands-on ADO experience managing engineering backlogs, sprints, capacity, dependencies, and delivery
• Strong Scrum leadership with the ability to create structure, accountability, and predictable engineering delivery
• Strong Python, SQL, Spark/PySpark, pipeline, orchestration, CI/CD, and modern data engineering knowledge
• Ability to translate architecture and product priorities into scalable, production-ready engineering solutions
• Healthcare or other highly regulated industry experience
• Experience working with clinical, EHR, claims, financial, or healthcare operational data
• Experience with Unity Catalog and enterprise Databricks governance
• Experience with Azure Data Factory, Azure Data Lake, Azure Functions, dbt, and related Azure data technologies
• Experience supporting analytics, data science, machine learning, or AI workloads
• Experience supporting enterprise growth, acquisitions, or integration of new business units and data sources