Director of Technology, Enterprise Data Engineering

EMPLOYERS is hiring a Director of Technology, Enterprise Data Engineering to lead its data engineering team and modernize its enterprise data platform on Databricks. The role sets technical direction and engineering standards, maintains reliable data pipelines, leads platform migration and governance, and enables distributed teams to build and operate pipelines safely, including through responsible AI adoption.

Responsibilities:

• Builds, coaches, and develops a team of data engineering professionals, including performance reviews, workload balancing, and support for career growth
• Leads hiring and staffing decisions to ensure the team has the skills needed to run the business, execute the Databricks migration, support the platform long-term, and deliver new product data
• Manages a mix of employees, contractors, and vendor-supplied holding all of them to consistent standards and accountability
• Coaches and holds accountable engineers embedded in cross-functional domain pods, even where those engineers do not report directly into this role
• Builds redundancy into critical skills and keeps knowledge, runbooks, and design decisions documented so continuity does not depend on individual availability
• Partners with Enterprise Architecture (Data Architect) to translate architectural direction into built pipelines, data models, and platform design decisions
• Partners with Analytics Operations, Enterprise Data, and Business Intelligence/Strategy to ensure engineering output is structured to be consumed, certified, and trusted for all downstream needs without rework
• Drives strategic planning, sequencing, and trade-off decisions for the data modeling of all initiatives and workstreams, applying the EMPLOYERS Delivery Life Cycle (DLC) methodology to deliver on time and within budget
• Serves as the internal technical counterpart to the System Integration Partners and the Databricks advisory engagement, absorbing, contributing to, and documenting architectural and design decisions rather than deferring them externally
• Maintains regular communication with Director peers, Digital & Data VPs, department heads, and stakeholders, translating complex technical topics into clear, business-friendly terms and flagging risks early
• Enables autonomous development teams to build and operate Databricks pipelines thru self-service platform tooling, templates, and reference patterns, shifting from centralized delivery toward a “paved road” model as EIG moves toward a new operating model
• Owns data ingestion and the build of medallion architecture standards (bronze/silver/gold) across all domains
• Builds and owns the semantic data layer (Unity Catalog objects, curated data sets) — partnering with Enterprise Data on business definitions and Analytics Operations on governance alignment — so that analytics, governance, and AI-assisted query tools all consume one agreed version of each measure
• Collaborates with Enterprise Architecture on data modeling standards (naming, keys, SCD handling, domain boundaries) applied consistently across all pipeline builds
• Sequences and delivers the domain-by-domain migration from legacy Data Lake/CDS and Curated Data Store to Databricks, resolving data and process gaps between development, QA, and production environments
• Sets the team's engineering standards to ensure reproducible, version-controlled pipeline development promoted through governed environments rather than built directly in production
• Ensures core data pipelines and platform infrastructure remain available, reliable, and performant, including production support that minimizes downtime
• Collaborates on the platform’s paved road — reusable pipeline templates, scaffolding, and AI-assisted code-generation tooling — so teams can stand up compliant, production-ready pipelines with minimal central hand-holding
• Leads troubleshooting of complex pipeline and data quality issues alongside technical staff, restoring functionality quickly and with minimal disruption
• Guides technical design sessions to ensure data architecture is secure, scalable, and well-founded
• Researches, evaluates, and pilots emerging platform capabilities (e.g., Lakebase, Agent Bricks, MLflow) as they mature, recommending adoption where it aligns with strategic objectives
• Maintains and enforces data engineering standards, procedures, and documentation (architecture decisions, migration runbooks, support procedures), keeping them accurate and current
• Uses AI tools (e.g., Claude, Copilot, ChatGPT, Gemini) in daily work to improve efficiency, quality, and decision-making
• Validates the accuracy and appropriateness of AI-generated outputs produced by the team before they reach stakeholders and is accountable for the quality of the team's output
• Provides human oversight of AI agents and copilots operating within team workflows, intervening when outcomes fall outside expected bounds
• Coaches team members on where and how to apply AI tools, promoting adoption of AI-driven best practices across the department
• Champions AI and automation use cases within the team's domain (e.g., pipeline development, testing, documentation) and measures their impact on productivity, quality, and cost
• Sets standards for AI-driven, agentic development practices (e.g., spec-driven development, autonomous coding agents, automated code review and testing) so that small, distributed teams can safely multiply their output using AI
• Structures engineering output — tagging, lineage, documentation — to meet data governance and data quality certification requirements owned by Enterprise Data
• Partners with Information Security to ensure pipelines, infrastructure, and AI tool usage comply with company policy and regulatory requirements (e.g., state insurance regulations, NAIC guidelines, data privacy standards)
• Supports audits and remediates vulnerabilities in collaboration with technical staff and Information Security
• Establishes lightweight AI governance practices for the team, including acceptable use and responsible handling of sensitive data in AI tools
• Serves as the team's escalation point for AI-related incidents (bad output, data exposure, agent misfire)
• Partners across the organization on data classification, access, and platform consumption cost as the migration scales
• Defines and monitors KPIs and SLAs for data pipelines and platform services, using performance data to drive accountability for vendors and team members alike
• Leverages automated guardrails (policy-as-code, access controls, CI/CD gating) that let autonomous teams build and ship independently while remaining compliant with governance, security, and regulatory requirements
• Defines oversight and escalation practices for AI coding agents operating with delivery authority within teams, including audit trails for AI-generated changes
• Stays current on data platform capabilities, medallion architecture patterns, and AI-assisted engineering tooling
• Measures the effectiveness of AI adoption within the team and adjusts practices to improve outcomes
• Takes an active role in cross-departmental projects to drive continuous improvement
• Other duties as assigned

Requirements:

• 10+ years of experience in data engineering or a related specialized technology area, including 3+ years leading a team
• Experience leading a legacy-to-cloud data platform migration
• Experience managing vendor SaaS, as well as home grown system data integrations
• Bachelor's degree in Computer Science, Information Systems, or related field, or equivalent experience
• Deep hands-on expertise with Databricks (Unity Catalog, medallion architecture, Delta Lake, DABs/YAML)
• Strong data modeling background (dimensional modeling, semantic layers, domain-driven design), including expert-level SQL/PySpark against large, governed datasets
• Experience building semantic layers or certified data models that support self-service analytics or AI-assisted natural language query tools
• Demonstrated ability to translate ambiguous architectural direction into a built, production-ready data platform
• Working knowledge of CI/CD practices as they apply to data pipelines, partnering with the platform and CI/CD teams rather than owning those functions directly
• Experience partnering with Information Security, including knowledge of security requirements, data privacy standards, and regulatory considerations (e.g., NAIC guidelines, state insurance regulations)
• Knowledge of agile methodologies, product development frameworks, project management tools, and resource planning
• Experience managing SI/vendor relationships and absorbing design ownership from external partners
• High-level proficiency with MS Office, the Atlassian suite, and Lucid Charts
• Experience designing internal developer platforms or self-service infrastructure that lets distributed/autonomous teams build independently
• Working knowledge of data quality frameworks, automated testing, and reconciliation for analytical data, and with managing cloud platform consumption against a committed spend
• Experience with AI-driven, agentic software development practices (e.g., spec-driven development, AI coding agents, automated code review and testing pipelines)
• Practical experience using AI tools (e.g., Claude, Copilot, ChatGPT, Gemini) to support daily work, with the ability to identify use cases and quantify impact
• Working knowledge of AI output validation and quality assurance practices
• Basic understanding of responsible AI / AI governance principles
• Ability to coach others on effective, responsible AI tool adoption
• Experience designing or governing agentic AI development workflows and assessing their impact on team velocity, quality, and risk
• Strong analytical skills, with experience analyzing business processes, workflows, and data to solve complex problems and translate insights into action
• Excellent communication and interpersonal skills, with the ability to influence cross-functional teams and senior stakeholders
• Experience holding resources and stakeholders accountable to quality standards and timely deliverables, and willingness to challenge existing approaches and raise concerns early with evidence
• Bachelor's Degree in Computer Science, Information Systems, or other related field or equivalent work experience
• only open to candidates currently located in the United States and able to work without sponsorship
• Insurance industry experience
• Lakebase experience
• Experience with IBM DataStage, PostgreSQL, and/or Oracle source systems
• Familiarity with AWS-based data platforms
• Lakebase, Agent Bricks, or MLflow experience
• Formal mentoring and coaching experience
• Prior experience in insurance or another regulated industry