Senior Data Engineer

Hylant is an insurance brokerage seeking a Senior Data Engineer to operationalize its enterprise data architecture on Databricks and Azure. The role focuses on building reliable data pipelines, creating business-ready datasets, enforcing governance and security, supporting reporting, and optimizing data platform performance and cost.

Responsibilities:

• Design, build, and maintain data transformation logic, business rules, and analytics-ready datasets using Databricks notebooks, PySpark, and SQL
• Develop, orchestrate, and monitor ETL/ELT pipelines using Azure Databricks and Azure Data Factory, including scheduling, dependency management, and job performance tuning
• Create target schemas, curated tables, and analytics-ready data layers aligned to architectural designs and business requirements, including source-to-target mapping, historical data loads, and backfills for new data domains
• Implement and enforce data governance, security, and access controls within Unity Catalog — including group-based grants, attribute-based access policies, and row- and column-level security — in accordance with standards defined by the Data Architecture team
• Conduct data profiling, validation, reconciliation, and system integration testing to ensure end-to-end data quality, accuracy, and reliability; partner with source system owners and downstream consumers to investigate and resolve discrepancies
• Serve curated, trusted datasets through Databricks SQL and Power BI datasets to support operational, financial, and analytical reporting
• Own the CI/CD process for Databricks and Azure Data Factory using Azure DevOps Pipelines
• Monitor, tune, and optimize the data platform for performance and cost, including cluster sizing, autoscaling, and job execution efficiency
• Participate in agile delivery as part of the Data & Digital Solutions team, including sprint planning, backlog refinement, and delivery commitments

Requirements:

• Bachelor's degree in computer science, engineering, data science, or a related field, or equivalent practical experience (desired)
• 5–7+ years of experience as a Data Engineer, including designing, implementing, and validating production-grade data pipelines
• Hands-on experience with Azure Databricks, Azure Data Factory, Delta Lake, Unity Catalog, and core Azure cloud services
• Strong proficiency in SQL and PySpark for data transformation and pipeline development
• Experience implementing data quality checks, reconciliations, and validation processes in complex data environments
• Experience owning or contributing to CI/CD pipelines for data platforms (Azure DevOps Pipelines preferred)
• Ability to troubleshoot and resolve complex data, pipeline, and performance issues
• Strong collaboration and communication skills, with the ability to bridge technical and non-technical stakeholders
• Must be legally authorized to work in the United States
• This is a fully remote position, open to candidates located within the United States
• Periodic in-person attendance may be required for team meetings, onsite collaboration, or other business-related functions; reasonable advance notice will be provided
• Work is generally performed in a home office or standard office environment with limited exposure to physical risk or physical effort required
• Reasonable accommodation may be made to enable individuals with disabilities to perform the essential functions
• Insurance industry experience (brokerage, policy administration, claims, billing, underwriting, or risk data) preferred but not required
• Exposure to applying AI or automation within data engineering workflows a plus
• Exposure to PowerBI is a plus