Senior Data Engineer

Healthmap Solutions is the future of specialty health management that focuses on progressive diseases, particularly in kidney health populations. The Senior Data Engineer drives the design, development, and operational excellence of the data platform, requiring expertise in scalable ETL/ELT and data governance.

Responsibilities:

• Design and implement scalable data pipelines using Delta Lake and manage enterprise-wide data access, security, and lineage using Unity Catalog
• Optimize large-scale Spark jobs (PySpark/SQL) and cluster configurations (Photon) to meet stringent SLA and cost performance targets across all workflows
• Build resilient data scheduling via Databricks Workflows (Jobs) and establish automated CI/CD pipelines for reliable code promotion across Dev, Staging, and Prod workspaces
• Migrate data and models from relational databases to Databricks
• Ensure best practices for development using industry standard development patterns
• Monitor data pipelines performance
• Partner with Data Management and Full Stack development engineers to operationalize models with current applications and processes
• Support existing data pipelines to ensure business continuity
• Stay updated with the latest trends and technologies in data engineering and cloud computing
• Perform other related duties as assigned

Requirements:

• Bachelor's degree is required
• 5+ years of experience in Data Engineering, with a significant focus on data warehousing, ETL/ELT development, and distributed systems
• 3+ years of hands-on experience developing enterprise solutions on the Databricks platform
• Expertise in PySpark and high-performance SQL
• Deep understanding practical knowledge of Delta Lake architecture and optimal maintenance best practices
• Experience with cloud platforms (AWS preferred) and integrating Databricks with native cloud services (S3, Secret Manager, IAM)
• Solid experience implementing CI/CD for Databricks notebooks and associated libraries
• Healthcare experience is preferred