Lead Data Engineer

UST is a mission-driven technology company that helps clients create transformative, human-centered solutions. UST is seeking a Lead Data Engineer to architect, build, and optimize scalable cloud-native data systems, lead enterprise data platform development using Databricks and AWS, and support secure healthcare data processing and governance.

Responsibilities:

• Lead the design, implementation, and maintenance of robust data pipelines on AWS using Databricks
• Architect scalable lakehouse structures leveraging Apache Iceberg to ingest, process, and store complex healthcare data types
• Establish comprehensive data governance and data lineage frameworks using Databricks Unity Catalog
• Implement secure access controls, data masking, and row-level security through Role-Based Access Control (RBAC)
• Architect and scale ingestion frameworks capable of processing HL7 and FHIR data feeds natively into the lakehouse ecosystem
• Oversee end-to-end processing of Medicare Advantage data assets, ensuring accurate risk adjustment and reporting workflows
• Ensure strict adherence to healthcare regulations, including HIPAA compliance and robust data governance frameworks
• Mentor junior engineers, conduct code reviews, and drive technical best practices across the data team

Requirements:

• 7+ years of data engineering experience
• At least 2 years in a technical lead role
• Production experience building and optimizing large-scale pipelines within Databricks
• Production experience deploying data infrastructure on AWS (S3, EMR, Lambda, IAM, Glue)
• Explicit, hands-on experience processing Medicare Advantage data (CMS enrollment, RAPS/EDPS submissions, MMR/MOR files, HCC risk adjustment models)
• Proven experience working with healthcare interoperability standards (FHIR and HL7 models)
• Proven experience implementing enterprise-level Role-Based Access Control (RBAC) and row-level security for PHI
• Mastery of Python, PySpark, and advanced SQL for complex distributed computing tasks
• Deep understanding and practical experience with Apache Iceberg (open table formats)
• Expert-level knowledge of Databricks Unity Catalog for centralized data governance
• Comprehensive understanding of US Healthcare data architecture
• Strong data pipeline design and optimization capabilities
• Security and compliance implementation expertise
• Deep domain expertise in US Healthcare systems
• In-depth knowledge of Medicare Advantage programs and workflows
• Understanding of healthcare regulations and HIPAA compliance requirements
• Knowledge of HL7 and FHIR healthcare interoperability standards
• Databricks Certified Data Engineer Professional
• AWS Certified Data Analytics certification
• Familiarity with orchestration tools like Apache Airflow or AWS Step Functions