Lead Data Engineer

UST is a mission-driven technology company that creates transformative, human-centered solutions through innovation and agility. UST is seeking a Lead Data Engineer to architect, build, and optimize scalable cloud-native data systems, lead enterprise data platform development using Databricks and AWS, and support secure healthcare data processing and governance.

Responsibilities:

• Lead the design, implementation, and maintenance of robust data pipelines on AWS using Databricks
• Architect scalable lakehouse structures leveraging Apache Iceberg to ingest, process, and store complex healthcare data types
• Establish comprehensive data governance and data lineage frameworks using Databricks Unity Catalog
• Implement secure access controls, data masking, and row-level security through Role-Based Access Control (RBAC)
• Architect and scale ingestion frameworks capable of processing HL7 and FHIR data feeds natively into the lakehouse ecosystem
• Oversee end-to-end processing of Medicare Advantage data assets, ensuring accurate risk adjustment and reporting workflows
• Ensure strict adherence to healthcare regulations, including HIPAA compliance and robust data governance frameworks
• Mentor junior engineers, conduct code reviews, and drive technical best practices across the data team

Requirements:

• · 7+ years of data engineering experience
• · At least 2 years in a technical lead role
• · Production experience building and optimizing large-scale pipelines within Databricks
• · Production experience deploying data infrastructure on AWS (S3, EMR, Lambda, IAM, Glue)
• · Explicit, hands-on experience processing Medicare Advantage data (CMS enrollment, RAPS/EDPS submissions, MMR/MOR files, HCC risk adjustment models)
• · Proven experience working with healthcare interoperability standards (FHIR and HL7 models)
• · Proven experience implementing enterprise-level Role-Based Access Control (RBAC) and row-level security for PHI
• · Mastery of Python, PySpark, and advanced SQL for complex distributed computing tasks
• · Deep understanding and practical experience with Apache Iceberg (open table formats)
• · Expert-level knowledge of Databricks Unity Catalog for centralized data governance
• · Comprehensive understanding of US Healthcare data architecture
• · Strong data pipeline design and optimization capabilities
• · Security and compliance implementation expertise
• · Deep domain expertise in US Healthcare systems
• · In-depth knowledge of Medicare Advantage programs and workflows
• · Understanding of healthcare regulations and HIPAA compliance requirements
• · Knowledge of HL7 and FHIR healthcare interoperability standards
• · Databricks Certified Data Engineer Professional
• · AWS Certified Data Analytics certification
• · Familiarity with orchestration tools like Apache Airflow or AWS Step Functions