AWS Databricks Engineer - Healthcare Payer (Fully Remote)

Dice is hiring an AWS Databricks Engineer to design, build, and maintain scalable data pipelines and Lakehouse architectures for healthcare payer data. The role focuses on PySpark, Python, SQL, AWS services, data governance, orchestration, CI/CD, and collaboration with stakeholders to prepare datasets and build feature stores.

Responsibilities:

• Design and build batch and streaming data pipelines using PySpark, Delta Lake, Autoloader and Delta Live Tables (DLT) to ingest data sets into Databricks
• Build cloud-native data solutions leveraging AWS services (e.g., S3 for storage, IAM for access management, and AWS Glue or Lambda for serverless tasks
• Configure and manage access controls using Databricks Unity Catalog to ensure compliance and monitor lineage
• Automate pipeline deployments using Databricks Workflows, Apache Airflow, and CI/CD tools (e.g., GitHub, GitLab). [1, 2, 3]
• Partner closely with stakeholders to build robust feature stores and prepare datasets for various consumption needs

Requirements:

• Industry: Healthcare Payer industry experience. Have worked on MMIS data sets – claims, provider, member enrollment and similar data sets
• Experience: 3-5 years of hands-on data engineering experience, specifically on Databricks
• Programming: High proficiency in Python (specifically PySpark) and advanced SQL
• Big Data & Cloud: Strong understanding of Apache Spark, Data Lakehouse architecture, and working within a production AWS environment
• Databricks Ecosystem: Familiarity with the Databricks platform ecosystem, including notebooks, Delta Lake, and Unity Catalog