Home
Jobs
Saved
Resumes
Data Engineering Lead at The Muse | JobVerse
JobVerse
Home
Jobs
Recruiters
Companies
Pricing
Blog
Jobs
/
Data Engineering Lead
The Muse
Website
LinkedIn
Data Engineering Lead
United States
Full Time
1 week ago
$120,000 - $145,000 USD
No H1B
Apply Now
Key skills
Apache
AWS
Azure
ETL
PySpark
Python
Scala
Spark
SQL
ML
LLM
LangChain
MLflow
Data Engineering
Data Warehousing
Analytics
BI
Power BI
Snowflake
Databricks
Apache Spark
dbt
Azure DevOps
Logic Apps
GitHub
Version Control
Workday
CI/CD
About this role
Role Overview
Build and optimize distributed data processing jobs using Apache Spark on Databricks.
Implement Delta Lake, DLT pipelines, dbt transformations and Medallion architecture for scalable and reliable data workflows.
Design and automate ETL pipelines using Azure Data Factory, Databricks, and Synapse Analytics.
Integrate data from diverse sources including Duck Creek, Intacct, Workday and external APIs.
Develop dimensional models (Star/Snowflake schemas), stored procedures, and views for data warehouses.
Ensure efficient querying and transformation using SQL, T-SQL, and PySpark.
Leverage Azure DevOps, CI/CD pipelines, and GitHub for version control and deployment.
Utilize Azure Logic Apps and ML Flow for workflow automation and model training.
Implement role-based access control (RBAC), data encryption, and auditing mechanisms.
Work closely with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.
Mentor junior engineers and contribute to code reviews and architectural decisions.
Requirements
Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
5+ years of experience in data engineering with at least 2 years on Databricks.
Proficiency in Python, Scala, SQL, and Spark.
Hands-on experience with Azure Data Services (ADF, ADLS, Synapse).
Strong understanding of ETL, data warehousing, and data modeling concepts.
Experience with Microstrategy, Power BI, including DAX and advanced visualizations.
Familiarity with MLflow, LangChain, and LLM integration is a plus.
Knowledge of Duck Creek a plus.
Insurance Domain knowledge preferred.
Preferred Certifications Databricks Data Engineering Professional Azure/AWS Data Engineering Certifications
Tech Stack
Apache
AWS
Azure
ETL
PySpark
Python
Scala
Spark
SQL
Benefits
Opportunities to stretch and grow: your professional and personal development matters to us.
Clarity and kindness: you can rely on us to be open, honest and supportive, offering clarity on what success looks like.
Support in good times and bad: we believe in showing up for each other consistently, not only when it’s easy.
A community that cares: we are committed to sustaining a community in which each person feels cared for as an individual.
Apply Now
Home
Jobs
Saved
Resumes