Design, build, and maintain a scalable data platform on Databricks and AWS, following Medallion (Bronze/Silver/Gold) architecture principles.
Develop and manage data ingestion and ETL/ELT pipelines from core business systems, ensuring data quality, reliability, and performance.
Implement data quality monitoring (freshness, completeness, validity, anomaly detection) with automated alerting.
Establish and maintain Golden Record identity resolution and reverse sync across source systems.
Govern metadata, lineage, and permissions using Databricks Unity Catalog and platform standards.
Collaborate with cross-functional teams to deliver analytics
and AI-ready datasets that support business objectives.
Apply best practices for infrastructure as code, CI/CD, and version control for data pipelines.
Communicate effectively with stakeholders, manage ambiguity, and drive results in a fast-paced environment.

8+ years of experience in data engineering, building production-grade ETL/ELT pipelines at scale.
Strong proficiency in Python (PySpark, scripting, API integrations, pipeline automation).
Hands-on experience with Databricks (notebooks, jobs, workflows, Unity Catalog) and AWS (Lambda, ECS/Fargate, S3, Glue, Athena, DynamoDB, EventBridge, Step Functions, IAM).
Expertise in data modeling and lakehouse architectures (Medallion, Delta Lake, Parquet, schema evolution, incremental upserts).
Proficiency in SQL and dbt for building modular, tested, and documented models.
Experience with REST API integration, entity resolution, and record matching strategies.
Familiarity with Git-based version control and CI/CD for data pipelines.

Senior Data Engineer

Key skills