Design, develop, and optimize complex batch and real-time data pipelines using Databricks, Spark, and related technologies
Build and maintain scalable lakehouse architectures and high-performance data models to support business driven outcomes
Provide technical leadership and day-to-day coordination for a small offshore data engineering team
Define tasks, review deliverables, and ensure adherence to engineering standards and timelines
Lead the implementation of advanced ETL/ELT frameworks and reusable data engineering patterns
Solve complex data challenges including performance tuning, data skew, partitioning, and large-scale processing
Implement robust data quality, validation, and observability frameworks
Collaborate closely with API, Analytics, and Platform teams to deliver reliable, production-grade data assets
Drive best practices in coding standards, testing, and CI/CD for data pipelines
Troubleshoot and resolve production issues, ensuring high availability and reliability
Act as the primary liaison between onshore stakeholders and offshore delivery teams

8–12 years of experience in data engineering
Proven experience leading or coordinating offshore or distributed engineering teams
Deep hands-on expertise with Databricks, Spark, and distributed data processing
Advanced proficiency in Python and/or Scala, plus strong SQL skills
Strong experience with performance tuning and optimization of large-scale data pipelines
Experience with cloud data platforms (Azure strongly preferred)
Solid understanding of data modeling, streaming architectures, and data lifecycle management
Experience implementing data quality, lineage, and governance controls

Lead Data Engineer

Key skills