HYR Global Source Inc is seeking an experienced Lead Data Engineer to drive large-scale data platform modernization initiatives. This role involves leading Databricks migration and designing enterprise data architecture while collaborating with cross-functional stakeholders.
Responsibilities:
- Lead end-to-end Databricks migration initiatives from legacy platforms (e.g., DataStage) to modern cloud-based architectures
- Design and implement scalable data architecture frameworks in Azure environments
- Build and optimize ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, PySpark, and SQL
- Architect robust data lakes and lakehouse solutions
- Develop and maintain high-performance data workflows using Python and PySpark
- Collaborate with Data Analysts, Data Scientists, and Business stakeholders to define data requirements
- Ensure data quality, governance, security, and performance optimization
- Mentor and lead junior/mid-level data engineers
- Participate in architectural discussions and provide technical leadership
Requirements:
- 10+ years of experience in Data Engineering
- Strong hands-on experience with Databricks (implementation & migration projects)
- Strong hands-on experience with SQL (advanced query optimization and performance tuning)
- Strong hands-on experience with Azure Data Factory (ADF)
- Strong hands-on experience with DataStage
- Strong hands-on experience with Python & PySpark
- Experience designing enterprise-level data architecture solutions
- Strong knowledge of cloud data platforms (preferably Azure)
- Experience with data modeling and large-scale data processing