Azure Data Engineer (5-9 Years)

Highbrow Technology Inc is seeking a highly skilled Azure Data Engineer with expertise in Databricks and Azure data services. The role involves designing, building, and optimizing scalable data pipelines and enterprise-grade lakehouse architectures in hybrid cloud environments.

Responsibilities:

• Design, build, and orchestrate scalable data pipelines using Azure-native services
• Develop and optimize batch and near real-time data processing solutions using Spark and Databricks
• Implement and manage enterprise-grade Lakehouse architectures (Medallion architecture)
• Perform data processing, validation, and automation using Python (PySpark, Pandas, NumPy)
• Optimize performance of large-scale data systems, including Spark tuning and query optimization
• Work with CDC (Change Data Capture) pipelines and large-volume data ingestion
• Collaborate with cross-functional teams to deliver secure, scalable, and high-performance data platforms
• Implement CI/CD pipelines and DevOps best practices for data engineering workflows

Requirements:

• Strong experience with Azure Databricks, including notebooks, clusters, and Delta Lake
• Hands-on expertise in Azure Data Factory (ADF) and pipeline orchestration
• Proficiency in Python (PySpark, Pandas, NumPy) and SQL
• Experience with Spark SQL, performance tuning, and distributed data processing
• Solid understanding of Azure ecosystem: ADLS, Synapse Analytics, Azure Functions, Event Grid, Key Vault, Purview
• Knowledge of dimensional data modeling and analytics solutions