Senior Azure Data Engineer

Highbrow Technology Inc is seeking a highly skilled Senior Azure Data Engineer with strong expertise in Databricks and Azure data services. The role involves designing, building, and optimizing scalable data pipelines and enterprise-grade lakehouse architectures in hybrid cloud environments.

Responsibilities:

• Design, build, and orchestrate scalable data pipelines using Azure-native services
• Develop and optimize batch and near real-time data processing solutions using Spark and Databricks
• Implement and manage enterprise-grade Lakehouse architectures (Medallion architecture)
• Perform data processing, validation, and automation using Python (PySpark, Pandas, NumPy)
• Optimize performance of large-scale data systems, including Spark tuning and query optimization
• Work with CDC (Change Data Capture) pipelines and large-volume data ingestion
• Collaborate with cross-functional teams to deliver secure, scalable, and high-performance data platforms
• Implement CI/CD pipelines and DevOps best practices for data engineering workflows

Requirements:

• Strong experience with Azure Databricks, including notebooks, clusters, and Delta Lake
• Hands-on expertise in Azure Data Factory (ADF) and pipeline orchestration
• Proficiency in Python (PySpark, Pandas, NumPy) and SQL
• Experience with Spark SQL, performance tuning, and distributed data processing
• Solid understanding of Azure ecosystem: ADLS, Synapse Analytics, Azure Functions, Event Grid, Key Vault, Purview
• Experience designing and implementing data lake / lakehouse architectures
• Knowledge of dimensional data modeling and analytics solutions
• Familiarity with DevOps and CI/CD automation in data platforms
• Experience with Microsoft Fabric
• Proven track record of improving pipeline performance (e.g., 30–40% optimization)
• Exposure to hybrid cloud environments
• Prior experience working with large enterprise clients