AI focused DataBricks Developer (Digital Engineering)

Coherent Solutions is seeking an AI focused DataBricks Developer to work on a project for Allianz Life, a leading American life insurance company. The role involves developing and maintaining data pipelines within an Azure-based data platform while collaborating with a team to ensure data quality and reliability.

Responsibilities:

• Develop and maintain data pipelines within the Azure-based data platform
• Work with the Databricks ecosystem (Delta Tables, Unity Catalog, notebooks) to process and manage data
• Build and optimize data transformations using Python and SQL
• Develop and support AI-driven data pipelines and automation using Databricks (Mosaic AI or similar tools)
• Support and enhance existing production data systems
• Collaborate with a team of engineers on Databricks-based solutions
• Ensure data quality, consistency, and reliability across data pipelines

Requirements:

• 3+ years of experience as a Data Engineer
• Strong experience with Databricks ecosystem (Architecture, Unity Catalog, Delta tables, lineage, permissions, Databricks Genie, notebooks)
• Strong experience with Python
• Solid proficiency in SQL
• Experience working in Azure cloud environment
• Experience with Databricks pipelines AI automation (Mosaic AI or AWS Agentcore)
• Experience with Azure data integration tools (Synapse Pipelines, Azure Data Factory)
• Experience with Azure SQL pools
• Understanding of performance tuning for data workloads
• Experience with CI/CD, DevOps practices, Infrastructure as Code (IaC)