Azure Data Engineer AI & Data Platform

Dice is seeking an experienced Azure Data Engineer with expertise in cloud data platforms and AI-driven solutions. The role involves designing, building, and optimizing scalable data pipelines while integrating enterprise data sources and leveraging Azure AI technologies to enhance data engineering productivity.

Responsibilities:

• Design, develop, and maintain scalable data ingestion, transformation, and ETL/ELT pipelines using Azure services
• Build and optimize data solutions using Azure Data Factory, Azure Synapse Analytics, Microsoft Fabric, Azure Databricks, and Azure Data Lake Storage
• Develop batch and real-time data processing solutions using Spark, SQL, and Python
• Implement data modeling solutions aligned with enterprise standards and business requirements
• Manage data integration across multiple source systems, APIs, databases, and cloud platforms
• Develop AI-assisted solutions for automated data pipeline generation, code acceleration, and notebook creation
• Integrate Azure OpenAI and Azure AI Services into data engineering workflows
• Build intelligent data quality monitoring, anomaly detection, and predictive analytics solutions
• Design and implement AI-powered agents for operational monitoring, user notifications, and workflow automation
• Evaluate and implement emerging AI technologies to improve engineering efficiency and business outcomes
• Implement data quality frameworks, validation rules, and monitoring mechanisms
• Ensure compliance with security, privacy, and governance requirements
• Establish metadata management, lineage tracking, and data cataloging processes
• Support audit, compliance, and regulatory reporting requirements
• Monitor and troubleshoot data pipelines and platform issues
• Perform root cause analysis and implement preventive measures
• Support production deployments, release management, and operational excellence initiatives
• Collaborate with cross-functional teams to resolve data and platform issues

Requirements:

• SQL
• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Microsoft Fabric
• Azure Databricks
• Azure Data Lake Storage (ADLS)
• SQL, Python, PySpark
• Data Modeling and Data Warehousing
• Azure DevOps (ADO), CI/CD
• Git Version Control
• REST APIs and Data Integration
• Azure OpenAI Service
• Azure AI Services
• Generative AI and Large Language Models (LLMs)
• AI Agents and Workflow Automation
• Retrieval-Augmented Generation (RAG)
• Prompt Engineering
• Intelligent Monitoring and AI-driven Operations
• Problem Solving & Analytical Thinking
• Stakeholder Management
• Collaboration & Communication
• Innovation & Continuous Improvement
• Operational Excellence
• AI Adoption & Automation Mindset