Azure AI Architect
Chicago, IL( Monthly 2-3 days onsite)
Long term contract
Azure Databricks & Agentic AI Architect
· Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering
A forward-looking architect with 15+ years of Data & Analytics experience, strong Azure Databricks expertise, and hands-on experience building Agentic AI platforms, AI-powered SDLC frameworks, and autonomous data engineering ecosystems. The candidate should be comfortable leading enterprise-scale AI transformation initiatives and engaging CXO stakeholders on AI strategy and business value.
Role Summary:
We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.
The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.
Key Responsibilities
Agentic Data Engineering Leadership
· Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.
· Define autonomous workflows using AI Agents for:
· Data ingestion
· Data mapping
· Schema evolution
· Data quality remediation
· Metadata Enrichment
· Pipeline optimization
· Root cause analysis
· Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.
AI-Driven Data Lifecycle (AI-DLC)
· Lead architecture for AI-enabled Data Development Lifecycle across:
· Requirement analysis
· Data modeling
· Pipeline generation
· Automated testing
· Code review
· Documentation
· Deployment
· Monitoring
· Implement AI copilots to accelerate developer productivity.
· Enable automated lineage creation and intelligent impact analysis.
Lakehouse & Data Platform Architecture
· Design scalable Lakehouse platforms using:
· Azure Databricks
· Delta Lake
· Unity Catalog
· ADLS Gen2
· Databricks Workflows
· Delta Live Tables
Enterprise GenAI Integration
· Architect RAG-based solutions using enterprise data assets.
· Design agent orchestration frameworks using:
· Azure OpenAI
· LangGraph
· Semantic Kernel
· AutoGen
· MCP-enabled architectures
· Build domain-specific AI agents supporting Data Engineering and Analytics teams.
AI Governance & Responsible AI
· Define guardrails for enterprise GenAI adoption.
· Implement:
· Prompt governance
· Observability
· Cost monitoring
· Auditability
· Explainability
· Security controls
· Establish governance models for autonomous AI agents.
AI-Powered Platform Optimization
· Design self-healing data pipelines.
· Implement AI-driven:
· Incident triage
· Failure prediction
· Capacity planning
· Cost optimization
· SLA monitoring
· Enable intelligent workload placement and model routing.
Technical Skills
· Data Platform
· Azure Databricks
· Delta Lake
· Unity Catalog
· Azure Data Factory
AI & Agentic Frameworks
· Azure OpenAI
· Knowledge Graph
· RAG Architecture
· LangChain
· LangGraph
· MCP Protocol
· Vector Databases
· AI Agent Orchestration
Data Engineering
· PySpark
· Spark SQL
· Python
· SQL
· ELT/ETL Modernization
DevOps & AI-DLC
· Azure DevOps
· GitHub Actions
· CI/CD
· MLOps
· LLMOps
· Evaluation Frameworks
· AI Testing Frameworks
Leadership Expectations
· Drive AI-First Data Engineering transformation.
· Define enterprise patterns, accelerators, and reusable AI agents.
· Mentor architects, data engineers, and AI engineers.
· Lead executive conversations on AI adoption, ROI, and transformation roadmaps.
Preferred Certifications
· Databricks Certified Data Engineer Professional
· Databricks Certified Solution Architect
· Microsoft Azure Solution Architect (AZ-305)
· Azure Data Engineer (DP-203)
· Microsoft Applied Skills – Azure OpenAI
· Generative AI / Agentic AI Certifications
