Design and build end-to-end data and AI pipelines using Azure Databricks.
Develop robust ETL/ELT workflows using Python (PySpark) and SQL.
Implement CI/CD pipelines for Databricks deployments (jobs, notebooks, workflows).
Integrate Databricks with Azure services (Data Lake, Blob Storage, Key Vault, Azure OpenAI, Azure Functions, etc.).
Optimize jobs for performance, cost, and reliability.
Build reusable, modular code.
Collaborate with data scientists and platform teams to move models from experimentation to production.
Implement logging, monitoring, and error handling for production pipelines.
Develop and deploy ML and Generative AI models (LLMs, embeddings, RAG pipelines) for NLP, computer vision, and predictive analytics.
Fine-tune LLMs using LoRA/QLoRA and integrate with Azure OpenAI or Hugging Face models.
Implement vector search and retrieval pipelines using FAISS or Azure Cognitive Search.
Ensure responsible AI practices, including bias detection and model governance.

Lead AI Engineer

Key skills