Databricks Senior Data & AI Engineer

V2 Strategic Advisors is a boutique management and technology consulting firm that specializes in transforming media and advertising sales organizations. They are seeking a Senior Data & AI Engineer to design and implement production-grade Lakehouse solutions, build pipelines, and develop intelligent applications that deliver exceptional results for clients.

Responsibilities:

• Design and implement medallion architecture (bronze/silver/gold) using Delta Lake as the foundational layer across client environments
• Build and optimize scalable data pipelines using Apache Spark and Lakeflow (Declarative Pipelines, Jobs, and Connect for ingestion)
• Architect and implement Unity Catalog as the backbone that governs data lineage, fine-grained access control (including row/column-level security), Lakehouse Federation, and Delta Sharing
• Drive best practices for performance tuning, cost governance, and compute optimization on Databricks
• Implement and optimize Databricks SQL for enterprise analytics, including AI/BI Dashboards and BI tool integration (Tableau, Power BI, Looker)
• Design data models that serve both operational reporting and strategic analytics use cases
• Partner with business stakeholders to translate complex data requirements into reliable, performant solutions
• Build production AI solutions on the Databricks platform with tools including RAG pipelines using Mosaic Vector Search, agent development with the Mosaic AI Agent Framework, Model Serving endpoints, and external model APIs (OpenAI, Anthropic, Gemini, and others)
• Leverage Mosaic AI, AI/BI Genie, and Databricks Apps to develop intelligent, data-driven applications that deliver real client value
• Stay current on Databricks AI capabilities and bring emerging patterns to the practice proactively
• Serve as a foundational technical resource for V2's Databricks practice, helping shape standards, methodologies, and delivery frameworks
• Operate across client engagements as a hands-on practitioner and technical advisor, translating business problems into platform solutions
• Support pre-sales and solutioning efforts: scoping engagements, contributing to proposals, and demonstrating credibility in client conversations
• Supercharge the broader technical team by leading code reviews, conducting formal and informal knowledge transfer, and elevating the people around you

Requirements:

• Hands-on depth to design and implement production-grade Lakehouse solutions
• Build pipelines, models, and intelligent applications that deliver exceptional results for clients
• Design and implement medallion architecture (bronze/silver/gold) using Delta Lake as the foundational layer across client environments
• Build and optimize scalable data pipelines using Apache Spark and Lakeflow (Declarative Pipelines, Jobs, and Connect for ingestion)
• Architect and implement Unity Catalog as the backbone that governs data lineage, fine-grained access control (including row/column-level security), Lakehouse Federation, and Delta Sharing
• Drive best practices for performance tuning, cost governance, and compute optimization on Databricks
• Implement and optimize Databricks SQL for enterprise analytics, including AI/BI Dashboards and BI tool integration (Tableau, Power BI, Looker)
• Design data models that serve both operational reporting and strategic analytics use cases
• Partner with business stakeholders to translate complex data requirements into reliable, performant solutions
• Build production AI solutions on the Databricks platform with tools including RAG pipelines using Mosaic Vector Search, agent development with the Mosaic AI Agent Framework, Model Serving endpoints, and external model APIs (OpenAI, Anthropic, Gemini, and others)
• Leverage Mosaic AI, AI/BI Genie, and Databricks Apps to develop intelligent, data-driven applications that deliver real client value
• Stay current on Databricks AI capabilities and bring emerging patterns to the practice proactively
• Serve as a foundational technical resource for V2's Databricks practice, helping shape standards, methodologies, and delivery frameworks
• Operate across client engagements as a hands-on practitioner and technical advisor, translating business problems into platform solutions
• Support pre-sales and solutioning efforts: scoping engagements, contributing to proposals, and demonstrating credibility in client conversations
• Supercharge the broader technical team by leading code reviews, conducting formal and informal knowledge transfer, and elevating the people around you
• Platform mastery in Databricks, Delta Lake, Unity Catalog, DLT, Databricks SQL, and Spark
• Experience building pipelines, governing data platforms, and designing Lakehouse architectures in production environments
• Hands-on experience integrating LLMs, building RAG pipelines, and deploying AI-powered solutions using Databricks
• Understanding of how consulting engagements work and experience with scoping, client communication, delivery under pressure, and building trust with senior stakeholders
• Experience with cloud platforms (Azure, AWS, or GCP) and familiarity with adjacent data technologies, including Snowflake, Salesforce Data Cloud, or dbt
• Ability to operate with speed and agility in fast-paced, entrepreneurial environments
• Apply MLflow for experiment tracking, model registry, and deployment lifecycle management where engagements call for it
• Utilize Feature Engineering in Unity Catalog and production model serving to operationalize machine learning at scale
• Collaborate with data science experts on model development when needed - you don't need to be a data scientist, but you need to speak the language fluently
• Databricks Certifications (especially Data Engineer Professional and ML Engineer Professional) are strongly preferred