Azure Databricks Architect

Role Overview

• Architect and design enterprise-scale data platforms leveraging Azure Databricks and Azure cloud services;
• Define and implement Lakehouse architecture patterns using Delta Lake and Databricks best practices;
• Design scalable data ingestion, transformation, and serving frameworks supporting both batch and streaming workloads;
• Design and implement real-time and near real-time data processing solutions using Structured Streaming, CDC, Change Data Feed (CDF), and event-driven architectures;
• Design and manage multi-workspace Databricks environments and shared Metastore architectures.
• Establish platform governance, security controls, cluster policies, RBAC models, and access management frameworks;
• Design and implement CI/CD pipelines for Databricks assets, infrastructure, and data workloads.
• Automate infrastructure provisioning using Infrastructure as Code (Terraform, ARM, or Bicep);
• Develop deployment strategies and release management processes across development, staging, and production environments;
• Optimize Databricks workspaces, clusters, jobs, and storage configurations for performance and cost efficiency;
• Implement monitoring, alerting, logging, and observability solutions for platform operations;
• Define and enforce data architecture standards, data quality frameworks, and governance policies;
• Integrate Azure Databricks with Azure Data Factory, Azure Synapse, Azure Storage, Azure Key Vault, Azure Entra ID, and other Azure services;
• Support Lakeflow Connect, managed connectors, CDC implementations, and enterprise data ecosystem interoperability;
• Collaborate with engineering and business teams to translate requirements into scalable technical solutions;
• Provide architectural guidance, technical leadership, and mentorship to engineering teams.

Requirements

• 7+ years of experience in Data Engineering, Data Platform Engineering, Cloud Data Architecture, or Data Platform Consulting.
• 3+ years of hands-on experience architecting solutions with Azure Databricks.
• Strong expertise in Apache Spark, PySpark, Delta Lake, and Databricks Lakehouse architecture.
• Hands-on experience with Unity Catalog, data governance, lineage, permissions models, and catalog isolation strategies.
• Experience designing and managing Databricks Metastore and multi-workspace architectures.
• Strong knowledge of Databricks administration, user provisioning, RBAC, workspace management, and security controls.
• Extensive experience designing cloud-native solutions on Microsoft Azure.
• Strong knowledge of Azure Data Factory, Azure Storage, Azure Key Vault, Azure Monitor, Azure Entra ID, and Azure networking concepts.
• Experience designing streaming and low-latency data pipelines using Databricks Structured Streaming.
• Hands-on experience with CDC, Change Data Feed (CDF), and modern data ingestion patterns.
• Hands-on experience building and maintaining CI/CD pipelines using Azure DevOps and/or GitHub Actions.
• Experience implementing Infrastructure as Code using Terraform, ARM Templates, or Bicep.
• Strong understanding of DevOps principles, deployment automation, and environment management.
• Strong SQL and Python programming skills.
• Excellent communication, stakeholder management, and consulting skills.

Tech Stack

• Apache
• Azure
• Cloud
• PySpark
• Python
• Spark
• SQL
• Terraform
• Unity
• Vault

Benefits

• Digital-First Approach: Great talent knows no borders! You can work from wherever you are — we hire and collaborate with professionals worldwide.
• Remote Work Model: Balance your professional and personal life with our flexible working conditions, empowering you to deliver your best from anywhere.
• Exciting Projects: Dive into impactful projects across industries that challenge and spark creativity.
• Boost Your Expertise: Grow your career with continuous learning, development opportunities, and hands-on experience.
• Join the Best Team Ever: Collaborate with our diverse and cross-cultural team of passionate technologists and creative thinkers.