Hands-on experience with Azure Databricks platform support or administration.
Good understanding of Databricks workspaces, compute, jobs, notebooks, SQL Warehouses, libraries, and permissions.
Experience troubleshooting Databricks production issues.
Working knowledge of Unity Catalog and ADLS Gen2 integration.
Understanding of Azure networking basics: VNet, subnet, NSG, route table, NAT Gateway, firewall, private endpoint, Private Link, and DNS.
Detailed JD
We are seeking an Azure Databricks Platform Support Engineer to ensure the reliability, performance, and security of our Databricks environment. The role focuses on day-to-day platform administration, production troubleshooting, and enabling data engineering and analytics teams across the organisation.
Key Responsibilities
Administer and support Azure Databricks workspaces, compute, jobs, notebooks, and SQL Warehouses.
Troubleshoot and resolve production incidents by analysing job logs, cluster events, error messages, and audit information.
Manage Unity Catalog governance and ADLS Gen2 integration, including permissions and access controls.
Configure and maintain compute policies, instance pools, autoscaling, auto-termination, and cost controls.
Support workspace networking and connectivity, including VNet injection, private endpoints, and Private Link.
Maintain operational documentation, runbooks, and change records for the platform.
Required Skills
Hands-on experience with Azure Databricks platform support or administration.
Good understanding of Databricks workspaces, compute, jobs, notebooks, SQL Warehouses, libraries, and permissions.
Experience troubleshooting Databricks production issues.
Working knowledge of Unity Catalog and ADLS Gen2 integration.
Understanding of Azure networking basics: VNet, subnet, NSG, route table, NAT Gateway, firewall, private endpoint, Private Link, and DNS.
Familiarity with compute policies, instance pools, runtime versions, autoscaling, auto-termination, and cost controls.
Ability to read job logs, cluster events, error messages, and audit information.
Strong communication and documentation skills.
Preferred Skills
Experience with VNet-injected Databricks workspaces and Secure Cluster Connectivity.
Experience with Azure Key Vault, Azure Monitor, Log Analytics, and diagnostic logs.
Knowledge of PySpark, Spark SQL, Delta Lake, and Medallion Architecture.
Experience with Databricks CLI, REST APIs, Terraform, Bicep, Azure DevOps, or GitHub.
ITIL-style incident, problem, and change management experience.
Tools & Technologies
Azure Databricks Unity Catalog Delta Lake Apache Spark PySpark Spark SQL ADLS Gen2 Azure Key Vault Azure Monitor Log Analytics Azure Networking Databricks CLI REST APIs Azure DevOps GitHub
