NEP Group, Inc. is a leader in live production and virtual solutions for content creators. They are seeking a hands-on DevOps Engineer to build and maintain a modern Azure Databricks Lakehouse, focusing on scalability, security, and automation across environments.
Responsibilities:
- Build and maintain Terraform IaC for Azure + Databricks (workspaces, ADLS Gen2, ADF, Key Vault, networking, Unity Catalog, jobs, SQL Warehouses)
- Create a modular, reusable Terraform structure (modules + environment-specific configs) to support dev/prod consistency
- Own remote state + drift detection (state locking, scheduled plans, reproducibility across environments)
- Design and run CI/CD pipelines (Azure DevOps or GitHub Actions) for infrastructure and Databricks deployments using Databricks Asset Bundles (DABs)
- Automate quality gates: linting + unit tests + notebook validation on PRs, with approval gates into production
- Partner with security and platform stakeholders to implement RBAC, SCIM/Entra ID group management, and secret management using Key Vault + secret scopes
- Implement monitoring, alerting, and FinOps controls for job health, data freshness SLAs, and cost anomalies (DBU + Azure budgets/tags)
Requirements:
- Strong Terraform experience (Azure + Databricks providers), including modules, remote state, and multi-environment patterns
- Proven CI/CD ownership (Azure DevOps or GitHub Actions) with automated testing and release controls
- Solid Azure platform knowledge: ADLS Gen2, Key Vault, ADF, Entra ID, VNets/subnets/NSGs, and cost management
- Databricks administration experience: Unity Catalog, cluster policies, SQL Warehouses, secret scopes, and workspace configuration
- Comfort building operational visibility: dashboards, alerts, and usage/cost tracking
- DLT, dbt, Power BI, Collibra, Airflow
- Databricks REST API automation
- Familiarity with data engineering patterns (PySpark/Delta/Medallion)
- Docker/Kubernetes exposure