EDP Platform Engineer, Databricks Administrator

Role Overview

• Administer Databricks account and workspaces across SDLC environments; standardize configuration, naming, and operational patterns
• Configure and maintain clusters/compute, job compute, SQL warehouses, runtime versions, libraries, repos, and workspace settings
• Implement platform monitoring/alerting, operational dashboards, and health checks; maintain runbooks and operational procedures
• Provide Tier 2/3 operational support: troubleshoot incidents, perform root-cause analysis, and drive remediation and preventive actions
• Manage change control for upgrades, feature rollouts, configuration changes, and integration changes; document impacts and rollback plans
• Enforce least privilege across platform resources (workspaces, jobs, clusters, SQL warehouses, repos, secrets) using role/group-based access patterns
• Configure and manage secrets and secure credential handling (secret scopes / key management integrations) for platform and data connectivity
• Enable and maintain audit logging and access/event visibility; support security reviews and evidence requests
• Administer Unity Catalog governance: metastores, catalogs/schemas/tables, ownership, grants, and environment/domain patterns
• Configure and manage external locations, storage credentials, and governed access to cloud object storage
• Partner with governance stakeholders to support metadata/lineage integration, classification/tagging, and retention controls where applicable
• Coordinate secure connectivity and guardrails with cloud/network teams: private connectivity patterns, egress controls, firewall/proxy needs
• Configure cloud integrations required for governed data access and service connectivity (roles/permissions, endpoints, storage integrations)
• Implement cost guardrails: cluster policies, auto-termination, scheduling, workload sizing standards, and capacity planning
• Produce usage/cost insights and optimization recommendations, address waste drivers (idle compute, oversized clusters, inefficient jobs)
• Automate administration and configuration using APIs/CLI/IaC (e.g., Terraform) to reduce manual drift and improve repeatability
• Maintain platform documentation: configuration baselines, security/governance standards, onboarding guides, and troubleshooting references
• Design and implement backup and disaster recovery procedures for workspace configurations, notebooks, Unity Catalog metadata, and job definitions; maintain recovery runbooks and perform periodic DR testing aligned to RTO/RPO objectives
• Monitor and optimize platform performance, including SQL warehouse query tuning, cluster autoscaling configuration, Photon enablement, and Delta Lake optimization guidance (OPTIMIZE, VACUUM, Z-ordering strategies)
• Administer Delta Live Tables (DLT) pipelines and coordinate with data engineering teams on pipeline health, data quality monitoring, failed job remediation, and pipeline configuration best practices
• Manage third-party integrations and ecosystem connectivity, including BI tool integrations (e.g., Power BI), and external metadata catalog integrations
• Implement Databricks Asset Bundles (DABs) for standardized deployment patterns; automate workspace resource deployment (jobs, pipelines, dashboards) across SDLC environments using bundle-based CI/CD workflows
• Conduct capacity planning and scalability analysis, including forecasting concurrent user/workload growth, platform scaling strategies, and proactive resource allocation during peak usage periods
• Facilitate user onboarding and enablement, including new user/team onboarding procedures, training coordination, workspace access provisioning, and creation of self-service documentation/guides

Requirements

• 7+ years in cloud/data platform administration and operations, including 4+ years supporting Databricks or similar platforms
• Experience with the Scrum framework, Agile engineering, Lean methodologies or DevOps
• Experience with one or more of the following: system development, software development, hardware development, or mission support
• Experience working with DevOps CI/CD related technologies (Azure DevOps, Git, Jenkins, Puppet, Docker, Confluence, Sonar Lint, and J-Unit
• Hands-on experience administering Databricks (workspace administration, clusters/compute policies, jobs, SQL warehouses, repos, runtime management) and expertise using Databricks CLI
• Strong Unity Catalog administration: metastores; catalogs/schemas; grants; service principals; external locations; storage credentials; governed storage access
• Identity & Access Management proficiency: SSO concepts, SCIM provisioning, group-based RBAC, service principals, least-privilege patterns
• Security fundamentals: secrets management, secure connectivity, audit logging, access monitoring, and evidence-ready operations
• Cloud platform expertise (AWS): IAM roles/policies, object storage security patterns, networking basics (VPC concepts), logging/monitoring integration
• Automation skills: scripting and/or IaC using Terraform/CLI/REST APIs for repeatable configuration and environment promotion
• Experience implementing data governance controls (classification/tagging, lineage/metadata integrations) in partnership with governance teams
• CI/CD practices for jobs/notebooks/config promotion across SDLC environments
• Understanding of lakehouse concepts (e.g., Delta, table lifecycle management, separation of storage/compute)
• SQL proficiency and data engineering fundamentals for troubleshooting query performance issues, understanding ETL/ELT workflow patterns, and debugging data pipeline failures; basic Python/Scala familiarity for notebook/code issue diagnosis
• Experience with compliance and regulatory frameworks (FedRAMP, HIPAA, SOC2, or similar) including implementation of data residency requirements, retention policies, and audit-ready evidence collection
• Hands-on experience with AWS security and networking services, including PrivateLink, Secrets Manager/Systems Manager integration, CloudWatch/CloudTrail integration, S3 bucket policies, cross-account access patterns, and KMS encryption key management
• Experience administering Databricks serverless compute, Workspace Git integrations (GitLab), Databricks Asset Bundles (DABs) for deployment automation, and modern workspace features supporting DevOps workflows
• SLA/SLO management and stakeholder communication skills; ability to define platform service levels, produce operational reports, translate technical issues to business stakeholders, and manage vendor relationships (Databricks account teams)

Tech Stack

• AWS
• Azure
• Cloud
• Docker
• ETL
• Jenkins
• Puppet
• Python
• Scala
• SDLC
• SQL
• Terraform
• Unity

Benefits

• Health insurance plans
• Health Savings Account (HSA)
• Dental
• Vision
• Long-term disability
• Short-term disability
• Basic term life insurance
• Supplemental term life insurance for employees, spouses, and dependents
• Simple IRA
• Parking/Commuting expense reimbursement
• Training/Education