Lead Databricks Engineer
Atlanta, Texas, United States of America
Contract
2 weeks ago
Key skills
PythonScalaSQLETLELTDatabricksSparkAirflowApache AirflowAWSAzureGoogle CloudTerraformGitHub ActionsLambdaS3GlueAzure DevOpsGitHubPerformance OptimizationCI/CDLeadershipMentoring
About this role
Responsibilities
- Architecture & Strategy: Designing scalable Medallion architectures (Bronze, Silver, Gold layers) and defining enterprise data strategies.
- Pipeline Engineering: Leading the development of robust, end-to-end ETL/ELT pipelines for both batch and real-time streaming data.
- Performance Optimization: Tuning Spark workloads, optimizing Delta Lake storage (e.g., Z-Ordering, partitioning), and managing DBU (Databricks Unit) costs.
- Governance & Security: Implementing Unity Catalog for centralized access control, data lineage, and auditability.
- DevOps & Automation: Establishing CI/CD practices using tools like Terraform, Azure DevOps, or GitHub Actions for automated deployments.
- Leadership & Mentorship: Guiding cross-functional teams, performing code reviews, and mentoring junior engineers.
Key Technical Skills
- Databricks Ecosystem: Deep expertise in PySpark, Spark SQL, Delta Lake, and Delta Live Tables (DLT).
- Programming: Mastery of Python or Scala, and advanced proficiency in SQL.
- Cloud Platforms: Strong hands-on experience with at least one major provider Azure (ADF, ADLS), AWS (S3, Glue, Lambda), or Google Cloud Platform.
- Orchestration: Experience with workflow tools like Apache Airflow or Databricks Workflows