Lead Databricks Engineer (Payment)
Atlanta, Texas, United States of America
Contract
2 weeks ago
Key skills
PythonScalaSQLETLELTDatabricksSparkAirflowApache AirflowAWSAzureGoogle CloudTerraformGitHub ActionsLambdaS3GlueAzure DevOpsGitHubPerformance OptimizationCI/CDLeadershipMentoring
About this role
We have an opening for Lead Databricks Engineer (Payment) at Atlanta, GA. Kindly apply with your updated resume if you are interested. Please find the job details below:
Position: Lead Databricks Engineer (Payment)
Location: Atlanta, GA
Duration: 12 Months contract position
Responsibilities
- Architecture & Strategy: Designing scalable Medallion architectures (Bronze, Silver, Gold layers) and defining enterprise data strategies.
- Pipeline Engineering: Leading the development of robust, end-to-end ETL/ELT pipelines for both batch and real-time streaming data.
- Performance Optimization: Tuning Spark workloads, optimizing Delta Lake storage (e.g., Z-Ordering, partitioning), and managing DBU (Databricks Unit) costs.
- Governance & Security: Implementing Unity Catalog for centralized access control, data lineage, and auditability.
- DevOps & Automation: Establishing CI/CD practices using tools like Terraform, Azure DevOps, or GitHub Actions for automated deployments.
- Leadership & Mentorship: Guiding cross-functional teams, performing code reviews, and mentoring junior engineers.
Key Technical Skills
- Databricks Ecosystem: Deep expertise in PySpark, Spark SQL, Delta Lake, and Delta Live Tables (DLT).
- Programming: Mastery of Python or Scala, and advanced proficiency in SQL.
- Cloud Platforms: Strong hands-on experience with at least one major provider Azure (ADF, ADLS), AWS (S3, Glue, Lambda), or Google Cloud Platform.
- Orchestration: Experience with workflow tools like Apache Airflow or Databricks Workflows