Senior Databricks Engineer

EXL is seeking a high-caliber Senior Databricks Engineer to lead the architecture, development, and optimization of their next-generation Lakehouse platform. The role involves driving technical direction, establishing robust data governance, and delivering scalable data solutions that bridge the gap between raw data and actionable business intelligence.

Responsibilities:

• Design and optimize high-volume ETL/ELT pipelines using Delta Live Tables (DLT) and PySpark, ensuring data integrity across the Bronze, Silver, and Gold layers
• Develop and maintain sophisticated pipelines using Databricks Workflows or Airflow, focusing on modularity, reusability, and automated error handling
• Implement real-time data flows utilizing Structured Streaming and Kafka/Event Hubs to enable immediate data availability for downstream consumption
• Enforce data anonymization and fine-grained access controls to ensure compliance with global regulations (GDPR/CCPA/HIPAA)
• Implement CI/CD patterns using Databricks Asset Bundles (DABs), Terraform, and Git to automate environment parity and deployments
• Manage and optimize Delta Lake storage, utilizing advanced features like Liquid Clustering, Z-Ordering, and Change Data Feed (CDF)
• Drive cost efficiency and performance by optimizing Spark configurations, Photon engine utilization, and Serverless SQL Warehouses
• Integrate comprehensive monitoring and alerting (e.g., Databricks System Tables, Grafana, or Splunk) to rapidly identify bottlenecks and troubleshoot production issues

Requirements:

• 6+ Years of hands-on, progressive experience in Data Engineering, with at least 5 years focused heavily on the Databricks platform
• Expert knowledge of Medallion Architecture, Data Vault 2.0 or Dimensional Modeling, and modern Lakehouse design patterns
• Proven track record of building and managing large-scale data infrastructure (Petabyte-scale) in cloud-native environments
• Technical Toolset: Cloud Environment: Azure (preferred), AWS
• Technical Toolset: Databricks Stack: Unity Catalog, Delta Live Tables, Databricks SQL, MLflow
• Technical Toolset: Core Languages: Expert-level SQL, Python, and PySpark
• Technical Toolset: Supporting Tools: dbt (Databricks adapter), Git, and Orchestration tools
• Experience in the Insurance or Financial Services industry is preferred (focusing on claims, policy, or risk data)