Databricks Data Engineer

Capgemini is seeking a Databricks Engineer to lead the design and implementation of a scalable Sales Data Platform as part of the OneData initiative on Databricks running on AWS. The role involves hands-on architecture-driven tasks focused on data ingestion, transformation, modeling, and optimization using modern lakehouse patterns.

Responsibilities:

• Define end-to-end lakehouse architecture on Databricks (AWS) for Sales data domains
• Design medallion architecture (Bronze / Silver / Gold) aligned with OneData standards
• Establish data modeling standards for Sales facts, dimensions, hierarchies, and aggregations
• Define scalable ingestion patterns for batch and incremental loads
• Drive performance, scalability, and cost optimization best practices
• Build and guide development of PySpark-based data pipelines in Databricks
• Implement Delta Lake features:
• ACID transactions
• Schema evolution & enforcement
• Time travel & versioning
• Design and optimize large-scale joins, aggregations, and window functions
• Implement CDC and incremental processing using watermarking and change detection
• Ensure idempotent, restartable, and fault-tolerant pipelines
• Architect solutions using AWS services:
• Amazon S3 (data lake storage)
• IAM (security & access control)
• AWS Glue / Glue Catalog
• CloudWatch (monitoring & logging)
• Optimize Databricks cluster configurations (job vs all-purpose clusters)
• Implement secrets management and secure connectivity patterns
• Define and implement data quality checks and validations
• Ensure data lineage and metadata capture
• Implement error handling, auditing, and reconciliation frameworks
• Support data governance and access control requirements
• Implement CI/CD pipelines for Databricks notebooks and jobs
• Enforce code versioning, reviews, and deployment standards
• Design monitoring, alerting, and SLA tracking for pipelines
• Support production stabilization and performance tuning
• Act as technical lead / mentor for Databricks data engineers
• Collaborate with:
• Source system teams (Sales, CRM, ERP)
• Data consumers (analytics, downstream apps)
• Cloud/platform teams
• Translate business requirements into robust technical designs

Requirements:

• 8+ years in Data Engineering / Data Architecture roles
• 4+ years hands-on experience with Databricks
• Strong expertise in PySpark & Spark SQL
• Strong expertise in dbt
• Deep experience with Delta Lake
• Strong knowledge of AWS cloud services (S3, IAM, Glue, CloudWatch)
• Sales data domain experience (orders, revenue, pricing, customers, products)
• Strong understanding of Fact & dimension modeling
• Strong understanding of Slowly Changing Dimensions (SCD Type 1 / 2)
• Strong understanding of Large-scale data processing patterns
• Experience handling high-volume, high-velocity datasets
• Databricks job orchestration and scheduling
• Cluster sizing and performance tuning
• CI/CD for data platforms
• Strong troubleshooting and debugging skills
• Experience with enterprise OneData / Data Mesh programs
• Exposure to real-time or near-real-time ingestion patterns
• Experience integrating CRM / Sales systems (e.g., Salesforce, SAP Sales data)
• AWS certifications or Databricks certifications