Data Engineer / Architect – Databricks & dbt

Capgemini is seeking an experienced Data Engineer/Architect to support enterprise data pipelines on a modern cloud-based data platform. The role will define scalable Databricks and dbt engineering patterns, develop data pipelines, establish reusable frameworks and standards, and provide technical guidance and mentorship to data engineering teams.

Responsibilities:

• Define the technical architecture and implementation patterns for dbt-based data transformations on Databricks
• Design and develop scalable curated and consumption-layer data pipelines using Databricks and dbt
• Establish dbt project structure, model organization, dependencies, naming conventions, coding standards, and reusable development patterns
• Define and implement appropriate usage of dbt models, sources, tests, macros, snapshots, incremental models, packages, and documentation
• Design efficient transformation patterns leveraging Databricks, Delta Lake, Spark SQL, and PySpark
• Establish reusable and standardized pipeline development patterns for adoption across the engineering team
• Work with architecture teams to ensure implementation aligns with enterprise architecture, governance, security, and data-platform standards
• Provide hands-on development support and technical guidance to Data Engineers
• Review existing data transformation and pipeline patterns and determine the appropriate implementation using Databricks and dbt
• Define data quality, reconciliation, testing, logging, monitoring, and error-handling approaches
• Design and implement appropriate incremental data-processing strategies
• Support performance optimization and troubleshooting of dbt models and Databricks workloads
• Work with DevOps teams to integrate dbt and Databricks development into the enterprise CI/CD framework
• Support orchestration and scheduling integration with Databricks Workflows and enterprise scheduling platforms
• Conduct technical design reviews and code reviews and ensure adherence to defined engineering standards
• Support testing, deployment, production readiness, and troubleshooting activities
• Develop reusable templates, frameworks, utilities, and implementation guidelines to accelerate development
• Prepare technical documentation and provide knowledge transfer to engineering and support teams
• Mentor Data Engineers and provide technical leadership throughout design, development, testing, and deployment

Requirements:

• 8+ years of experience in Data Engineering, Data Architecture, or related areas
• Strong hands-on experience with Databricks
• Strong hands-on experience implementing enterprise data solutions using dbt
• Strong understanding of dbt models, sources, macros, tests, snapshots, incremental models, packages, and documentation
• Strong programming and development skills in SQL, Spark SQL, and PySpark
• Strong experience with Delta Lake and Databricks data-engineering capabilities
• Experience designing and implementing Medallion Architecture / multi-layered data platforms
• Strong understanding of logical and physical data modeling concepts
• Experience building scalable, reusable, and metadata-driven data-engineering solutions
• Experience implementing data-quality, validation, reconciliation, lineage, monitoring, and observability capabilities
• Strong understanding of CI/CD and DevOps practices for dbt and Databricks
• Experience with Git-based source control and automated deployment processes
• Strong understanding of performance optimization and troubleshooting of large-scale data pipelines
• Ability to translate architecture standards and business requirements into practical engineering solutions
• Experience conducting technical design and code reviews
• Ability to mentor and provide technical direction to Data Engineers
• Strong analytical, problem-solving, communication, and stakeholder-management skills
• Databricks certification
• dbt certification or significant production implementation experience
• Experience with AWS-based Databricks environments
• Experience with enterprise data governance, metadata management, lineage, and data-product concepts
• Experience with Control-M or other enterprise workload scheduling/orchestration platforms
• Experience working on large-scale enterprise data-platform transformation programs