Databricks Data Engineer

CGI is a global IT and business consulting services firm that delivers technology and business solutions. The Databricks Data Engineer will design, develop, and maintain Azure Databricks pipelines for high-volume credit card data, support platform conversions and modernization, and implement data quality, reconciliation, security, and exception-handling controls.

Responsibilities:

• Design, develop, and maintain Azure Databricks pipelines for credit card data ingestion, transformation, integration, and downstream consumption
• Develop scalable processing solutions using PySpark, Python, Spark SQL, SQL, Delta Lake, and ADLS for high volume card datasets
• Build transformation logic for cardholder, account, transaction, authorization, payment, balance, fee, interest, rewards, and servicing data
• Translate business requirements and source to target mappings into reusable Databricks notebooks, workflows, jobs, and ETL/ELT processes
• Support credit card platform conversion and modernization, including data mapping, historical/incremental loads, transformation, and reconciliation
• Implement automated data quality, balancing, reconciliation, cleansing, and exception handling controls across source and target environments
• Protect sensitive cardholder information, including PAN and PCI sensitive data, through appropriate masking, tokenization, encryption, and data handling practices
• Troubleshoot and optimize Databricks pipelines while collaborating with credit card SMEs, architects, analysts, and application teams

Requirements:

• . 5+ years of Data Engineering, ETL/Data Warehouse, or Big Data experience, including 3+ years of hands on Azure Databricks development
• . Strong hands on development experience with Databricks, PySpark, Python, Spark SQL, SQL, Delta Lake, and ADLS
• . Experience designing and developing scalable ETL/ELT pipelines for large volume transactional and financial datasets
• . Strong experience with data transformation, source to target mapping, cleansing, reconciliation, data quality, and exception handling
• . Credit card/payments domain experience with knowledge of card accounts, transactions, authorizations, payments, balances, billing, and settlement
• . Experience with Azure DevOps, Git, CI/CD, and Agile/Scrum development practices
• . Strong banking/financial services experience and understanding of financial data controls and reconciliation
• . Bachelor's Degree
• . Experience with TSYS or another major credit card processing platform
• . Experience supporting credit card conversions, migrations, or platform modernization
• . Knowledge of the end to end card lifecycle, including authorization, authentication, clearing, settlement, posting, and reconciliation
• . Experience with Mastercard and/or Visa data, interfaces, reporting, rewards, or loyalty programs
• . Understanding of PCI DSS, PAN protection, tokenization/de tokenization, encryption, and sensitive cardholder data management
• . Experience with advanced Databricks capabilities such as Unity Catalog, Workflows, Auto Loader, and Lakeflow/Delta Live Tables
• . Experience modernizing Informatica PowerCenter or SSIS workloads to Databricks/PySpark
• . Azure and/or Databricks certifications and experience with large scale banking conversion initiatives