Data Engineer with Data Testing Experience

Quantiphi is an AI-first digital engineering company that delivers digital, cloud, and AI-driven transformation solutions. The company is seeking an experienced Data Engineer to build and maintain scalable data pipelines while performing comprehensive data and ETL testing. The role focuses on data quality, validation, pipeline reliability, and collaboration with engineering, QA, and business teams.

Responsibilities:

• Design, develop, and maintain scalable data pipelines using Python and PySpark
• Develop efficient batch data processing and transformation workflows using Apache Spark/PySpark
• Build data ingestion and transformation processes across structured and semi-structured data sources
• Develop and optimize ETL/ELT pipelines for large volumes of data
• Implement data transformations, cleansing, enrichment, and validation processes
• Optimize Spark jobs for performance, scalability, and resource utilization
• Work with SQL databases, cloud data platforms, data lakes, and data warehouses
• Troubleshoot production data pipelines and resolve data processing issues
• Develop, maintain, and monitor Apache Airflow DAGs for data pipeline orchestration
• Make data and ETL testing a core part of the data engineering lifecycle
• Develop and execute test cases for ETL/ELT pipelines and data transformations
• Validate source-to-target data reconciliation and transformation logic
• Perform data completeness, accuracy, consistency, uniqueness, and integrity checks
• Validate record counts, aggregates, data types, null values, duplicates, and business rules
• Identify data quality issues and work with engineering teams to determine root causes
• Develop automated data validation and testing frameworks using Python/PySpark/SQL where appropriate
• Perform regression testing following pipeline, schema, or transformation changes
• Validate end-to-end data flows across ingestion, transformation, storage, and consumption layers
• Support production validation and post-deployment data quality checks

Requirements:

• 5+ years of experience in Data Engineering or a related field
• Experience with GCP Cloud is must
• Strong hands-on experience with Python and PySpark
• Strong SQL development and data analysis skills
• Experience developing and maintaining ETL/ELT data pipelines
• Hands-on experience with Apache Airflow and DAG development
• Strong understanding of distributed data processing and Apache Spark
• Experience with data validation, reconciliation, and ETL/data testing
• Ability to analyze complex data issues and perform root-cause analysis
• Experience working with relational and/or cloud-based data warehouses
• Familiarity with data lakes, file-based data, APIs, and structured/semi-structured data
• Experience with Git and modern software development practices
• Strong understanding of data quality, data integrity, and pipeline reliability
• Healthcare domain experience
• Experience with HL7, FHIR, healthcare claims, clinical, provider, member/patient, or pharmacy data
• Experience with Snowflake, Databricks, Delta Lake, or similar modern data platforms
• Experience with automated data testing frameworks
• Experience with CI/CD and DevOps practices for data engineering
• Familiarity with data observability and data quality tools
• Experience working in Agile/Scrum environments