Validate data pipelines for accuracy, completeness, consistency, and timeliness
Build SQL-based validations for business rules and transformations
Implement reconciliation between source and downstream systems
Ensure data lineage and traceability
Test pipelines built on AWS (Glue, Lambda, EMR, Step Functions)
Validate transformations using SQL and Python
Test ingestion, transformation, aggregation, and serving layers
Handle backfills, reprocessing, and historical data loads
Validate Spark pipelines (PySpark/Scala) on Databricks Streaming (Kafka)
Validate data integrity, ordering, and delivery guarantees
Test producer and consumer logic and serialization formats (Avro, JSON, Protobuf)
Validate topics, partitions, offsets, retention, and schema evolution
Simulate late events, duplicates, and failure scenarios
Build Python-based data testing frameworks
Develop reusable validation utilities and synthetic datasets
Integrate data tests into CI/CD pipelines
Enable automated alerts for data quality issues
Validate throughput, latency, and concurrency at scale
Test retry logic, idempotency, and recovery mechanisms
Perform regression, soak, and failover testing
Validate logs, metrics, and alerts using tools such as CloudWatch, Prometheus, and Grafana
Define and monitor data SLAs and SLOs
Support incident response, root cause analysis, and postmortems

7+ years of total experience in QA, SDET, or Data Quality Engineering
Minimum 4–6 years of hands-on experience working with data platforms, data pipelines, or data engineering ecosystems
3+ years of hands-on experience with Databricks and Apache Spark
Strong SQL skills for data validation, reconciliation, and complex analysis
Proficiency in Python for automation and data validation
Experience testing ETL/ELT pipelines (batch and streaming)
Hands-on experience with Kafka or similar streaming platforms
Strong understanding of AWS data services (S3, Glue, Lambda, Redshift, Athena)
Experience working with large-scale distributed data systems
Strong debugging, analytical, and problem-solving skills

Data Quality Engineer

Key skills