Key skills
PythonPySparkSpark performance tuningData pipeline designData pipeline developmentData orchestrationData monitoringBatch data processingStreaming data processingSnowflakeDatabricksMedallion ArchitectureAutoscalingCluster optimizationCost-efficient cloud processingCI/CDVersion controlDevOps practicesAnalyticsSparkTerraformVersion ControlCaching
About this role
Qualitest is looking for a skilled and motivated Data Engineer to join their growing team. The ideal candidate will have strong experience in building scalable data pipelines and working with modern cloud-based data platforms.
Responsibilities:
- Design, develop, and maintain scalable data pipelines for batch and streaming data processing
- Work with large datasets using distributed data processing frameworks
- Implement robust data orchestration, monitoring, and optimization strategies
- Collaborate with cross-functional teams including business, analytics, and engineering to deliver reliable data solutions
- Optimize data workflows for performance, scalability, and cost efficiency
- Contribute to CI/CD pipelines and follow DevOps best practices
Requirements:
- Strong hands-on experience with Python and PySpark
- Solid understanding of Spark concepts, including performance tuning (joins, partitioning, caching, optimization)
- Experience with modern data platforms such as Snowflake, Databricks, or similar
- Strong knowledge of data pipeline design, development, orchestration, and monitoring
- Experience with both batch and streaming data processing
- Familiarity with Medallion Architecture (bronze, silver, gold layers)
- Understanding of autoscaling, cluster optimization, and cost-efficient processing in cloud environments
- Experience with CI/CD, version control, and DevOps practices
- Exposure to Terraform or other infrastructure-as-code tools is a plus
- Experience building end-to-end data pipelines (batch & streaming, orchestration, monitoring)
- Hands-on experience with modern cloud data platforms (Databricks, Snowflake)
- Strong PySpark expertise with Spark performance tuning