Data Engineering with Python

Diverse Lynx is seeking a Data Engineer with strong expertise in Python-based data engineering and CI/CD automation using GitHub Actions. The candidate will play a critical role in building scalable data pipelines, automating deployments, and enabling high-quality data delivery across cloud platforms.

Responsibilities:

• Design, build, and maintain scalable data pipelines using Python
• Develop ETL/ELT workflows for batch and near real-time data processing
• Work with structured and unstructured data across cloud/platform ecosystems
• Implement data transformation, validation, and quality frameworks
• Build and maintain CI/CD pipelines using GitHub Actions to automate: Code builds and validations, Unit/integration testing, Deployment of data pipelines and services
• Implement branching strategies, code reviews, and version control using Git
• Enable automated testing and release management for data workflows
• Integrate CI/CD pipelines with cloud platforms and data services

Requirements:

• strong expertise in Python-based data engineering
• CI/CD automation using GitHub Actions
• building scalable data pipelines
• automating deployments
• enabling high-quality data delivery across cloud platforms
• designing, building, and maintaining scalable data pipelines using Python
• developing ETL/ELT workflows for batch and near real-time data processing
• working with structured and unstructured data across cloud/platform ecosystems
• implementing data transformation, validation, and quality frameworks
• building and maintaining CI/CD pipelines using GitHub Actions
• automating code builds and validations
• unit/integration testing
• deployment of data pipelines and services
• implementing branching strategies, code reviews, and version control using Git
• enabling automated testing and release management for data workflows
• integrating CI/CD pipelines with cloud platforms and data services