Data engineer

Diverse Lynx is seeking a Data Engineer to develop, optimize, and maintain ETL processes using Azure Data Factory. The role involves working with large datasets, collaborating with business and technical teams, and implementing best practices for data quality and governance.

Responsibilities:

• Develop, optimize, and maintain ETL processes using Azure Data Factory
• Work with large datasets, ensuring efficient data movement and transformation
• Collaborate with business and technical teams to understand requirements and translate them into scalable ETL solutions
• Troubleshoot data and performance issues, ensuring data accuracy and consistency
• Implement best practices for data quality, governance, security and compliance

Requirements:

• Strong experience with SQL, ADF (Azure Data Factory), SSIS, or similar ETL tools
• Experience working with closed-based ETL solutions in Azure
• Experience with Azure SQL Managed Instance Databases and SQL performance tuning
• Experience with integrating various Azure services to create seamless ETL workflows (Airflow, Blob Storage)
• Strong understanding of CI/CD and SDLC as it relates to development and quality assurance
• Ability to analyze complex data requirements and troubleshoot ETL failures efficiently
• Experience with Agile methodologies and tools (e.g. Rally) and collaborating across functions and teams
• Experience with Powershell for automation and management tasks
• Analytical mindset with a strong ability to interpret data and drive data-informed decisions
• Proactive and innovative mindset, strong problem-solving abilities, and ability to contribute to the design and architecture discussions