Python ETL Engineer

Largeton Group is seeking a Python ETL Engineer to develop and maintain ETL processes using Python. The role involves integrating data from multiple sources, writing processed data into time-series databases, and participating in Agile development cycles.

Responsibilities:

• Develop and maintain ETL processes using Python, focusing on custom data transformations and utilities
• Work with an existing Python ETL framework; copy and extend it as required, demanding deep Python expertise
• Integrate and process data from multiple sources, such as REST APIs and Kafka streams, applying custom transformations (e.g., pagination handling, timestamp normalization, data restructuring)
• Write processed data into time-series databases (currently InfluxDB; subject to change)
• Leverage and possibly enhance existing libraries for source processing and database writing, with significant custom logic between these steps
• Utilize multi-processing techniques within Python for scalable data processing (preferred)
• Validate completed work using Grafana dashboards; strong practical experience with Grafana is required
• Communicate complex technical concepts and custom ETL solutions clearly and effectively to stakeholders
• Participate in Agile development cycles, validating completed stories through Grafana visualizations

Requirements:

• Develop and maintain ETL processes using Python, focusing on custom data transformations and utilities
• Work with an existing Python ETL framework; copy and extend it as required, demanding deep Python expertise
• Integrate and process data from multiple sources, such as REST APIs and Kafka streams, applying custom transformations (e.g., pagination handling, timestamp normalization, data restructuring)
• Write processed data into time-series databases (currently InfluxDB; subject to change)
• Leverage and possibly enhance existing libraries for source processing and database writing, with significant custom logic between these steps
• Validate completed work using Grafana dashboards; strong practical experience with Grafana is required
• Communicate complex technical concepts and custom ETL solutions clearly and effectively to stakeholders
• Participate in Agile development cycles, validating completed stories through Grafana visualizations
• Remote position with online assessments in Python and Grafana as part of the interview process
• Utilize multi-processing techniques within Python for scalable data processing