Key skills
AirflowApacheAzureETLMongoDBMySQLNumpyPandasPostgresPySparkPythonSQLNumPyELTBIPower BIApache AirflowPostgreSQL
About this role
Role Overview
- Design, develop, and maintain ETL pipelines using PySpark, Apache Airflow, and Azure Data Factory (ADF)
- Build and optimize distributed data processing jobs using PySpark
- Orchestrate and schedule workflows using Apache Airflow
- Develop and manage data ingestion and transformation pipelines in Azure Data Factory
- Write clean, efficient, and reusable code using Python
- Develop and optimize complex SQL queries for MySQL and PostgreSQL databases
- Work with MongoDB for handling semi-structured and unstructured data
- Perform data analysis using Pandas and NumPy to support business insights
- Create basic to intermediate data visualizations using Matplotlib, Power BI, and Streamlit
- Monitor data pipelines, troubleshoot issues, and ensure data quality and performance
- Collaborate with cross-functional teams including analysts, data scientists, and product teams
Requirements
- 3
- 5 years of experience
- Proficiency in Python
- Strong experience with MySQL and PostgreSQL
- Hands-on exposure to MongoDB
- Experience building ETL/ELT pipelines
- Hands-on with PySpark for large-scale data processing
- Experience using Apache Airflow for workflow orchestration
- Experience with Azure Data Factory (ADF)
- Hands-on experience with Pandas and NumPy
- Ability to create visualizations using Matplotlib
- Experience with Power BI for dashboards and reporting
- Exposure to Streamlit for building data-driven applications
Tech Stack
- Airflow
- Apache
- Azure
- ETL
- MongoDB
- MySQL
- Numpy
- Pandas
- Postgres
- PySpark
- Python
- SQL
Benefits
- Flexible work arrangements
- Professional development opportunities