Key skills
ETLPySparkPythonSparkSQLData EngineeringData Warehousing
About this role
Role Overview
- Design and develop scalable data processing solutions using Spark and PySpark
- Implement robust data models and optimize SQL queries
- Collaborate with data analysts and stakeholders
- Build and maintain ETL pipelines leveraging Python, Hive, and SQL
- Conduct thorough code reviews, performance tuning, and debugging
- Monitor, troubleshoot, and resolve issues in production data workflows
- Document technical processes, data models, and workflow architectures
- Stay updated with industry trends in big data technologies
Requirements
- 2
- 4 years of experience in software development
- Advanced proficiency in Spark and PySpark
- Strong knowledge of SQL (basic and advanced)
- Expertise in Python programming for data processing
- Experience with Hive for data warehousing solutions
- Solid understanding of data modelling fundamentals
- Ability to design and optimize ETL pipelines
- Hands-on experience with large-scale data processing
- Proficient in performance tuning of SQL and Spark jobs
- Familiarity with distributed computing concepts
- Competence in debugging and troubleshooting data workflows
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field
- Relevant certifications in big data technologies or data engineering are advantageous
Tech Stack
- ETL
- PySpark
- Python
- Spark
- SQL