Design, develop, and optimize large-scale data pipelines using PySpark for structured, semi-structured, and unstructured data
Lead the building of ML pipelines for training, validation, and deployment of models in streaming/batch modes
Write high-quality, efficient code that supports data transformation, cleaning, and feature engineering
Collaborate with data scientists, analysts, and stakeholders to understand data requirements and deliver actionable insights
Build and maintain a reusable code base and automation scripts for data processing and model validation
Monitor pipeline performance, troubleshoot issues, and implement improvements to ensure robustness and scalability

7-12 years of experience in data engineering, analytics, or data science roles
Proven expertise in Python programming, emphasizing clean, maintainable, and scalable code
Hands-on experience with PySpark in both batch and streaming workflows
Deep knowledge of data manipulation and feature engineering, including Pandas, NumPy, and visualization libraries (matplotlib, seaborn)
Experience with Spark components like Spark SQL, DataFrames, and Spark MLlib
Familiarity with data storage solutions: SQL and NoSQL databases (e.g., Hive, Cassandra)
Knowledge of ETL tools such as Apache Airflow, Jenkins, or GitHub Actions for scheduling and automation
Experience working with cloud environments, especially Azure or AWS for big data processing
Bachelor's or Master's degree in Computer Science, Data Science, Mathematics, or a related field
Relevant certifications in big data, cloud platforms, or analytics (preferred)

PySpark Data Engineer, Big Data, Analytics

Key skills