Key skills
AirflowAWSCloudDockerETLHadoopJavaKafkaKubernetesNoSQLPythonScalaSparkSQLAIMachine LearningMLData EngineeringData WarehousingAnalyticsSnowflakeDatabricksAgileCI/CDMentoringPrototypingCommunicationTime Management
About this role
Role Overview
- Implementing data engineering processes and best-practice reviews in an Agile environment to ensure the delivery of high-quality data products and solutions.
- Designing, developing, and maintaining scalable, reliable, and architecturally sound data pipelines and data platforms.
- Collaborating with senior technical members, data scientists, analysts, and product owners to understand complex business and data requirements and translate them into robust data solutions.
- Building and optimizing batch and real-time data processing systems using modern big data and cloud technologies.
- Collaborating with data science and AI teams to build and optimize data pipelines that support machine learning and AI-driven solutions.
- Identifying opportunities to leverage AI-assisted development, automation, and intelligent data processing to improve engineering efficiency and platform capabilities.
- Taking ownership of complex data-related technical issues to ensure successful project delivery and data reliability.
- Ensuring data quality, governance, security, and compliance across enterprise data platforms.
- Mentoring and guiding junior team members in data engineering practices, coding standards, and platform optimization.
- Monitoring and improving data pipeline performance, scalability, and operational efficiency.
- Keeping up to date with emerging technologies and industry trends in data engineering, cloud computing, analytics platforms, and AI-enabled solutions.
Requirements
- Expertise in programming and data engineering technologies including but not limited to: Python, Scala, SQL, Java, Spark, Kafka, and Airflow.
- Strong understanding of scalable system design, distributed computing, and performance optimisation techniques.
- Strong experience with ETL processes, data ingestion frameworks, and distributed data processing systems.
- Expertise with relational SQL and NoSQL databases, data warehousing concepts, and cloud platforms such as AWS.
- Experience with big data technologies and ecosystem tools such as Spark, Hadoop, Databricks, Snowflake.
- Strong understanding of data modelling, schema design, data architecture, metadata management, and data governance best practices.
- Experience with containerization and orchestration technologies such as Docker and Kubernetes.
- Knowledge of CI/CD pipelines, DevOps practices, and infrastructure-as-code principles for data platforms.
- Familiarity with AI/ML data pipelines, feature engineering workflows, and supporting infrastructure for analytics and machine learning applications.
- Exposure to modern AI-enabled engineering practices, including automation, intelligent monitoring, or AI-assisted development tools, is a plus.
- An understanding of data quality and governance principles is required for reliable AI and analytics solutions.
- Ability to administer successful project delivery and solve complex problems through innovation, prototyping, risk assessment, and solution evaluation.
- Expertise in data optimization, tuning, monitoring, and troubleshooting to ensure high availability and performance of enterprise data systems.
- Strong organizational, project planning, time management, and stakeholder communication skills across multiple functional groups and departments.
- Experience mentoring junior engineers and leading technical discussions and architectural decision-making.
- Bachelor’s or Master’s degree in Computer Science, Information Systems, Data Engineering, or a related field preferred.
Tech Stack
- Airflow
- AWS
- Cloud
- Docker
- ETL
- Hadoop
- Java
- Kafka
- Kubernetes
- NoSQL
- Python
- Scala
- Spark
- SQL
Benefits
- Group Health Insurance Policy (covering self and family)
- Group Life insurance/accident policy
- Generous long-service awards
- New Baby gift
- Subsidised food provided (applies to India
- Bangalore/Chennai)
- Casual Leave, Sick Leave, Privilege Leave, Compassionate Leave, Special Sick Leave, Gazetted Public Holiday and Maternity/Paternity Leave
- Free Transport provided to and from the office (applies to India-Bangalore/Chennai)