Data Engineer

Role Overview

• Write efficient queries to extract and analyze large datasets while ensuring seamless systems integration by developing robust data workflows.
• Design, develop, and maintain scalable ETL solutions for data pipelines, enhancing and optimizing existing processes to meet evolving business needs.
• Develop and expand the organization’s data technology stack to support advanced data processing and analytics needs, leveraging tools like AWS MapReduce and PySpark for big data handling.
• Apply machine learning techniques to clean and process data, supporting future initiatives to integrate machine learning into analytics workflows.
• Continuously evaluate and adopt the most efficient tools and technologies for data extraction, transformation, analysis, and integration.
• Build a strong understanding of customer business needs to deliver tailored and impactful data solutions, collaborating with stakeholders to translate business requirements into technical implementations.
• Utilize AWS services, including MapReduce and related tools, to enable scalable data processing and pipeline optimization while ensuring security, scalability, and cost-efficiency.
• Monitor and improve the performance of data systems and processes, staying updated on industry trends and emerging technologies in data engineering and analytics.

Requirements

• Minimum 3 years of experience with AWS cloud data lake architectures including services like S3, Glue, Athena, and Redshift.
• Hands on experience with Apache Airflow for designing and managing complex data workflows.
• Deep understanding of data warehouse concepts, architectures, and structures.
• Expertise in cloud technologies, particularly AWS services, including AWS Glue, AWS Data Lake, and Amazon EMR for big data processing.
• Proven ability to design and build robust, scalable data pipelines for big data processing and transformation.
• Hands-on experience with big data tools and frameworks such as PySpark, Databricks, and related technologies.
• Proficiency in programming languages such as Python for data manipulation, transformation, and analysis.
• Advanced Python development skills, including experience with AWS SDKs (e.g. boto3) for interacting with cloud services.
• Proficiency in Apache Kafka for real-time data streaming and event-driven architectures.
• Strong SQL skills with a focus on complex query development and optimization.
• Familiarity with NoSQL databases such as DynamoDB or MongoDB.
• Experience integrating Python applications with RESTful APIs and external services.
• Deep understanding of data security best practices, including encryption at rest and in transit.
• Hands-on experience in implementing AWS KMS (Key Management Services) for managing encryption keys.
• Familiarity with IAM policies, VPC configurations, and security groups for securing data pipelines.
• Familiarity with machine learning concepts and their application to data cleaning and integration tasks.
• Advanced English proficiency, with excellent communication skills for collaborating with stakeholders.

Tech Stack

• Airflow
• Amazon Redshift
• Apache
• AWS
• Cloud
• DynamoDB
• ETL
• Kafka
• MapReduce
• MongoDB
• NoSQL
• PySpark
• Python
• SQL

Benefits

• Market-leading Salary
• Medical Coverage – Self & Dependents
• Parents Medical Coverage
• Provident Fund
• Employee Performance-based bonuses
• Home Internet Subsidy
• Conveyance Allowance
• Profit Sharing Plan [Tenured Employees Only]
• Life Benefit
• Child Care Facility
• Company Provided Lunch/Dinner
• Professional Development Budget
• Recreational area for in-house games
• Sporadic On-shore training opportunities
• Friendly work environment
• Leave Encashment