Key skills
ETLData EngineeringAnalyticsSnowflakeAWSKubernetesEKSS3GlueMongoDBElasticsearchCommunicationCollaboration
About this role
Brillio is one of the fastest growing digital technology service providers, renowned for its world-class professionals. They are seeking a Senior Data Engineer to provide production support for data pipelines and workflows, ensuring high availability and timely resolution of incidents.
Responsibilities:
- Provide production support for data pipelines and workflows, ensuring high availability and timely resolution of incidents
- Monitor, troubleshoot, and optimize PySpark-based data processing jobs
- Support and maintain data processing applications deployed on AWS services, including EKS and EMR
- Manage and orchestrate workflows using Control-M, including scheduling, monitoring, and error handling
- Perform root cause analysis and implement permanent fixes for recurring production issues
- Oversee data storage and management in MongoDB and Snowflake, ensuring data quality and consistency
- Monitor and analyze application and system logs using Elasticsearch Service (ESS) to proactively identify and resolve issues
- Collaborate with data engineering, analytics, and operations teams to deliver reliable data solutions
- Maintain and update technical documentation related to production support processes and incident management
- Participate in on-call rotations and respond to critical production alerts as needed
Requirements:
- 3+ years of hands-on experience in production support for data engineering or analytics applications
- Proficiency in PySpark for managing and troubleshooting data processing jobs
- Solid experience with AWS cloud services, including EKS (Elastic Kubernetes Service) and EMR (Elastic MapReduce), S3, Glue, Step Function
- Hands-on experience with Control-M for workflow orchestration and monitoring
- Working knowledge of MongoDB and Snowflake for production data management
- Experience with log monitoring and analysis using Elasticsearch Service (ESS)
- Good understanding of ETL processes, incident management, and production monitoring best practices
- Strong problem-solving, troubleshooting, and analytical skills
- Excellent communication and collaboration abilities