Key skills
PySparkAWSData ModellingSQLControl-MMongoDBSnowflakeProblem-solvingCommunicationCollaborationETLData EngineeringAnalyticsKubernetesEKSS3GlueElasticsearch
About this role
Brillio is a fast-growing digital technology service provider that partners with Fortune 1000 companies to enhance their digital adoption. The Senior Data Engineer will be responsible for providing production support for data pipelines, optimizing data processing jobs, and collaborating with various teams to deliver reliable data solutions.
Responsibilities:
- Provide production support for data pipelines and workflows, ensuring high availability and timely resolution of incidents
- Monitor, troubleshoot, and optimize PySpark-based data processing jobs
- Support and maintain data processing applications deployed on AWS services, including EKS and EMR
- Manage and orchestrate workflows using Control-M, including scheduling, monitoring, and error handling
- Perform root cause analysis and implement permanent fixes for recurring production issues
- Oversee data storage and management in MongoDB and Snowflake, ensuring data quality and consistency
- Monitor and analyze application and system logs using Elasticsearch Service (ESS) to proactively identify and resolve issues
- Collaborate with data engineering, analytics, and operations teams to deliver reliable data solutions
- Maintain and update technical documentation related to production support processes and incident management
- Participate in on-call rotations and respond to critical production alerts as needed
Requirements:
- 3+ years of hands-on experience in production support for data engineering or analytics applications
- Proficiency in PySpark for managing and troubleshooting data processing jobs
- Solid experience with AWS cloud services, including EKS (Elastic Kubernetes Service) and EMR (Elastic MapReduce), S3, Glue, Step Function
- Hands-on experience with Control-M for workflow orchestration and monitoring
- Working knowledge of MongoDB and Snowflake for production data management
- Experience with log monitoring and analysis using Elasticsearch Service (ESS)
- Good understanding of ETL processes, incident management, and production monitoring best practices
- Strong problem-solving, troubleshooting, and analytical skills
- Excellent communication and collaboration abilities