Key skills
PythonJavaScalaSQLNoSQLShellETLData LakeBIPower BISparkKafkaHadoopAWSTerraformJenkinsCloudFormationLambdaS3DynamoDBRDSCloudWatchGlueAthenaPostgreSQLMongoDBOracleSQL ServerSplunkGitAgileScrumCI/CDCollaboration
About this role
EXL is a consulting company seeking a Senior Big Data Engineer to design, develop, and maintain high-performance, cloud-native data solutions and ETL pipelines. The role involves collaborating with cross-functional teams to deliver secure, high-quality data products for the insurance and BFSI business domains.
Responsibilities:
- Architect and implement modern data lake and data warehouse solutions using AWS S3, Glue, and Athena
- Develop event-driven and batch data pipelines with Kafka, AWS Lambda, and PySpark
- Manage CI/CD workflows using Jenkins, Git, and Terraform
- Automate data ingestion, transformation, and provisioning processes, ensuring data quality, lineage, and governance across environments
- Collaborate with cross-functional stakeholders to deliver secure, high-quality data products for insurance and BFSI business domains
- Contribute to enterprise-level projects, driving innovation and continuous improvement in data platform capabilities
- Position may work at various and unanticipated worksites throughout the United States. Telecommuting permitted
Requirements:
- Master's degree in information technology, Computer Science, Software Engineering, or related IT field
- Six (6) years of Related professional experience
- Six (6) years of experience in designing and developing data pipelines using Spark (PySpark/Scala), Hadoop, Hive, and Kafka
- Scripting and programming with Python, Shell Scripting, and Java
- Five (5) years of experience with data warehouse and data lake architectures, including data modeling, ETL, and data governance
- Five (5) years of experience with DevOps tools, including Terraform, Jenkins, and CI/CD pipelines
- Five (5) years of experience with SQL and NoSQL databases, including PostgreSQL, Oracle, SQL Server, MongoDB, and DynamoDB
- Four (4) years of experience in monitoring and observability using Splunk, CloudWatch, or similar tools
- Three (3) years of experience in data visualization and reporting tools such as Power BI and Microsoft Fabric
- Two (2) years of experience with Agile methodologies, Scrum processes, and collaboration in cross-functional teams
- One (1) year of experience in AWS Cloud technologies including Glue, Lambda, Step Functions, Athena, CloudFormation, S3, RDS, and DynamoDB