EXL is a company focused on providing analytics solutions, and they are seeking a Senior Data Engineer specialized in Databricks. The successful candidate will be responsible for developing and optimizing data pipelines, ensuring data accuracy and integrity, and supporting data-driven decision-making processes in the insurance sector.
Responsibilities:
- Lead the design, development, and optimization of scalable, high-performance data pipelines , ensuring reliable data ingestion and transformation across distributed systems, while proactively identifying and resolving performance bottlenecks and system inefficiencies
- Define and enforce enterprise-grade data quality frameworks , implementing advanced validation rules, anomaly detection, and reconciliation processes to ensure data accuracy, completeness, and integrity across large-scale datasets
- Own end-to-end monitoring and reliability of data platforms , establishing observability practices (alerts, SLAs, dashboards) and driving root-cause analysis to minimize downtime and ensure seamless continuity of critical data workflows
- Govern and continuously improve data pipelines through periodic audits and reviews , ensuring adherence to internal standards, data governance policies, and regulatory compliance, while driving standardization and best practices across teams
- Collaborate with cross-functional stakeholders (architects, analysts, product owners) to align data solutions with business objectives, and actively contribute to backlog prioritization, sprint planning, and technical problem-solving in Agile environments
- Mentor junior engineers and drive technical excellence , promoting reusable frameworks, coding standards, and best practices to improve overall team productivity and solution quality
Requirements:
- 6-10 years experience on Data Engineering role working with Databricks & Azure Cloud technologies
- Bachelor's degree in computer science, Information Technology, or related field
- Strong proficiency in PySpark, Python, SQL
- Strong experience in data modeling, ETL/ELT pipeline development, and automation
- Hands-on experience with performance tuning of data pipelines and workflows
- Proficient in working on Azure cloud components Azure Data Factory, Azure DataBricks, Azure Data Lake etc
- Experience with data modeling, ETL processes, Delta Lake and data warehousing
- Experience on Delta Live Tables, Autoloader & Unity Catalog
- Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy
- Excellent communication and problem-solving skills to work effectively with diverse teams
- Excellent problem-solving skills and ability to work under tight deadlines
- Knowledge of the insurance industry and its data requirements