Senior Data Engineer - Databricks

EXL is a company focused on providing analytics solutions, and they are seeking a Senior Data Engineer specialized in Databricks. The successful candidate will be responsible for developing and optimizing data pipelines, ensuring data accuracy and integrity, and supporting data-driven decision-making processes in the insurance sector.

Responsibilities:

• Lead the design, development, and optimization of scalable, high-performance data pipelines , ensuring reliable data ingestion and transformation across distributed systems, while proactively identifying and resolving performance bottlenecks and system inefficiencies
• Define and enforce enterprise-grade data quality frameworks , implementing advanced validation rules, anomaly detection, and reconciliation processes to ensure data accuracy, completeness, and integrity across large-scale datasets
• Own end-to-end monitoring and reliability of data platforms , establishing observability practices (alerts, SLAs, dashboards) and driving root-cause analysis to minimize downtime and ensure seamless continuity of critical data workflows
• Govern and continuously improve data pipelines through periodic audits and reviews , ensuring adherence to internal standards, data governance policies, and regulatory compliance, while driving standardization and best practices across teams
• Collaborate with cross-functional stakeholders (architects, analysts, product owners) to align data solutions with business objectives, and actively contribute to backlog prioritization, sprint planning, and technical problem-solving in Agile environments
• Mentor junior engineers and drive technical excellence , promoting reusable frameworks, coding standards, and best practices to improve overall team productivity and solution quality

Requirements:

• 6-10 years experience on Data Engineering role working with Databricks & Azure Cloud technologies
• Bachelor's degree in computer science, Information Technology, or related field
• Strong proficiency in PySpark, Python, SQL
• Strong experience in data modeling, ETL/ELT pipeline development, and automation
• Hands-on experience with performance tuning of data pipelines and workflows
• Proficient in working on Azure cloud components Azure Data Factory, Azure DataBricks, Azure Data Lake etc
• Experience with data modeling, ETL processes, Delta Lake and data warehousing
• Experience on Delta Live Tables, Autoloader & Unity Catalog
• Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy
• Excellent communication and problem-solving skills to work effectively with diverse teams
• Excellent problem-solving skills and ability to work under tight deadlines
• Knowledge of the insurance industry and its data requirements