Data Engineer (Remote, US)

Allstate is seeking a Data Engineer for its Risk Data Team. The role builds scalable pipelines and governed, analytics-ready risk data products to improve data reliability and support reporting, decision-making, and future AI capabilities.

Responsibilities:

• Design, build, and maintain trusted risk data products and scalable batch and streaming data pipelines using cloud-native technologies
• Integrate data from risk platforms, control systems, vendor tools, operational applications, and external sources into governed, analytics-ready datasets
• Develop and optimize ETL/ELT workflows that automate data ingestion, transformation, validation, reconciliation, and delivery
• Build and manage data processing workloads within modern data lake and lakehouse environments, including Microsoft Fabric and OneLake
• Implement data quality, monitoring, lineage, and reconciliation processes to ensure data reliability, consistency, and accuracy
• Create curated datasets and analytical outputs that support operational reporting, leadership dashboards, trend analysis, risk identification, and decision-making
• Optimize data architectures, schemas, and processing patterns for scalability, performance, resilience, and cost efficiency
• Develop reusable frameworks, engineering standards, CI/CD pipelines, automated testing, and operational monitoring to improve productivity and maintainability
• Partner with analytics, product, risk, governance, security, and compliance stakeholders to establish data definitions, metrics, quality standards, and secure data access
• Participate in Agile delivery activities including sprint planning, backlog refinement, design reviews, and continuous improvement initiatives

Requirements:

• 4+ years of experience as a Data Engineer or in a similar role building and supporting production-grade data pipelines and data products
• Hands-on experience with Apache Spark for large-scale data processing and transformation
• Strong proficiency in Python, SQL, and modern data engineering best practices
• Experience developing ETL/ELT solutions within cloud-based data lake, lakehouse, or analytics platforms
• Experience designing and optimizing analytical data models that support reporting, dashboards, operational insights, and advanced analytics
• Strong understanding of data quality, validation, monitoring, reconciliation, and production support processes
• Experience with CI/CD pipelines, version control, automated testing, and infrastructure-as-code concepts
• Demonstrated ability to troubleshoot complex data quality, performance, and integration issues across multiple data sources
• Strong verbal and written communication skills with the ability to explain technical concepts to both technical and non-technical audiences
• Experience with Microsoft Fabric, OneLake, or similar modern analytics platforms
• Experience working with risk, controls, compliance, audit, governance, or regulatory data domains
• Experience building real-time, near real-time, or event-driven data processing solutions
• Familiarity with data governance, metadata management, data lineage, access controls, and enterprise data quality frameworks
• Experience with orchestration and workflow management tools for data pipelines
• Experience supporting operational reporting, executive dashboards, trend analysis, advanced analytics, or machine learning initiatives
• Interest in building governed, scalable datasets that support AI and data-driven products
• Apache Spark, Data Engineering, Data ETL, Python (Programming Language), Software Development, Structured Query Language (SQL)
• Experience with Microsoft Fabric, OneLake, or similar modern analytics platforms preferred