Data Engineer

Optum is a global organization that delivers care aided by technology to help millions of people live healthier lives. They are seeking a highly skilled Senior Data Engineer to design, build, and optimize scalable data and AI platforms on Azure, focusing on enterprise data pipelines and AI/ML model integration. This role offers the flexibility to telecommute from anywhere within the U.S.

Responsibilities:

• Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on Azure
• Build cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta Lake
• Collaborate with Data Science and AI teams to operationalize ML models and embed them into production workflows
• Develop and maintain feature stores, model input pipelines, and real-time/streaming frameworks
• Ensure data quality, governance, and security across the full data lifecycle
• Build reusable frameworks, accelerators, and automation scripts to improve engineering efficiency
• Optimize performance, scalability, and reliability of data workflows and batch/streaming pipelines
• Participate in Agile development processes, including sprint planning, code reviews, and CI/CD pipelines
• Provide production support and on-call coverage, ensuring system stability and rapid issue resolution
• Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI

Requirements:

• Bachelor's degree in Computer Science, Engineering, or IT related field
• 6+ years of experience in Data Engineering with Python/PySpark
• 6+ years of experience in building ETL/ELT pipelines using Databricks
• 6+ years of experience working in Agile environments
• 5+ years of strong experience in SQL / PL-SQL
• 4+ years of experience with Azure Databricks and Delta Lake architecture
• 4+ years of hands-on experience with CI/CD (GitHub Actions, Azure DevOps)
• 3+ years of hands-on experience with Azure cloud services (ADF, ADLS, Databricks)
• 2+ years of experience with Databricks Delta Live Tables (DLT)
• 2+ years of experience with unit testing, validation, and pipeline testing frameworks
• Familiarity with medallion architecture and SCD2 implementations
• AI builder: Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI
• Experience building enterprise-scale data platforms
• Strong skills in performance tuning and debugging large-scale pipelines
• Experience with real-time/streaming frameworks (Structured Streaming)
• Ability to work in distributed, cross-functional global teams
• Exposure to GenAI tools (e.g., GitHub Copilot) for engineering productivity
• Strong understanding of secure coding practices and vulnerability remediation
• Proven ability to analyze logs, troubleshoot production issues, and optimize performance
• Demonstrated capability to design and deploy AI-powered solutions responsibly