Senior Data Engineer - Remote

Optum is hiring a Senior Data Engineer to build, optimize, and maintain scalable data architectures and pipelines for healthcare data solutions. The role delivers governed cloud data platforms, supports analytics and AI/ML use cases, and promotes data quality, security, and reliable production operations.

Responsibilities:

• Lead the design and delivery of scalable, governed data engineering solutions, including technical direction, implementation planning, risk management, and production readiness for data products used by business, analytics, and AI/ML stakeholders
• Develop and manage robust ETL/ELT pipelines for data ingestion, transformation, and integration from heterogeneous sources, including relational databases, APIs, file shares, and end-user-computing assets
• Develop Python and PySpark solutions for distributed data processing, automation, integration, and reusable transformation logic
• Optimize data platform performance and cloud consumption through SQL tuning, Spark job optimization, storage format selection, partitioning strategies, cluster configuration, and workload scheduling
• Build and optimize workflows on Azure Cloud services and Databricks - including medallion (bronze/silver/gold) layering and orchestrated pipelines - to support enterprise data needs
• Migrate business logic embedded in legacy and end-user-computing systems into documented, testable transformation logic over governed tables
• Define and implement data quality, validation, reconciliation, and parity testing practices to ensure migrated pipelines are accurate, reliable, auditable, and consistent with the systems they replace
• Establish data lineage, ownership, and cataloguing, and define documented rollup and allocation rules where source systems operate at differing grains
• Implement security and compliance measures for handling PHI/PII and confidential business data in accordance with HIPAA or similar regulations, including de-identification and masking for controlled sharing
• Build and improve CI/CD, version control, and automated testing practices for data pipelines
• Implement monitoring, alerting, logging, and operational runbooks for data pipelines to ensure reliability, timely issue detection, incident response, and continuous improvement of production data platforms
• Support AI/ML and advanced analytical workflows by provisioning trusted data, scalable infrastructure, and governed consumption patterns for data scientists, analysts, and business partners
• Design and publish reusable data products, data services, APIs, and consumption layers that enable downstream applications, analytics platforms, and AI/ML use cases to consume trusted data consistently
• Partner with product owners, business stakeholders, architects, data scientists, analysts, and engineering teams to translate requirements into scalable data solutions and communicate technical tradeoffs clearly
• Design, develop, and deploy AI-enabled data solutions that address complex business challenges, support responsible AI use, and leverage AI-powered development tools to improve engineering productivity and delivery quality
• Mentor data engineers and analysts through design reviews, code reviews, best-practice guidance, and support for improving team delivery quality

Requirements:

• 6+ years of overall experience in data engineering, data warehousing, distributed data processing, or enterprise data platform delivery
• 6+ years of experience with SQL, data modeling, and performance tuning for large-scale systems
• 3+ years of experience with ETL/ELT frameworks, pipeline orchestration, and heterogeneous source-system integration
• 3+ years of experience with Python for data processing, automation, and integration
• 3+ years of experience with Databricks, Spark/PySpark, and Delta Lake
• 3+ years of experience with public cloud platforms such as Azure or GCP
• 1+ year of experience in integrating data from heterogeneous source systems
• 1+ year of experience with data quality, validation, reconciliation, observability, and production support practices for large-scale data pipelines
• 1+ year of experience with Git, CI/CD platforms, automated testing frameworks, release management, and deployment practices for production data pipelines
• 1+ year of experience in AI-powered development tools responsibly to improve engineering productivity and delivery quality
• Experience with batch and near-real-time data processing patterns, including event-driven ingestion, streaming pipelines, or comparable technologies where business needs require timely data availability
• Experience with Azure Data Factory or comparable orchestration tooling such as Airflow, dbt, or similar
• Experience migrating legacy or end-user-computing (spreadsheet- or macro-based) business logic into governed data pipelines
• Experience with data governance, lineage, and catalog tooling (Unity Catalog, Microsoft Purview, or similar)
• Demonstrated familiarity with AI/ML frameworks and supporting infrastructure for data science workflows
• Experience working with PHI/PII and restricted client data in compliance with HIPAA or similar regulations, including de-identification and masking
• Healthcare payer, actuarial, pricing, or financial-modelling domain experience