Senior Data Platform Engineer

Komodo Health builds healthcare data products, including its Healthcare Map. The Senior Data Platform Engineer will build and improve large-scale data pipelines and foundational systems that transform complex healthcare datasets into reliable, performant data products.

Responsibilities:

• Build, operate, and optimize large-scale batch pipelines using Python, SQL, Spark, and Airflow on AWS
• Make storage and data layout decisions (table formats, partitioning, file sizing, compaction) that keep large datasets fast to query and cost-efficient
• Ingest data from many external sources and keep pipelines correct through frequent schema changes
• Transform healthcare claims, EHR, and reference data into trusted, serving-ready data products for customers and internal teams
• Strengthen reliability through data quality checks, validation, lineage, observability, and alerting
• Debug complex data, system, and performance issues across compute-heavy workflows
• Contribute to system design, code quality, testing, documentation, CI/CD, and rotational production support

Requirements:

• Strong hands-on experience building, owning, and debugging production data pipelines at multi-terabyte scale
• Advanced Python, including performant, memory-efficient code for large-scale data processing, plus strong SQL
• Deep production experience with Airflow, including operating and scaling it, not just writing DAGs
• A solid understanding of how data is physically stored: columnar vs. row formats, and open table formats like Iceberg, Delta Lake, or Hudi (updates, compaction, small files, partitioning)
• Experience with Spark or a comparable distributed processing framework
• Proven experience designing and operating data systems in AWS
• Strong instincts for data correctness, root-cause analysis, and production troubleshooting
• The ability to communicate technical trade-offs clearly with engineering, product, and data partners
• Comfort using AI-assisted engineering tools for coding, debugging, documentation, and technical exploration
• You will be expected to leverage AI-augmented engineering tools, such as ChatGPT, Gemini, or Claude, to improve productivity and technical decision-making
• No healthcare background required
• Containers and orchestration (Docker, Kubernetes)
• Strong core software engineering fundamentals and experience in more than one language
• Data governance and lineage experience
• Experience delivering external-facing data products through APIs, serving layers, or customer access patterns
• Experience applying AI or agentic workflows to engineering, data quality, delivery, or operations
• Success in high-growth or ambiguous environments