Senior Software Engineer - Analytics Data Platform Lakehouse

Datadog is the leading observability and security platform for the AI era, providing businesses with unified visibility across the technology stack to manage complexity at scale. The Senior Software Engineer will design and operate core components of the lakehouse platform, driving adoption of open table formats and collaborating with teams to shape analytic data management.

Responsibilities:

• Design, build, and operate core components of our lakehouse platform, including Apache Iceberg table management (data compaction, data layout optimization, materialized view scheduling…) and Iceberg catalog
• Drive adoption of open table formats across internal teams, owning the integration of Trino, Spark and other query engines (DuckDB, Puppygraph…) with our Iceberg-based lakehouse at petabyte scale
• Build observability for managed iceberg tables, to identify query performance bottlenecks, cost drivers and contribute fixes back to upstream open-source projects (Iceberg, Trino, Spark, Open Lineage) where relevant
• Build self-serve tooling and abstractions that allow data engineering teams to reliably run thousands of pipelines per day against our lakehouse
• Collaborate with data engineers, analysts, and infrastructure teams to define the roadmap for our lakehouse architecture and shape how Datadog manages analytic data at scale

Requirements:

• You have a BS/MS/PhD in Computer Science, Engineering, or a related field, or equivalent professional experience
• You have deep, production-grade experience with one or more of Apache Iceberg, Trino, or Apache Spark, ideally demonstrated through significant open-source contributions: merged PRs, committer status, or PMC membership on projects
• You have built or operated large-scale distributed data systems
• You have a solid grasp of query planning, columnar file formats (Parquet, ORC), and table format internals (snapshots, manifests, partition evolution)
• You are fluent in Java, Scala or Go and comfortable with Python for pipeline tooling
• You have experience deploying and running data infrastructure on Kubernetes in cloud environments