Staff Data Engineer, Data Platform (Lakehouse & Streaming)

Duetto is the hospitality industry's leading revenue management platform, and they are seeking a Staff Data Engineer to own the data infrastructure that powers real-time pricing decisions for hotels. The role involves designing and maintaining the data lakehouse, driving data quality and governance, and building near-real-time streaming pipelines.

Responsibilities:

• You'll own the design, performance, and reliability of Duetto's data lakehouse — evolving the Python/PySpark pipeline framework across a bronze → silver → gold architecture on AWS, including Glue jobs, Iceberg MERGE operations, schema evolution, and partitioning strategies
• You'll architect the shift from batch to near-real-time streaming, building SQS-driven stream pipelines with Iceberg sinks and expanding ingestion, normalisation, and analytics layers across the full lakehouse
• You'll drive data quality and governance at scale — extending the Great Expectations framework, leading adoption of data contracts to formalise schemas between producers and consumers, and owning the Athena SQL layer that analysts and product teams depend on
• You'll strengthen observability and reliability through Datadog, Sentry, and Sumo Logic, while optimising Glue job performance — worker sizing, DPU allocation, Spark tuning, and cost management
• You'll build and maintain shared internal Python libraries published to JFrog, and drive improvements to GitHub Actions, Docker-based testing, and CI/CD deployment workflows
• You'll work AI-first every day — using Claude Code and MCP tools in your regular workflow, and contributing to AI-assisted pipeline generation, schema inference, and automated data quality alongside a custom multi-agent system with 17 specialised agents

Requirements:

• 7+ years building production data systems in Python
• Deep expertise in PySpark and distributed data processing — Glue, EMR, or Databricks
• Strong experience with lakehouse architectures: Iceberg, Delta Lake, or Hudi on S3
• Production experience with Airflow or a comparable workflow orchestrator
• Solid AWS production experience across S3, Glue, Athena, Lambda, and SQS
• A track record of improving data quality, governance, and pipeline reliability at scale
• Working knowledge of Java for reading upstream systems
• Experience with Trino or Presto for interactive SQL analytics at scale
• Experience with dbt for data transformation and modelling
• Familiarity with Great Expectations or similar data quality frameworks
• Genuine interest in AI-assisted development and LLM-based tooling
• Familiarity with hospitality data — reservations, rates, inventory, demand signals