Staff Software Engineer, AI Gateway

Docker is hiring a Staff Software Engineer to set technical direction and build core systems for its AI Gateway. The role focuses on scalable, secure model and tool routing, policy enforcement, observability, and agent evaluation infrastructure.

Responsibilities:

• Set technical direction for core systems powering the AI Gateway 0 the model router, MCP gateway, and control plane - and make sure architectural decisions hold up as usage scales by 10x
• Consistently identify gaps in our infrastructure, process, or tooling before they become blockers, and drive the fix from idea to production
• Take a significant, hands-on role in defining, delivering, and implementing the Gateway's technical strategy, in close partnership with engineering leadership
• Own the hardest, highest-ambiguity problems in the system: dynamic model selection, auto-failover, cost-based routing, and policy enforcement that has to work end-to-end
• Act as the go-to expert for secure credential storage, session summarization, caching, and rate limiting - the capabilities everything else depends on
• Build the infrastructure that lets us, and our customers, evaluate, replay, and backtest agent and LLM behavior with confidence
• Mentor senior and mid-level engineers, and raise the technical bar for the team through code, design review, and example
• Partner with product, design, and leadership to shape not just what the control plane UI does, but where the product goes next

Requirements:

• 8+ years of backend engineering experience, with a track record of owning production-grade systems at significant scale
• Deep specialization in distributed systems, cloud-native infrastructure, and API design - recognized by peers and cross-functional partners as a go-to expert, not just experienced
• Experience building secure, high-throughput services (gateways, proxies, load balancers, policy engines), and a history of making judgment calls other engineers rely on
• Fluency in Go and/or Rust
• A track record of identifying systemic gaps - in architecture, process, or tooling-— and driving the fix across team boundaries, not just within your own
• A strong product mindset, with real experience shaping technical direction rather than only executing against it
• Comfortable operating with wide latitude: given a problem and significant autonomy in how to solve it, including when and who to pull in
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience
• Prior experience with OpenAI, Anthropic, or similar LLM APIs
• Familiarity with RAG architectures, vector databases, or agent frameworks (e.g., LangChain, AutoGPT, CrewAI)
• Experience with policy engines (e.g., OPA), observability frameworks (e.g., OpenTelemetry), or API gateways
• Understanding of OAuth 2.1, secret management, and cloud IAM systems
• Experience with Kubernetes, Docker, and microservices architecture
• A track record of setting technical standards that other teams or engineers went on to adopt