Senior AI Product Engineer

Oyster is hiring a Senior AI Product Engineer to improve Oyster AI, its assistant for customers and team members. The role owns AI product work end to end, including agents, knowledge retrieval, evaluation, production reliability, and supporting other product teams with reusable AI patterns.

Responsibilities:

• End-to-end AI improvements
• Connect user needs to architecture, implementation, evaluation and safe rollout for complex Oyster AI work
• Reliable agents and tools
• Design clear tool contracts, robust error handling and bounded execution, with appropriate human control over consequential actions
• Better knowledge access
• Choose retrieval, embedding, indexing and freshness strategies that improve answers while respecting data access boundaries throughout retrieval and generation
• Evaluation you can trust
• Build evaluation loops with representative datasets, experiments, regression checks and, where useful, automated judges calibrated against human judgment
• Root-cause fixes in production
• Diagnose failures across prompts, retrieval, tools, models and infrastructure, fix the underlying cause and verify the fix with evidence
• Explicit trade-offs
• Choose models and system designs deliberately, and document decisions and rollout criteria
• AI capability across teams
• Support other product teams through technical guidance, code review, pairing and mentoring, so they can deliver AI features independently

Requirements:

• Senior-level engineering judgment and a track record of independently leading complex production work from design through operation and improvement
• Hands‑on experience shipping LLM‑powered product capabilities to production, with clear personal responsibility for implementation and delivery decisions
• Real depth in at least some of the following: retrieval/RAG and indexing, evaluation datasets and experiments, LLM-as-judge, or agents and tool calling
• The ability to tell model, retrieval, tooling and application failures apart, and to validate improvements with evidence
• Strong fundamentals in coding, debugging, testing, API integration and system design, plus willingness to work in Ruby/Rails and React
• Sound judgment about sensitive data, access control, model limitations and operational risk
• Clear communication of trade-offs, effective async collaboration with product and engineering partners, and practical mentoring
• Production experience with Ruby on Rails, especially integrating AI capabilities into an established backend
• Full-stack delivery of conversational or assistant experiences
• Slack or in‑app assistant integrations with clear user and permission context
• Operating retrieval pipelines, agent tools, evaluation infrastructure, or model routing and cost controls at production scale
• Helping multiple teams adopt reusable AI patterns while balancing shared foundations against local needs