Product Manager, Toast IQ Evaluation

Toast builds technology for restaurants and local businesses. The Product Manager will evaluate Toast IQ’s performance for restaurant operators by reviewing usage, creating datasets and quality standards, prioritizing failures, and helping teams improve the assistant.

Responsibilities:

• Understand the operator’s experience. Review real Toast IQ sessions every week. Describe what went wrong in plain operator language. Work directly with operators, including in live sessions, to check that our evaluations reflect what matters to them
• Find and prioritize recurring failures. Group observations into failure modes, measure how often they occur, and assess their impact on operators. Use that evidence to identify the problems most worth solving
• Build trustworthy evaluation datasets. Create and maintain labeled examples covering common workflows, edge cases, and known failures
• Verify that improvements hold up. Validate fixes in real sessions and turn important production failures into regression cases that catch the same problems in future releases
• Help teams across Toast evaluate their work. Lead reviewer calibration sessions and help product teams create and maintain evaluation cases for their capabilities
• Turn findings into product decisions. Surface tools that aren’t delivering value, intents Toast IQ can’t handle, and workflows that need a product team’s attention. Make the operator’s problem and its impact clear enough to act on
• Build tools that make the work better. Use AI coding tools to create faster review interfaces, automate recurring analysis, monitor common failures, and prototype fixes. Look for opportunities to improve every step from session review through validation

Requirements:

• You've worked in a restaurant or closely with operators
• You understand how decisions about menus, labor, sales, and inventory play out during a busy Friday night
• You're comfortable reading hundreds of transcripts, labeling examples by hand, and keeping datasets accurate and current
• You can investigate data using a database, queries, or an AI coding tool and check that the results are trustworthy
• You can turn an ambiguous observation into a clear failure definition or scoring rule that someone else can apply consistently
• You can explain what happened, why it matters to operators, and what should change in language that engineers, product teams, and leadership can use
• You want to understand how large language model systems fail and are willing to learn new evaluation methods on the job
• Product management experience is highly valued, but it is not required
• Familiarity with Toast's products