Lead the AI Evaluation team, owning staffing, coaching, performance management, and delivery of evaluation and testing frameworks
Manage the AI evaluation lifecycle — including pre-launch testing, simulation, and post-deployment health monitoring
Create domain-specific evaluation tracks (e.g., Compliance & Risk, Bot Experience, Agent Experience)
Operationalize human-in-the-loop testing, integrating reviewer feedback into continuous improvement loops
Oversee simulation environments for stress-testing LLMs and identifying hallucinations or performance regressions
Partner closely with AI Platform & Governance to implement evaluation metrics, reporting, and health signals
Develop dashboards and reporting frameworks to track evaluation coverage, accuracy, and confidence scores
Collaborate with Enablement, Speech Analytics, and Data Operations to ensure AI evaluation results inform retraining

7+ years in AI/ML operations, quality, or evaluation
At least 2+ years of people leadership experience
Deep understanding of LLM behavior, prompt testing, and evaluation methodologies
Familiarity with human-in-the-loop frameworks and prompt testing tools
Strong program management and stakeholder communication skills
Technical proficiency in SQL, Python (preferred), or data visualization platforms (Looker, Snowflake)
Experience collaborating with Engineering, Data Science, and Risk/Compliance partners on AI-related initiatives
A passion for operational excellence and responsible innovation

Competitive salary based on experience
401k match
Medical, dental, vision, life, and disability benefits
Generous vacation policy and company-wide Chime Days
Paid parental leave for birthing and non-birthing parents
Annual wellness stipend
Access to family planning tool and reimbursement for fertility treatments and adoption
In-person and virtual events to connect with colleagues

Manager, AI Operations – Evaluation

Key skills