Human Evaluation – Program Manager

Role Overview

• Lead end-to-end execution of human evaluation and data operations initiatives—from intake and scoping to delivery
• Develop and operationalize frameworks for evaluating GenAI and ML outputs
• Collaborate across research, product, UX, and engineering to embed evaluation into model development cycles
• Build and maintain project timelines, proactively manage blockers, and ensure timely execution
• Develop clear, scalable guidelines and scoring rubrics to ensure consistent rater judgment
• Oversee rater onboarding, calibration, and QA workflows
• Define and monitor success metrics such as speed to IRR, throughput, and task effectiveness
• Pilot and refine evaluation tasks to improve clarity, inter-rater reliability, and feedback quality
• Build foundational documentation and drive adoption of best practices across teams
• Track evaluation health and proactively communicate progress to stakeholders clearly and proactively
• Anticipate and proactively resolve bottlenecks and blockers
• Act as the connective tissue across multiple partners to ensure alignment and effective execution of evaluations at scale

Requirements

• 4+ years of experience working in human evaluations, data collection, labeling, or annotation operations in GenAI/ML environments
• Track record of implementing process improvements or quality control systems for data collection needs
• Prior experience managing human annotation vendors, raters, or data labeling teams
• Strong understanding of evaluation design, including guidelines, rubrics, and scoring protocols
• Proven ability in end-to-end management of complex, cross-functional programs, demonstrating strong Program Management skills and clear accountability for successful delivery
• Experience with human labeling platforms
• Excellent written and verbal communication skills
• Ability to synthesize feedback into clear recommendations and process improvements
• Familiarity with responsible AI principles and how to embed them into evaluation design
• Strong organizational skills and executional focus; ability to track details while seeing the bigger picture

Benefits

• Health Plans
• Mental Health support
• 401(k) Retirement Plan with employer match
• Stock Option Program
• Disability Programs
• Health Savings and Flexible Spending Accounts
• Family-forming benefits
• Life and Serious Injury Benefits
• Paid leave of absence programs
• 35 days annually for paid time off for full-time hourly employees (vacation, holidays, sick)