Technical Product Manager, Pedagogy & AI Quality

Medley is hiring a Technical Product Manager, Pedagogy & AI Quality to ensure its AI-powered instructional scaffolds meet high standards for quality and usability. The role develops evaluation systems and prompts, analyzes AI outputs, and partners with engineering and educators to prioritize and ship product improvements for multilingual learners and students with diverse needs.

Responsibilities:

• Define what "excellent" looks like. Translate research-based practice in literacy, special education, and multilingual learner instruction into clear, usable quality standards for Medley scaffolds, and set targets for AI output quality
• Track and monitor quality with a robust evals system. Build and maintain datasets, rubrics, and eval suites and conduct regular error analysis and trace review, combining human labeling, LLM-as-judge and rule-based evals
• Refine prompts and models as needed. Write, test, and refine prompts; run model comparisons and regression tests, making changes as needed to ensure outputs are consistently meeting target metrics and passing quality thresholds
• Stay close to our users and represent their voice in product decisions. Regularly review signals from educators, data from support tickets, product analytics, and conduct user research with students and teachers to inform product decisions
• Prioritize and ship AI quality and scaffold-related product improvements. Maintain a prioritized backlog of AI quality and scaffold UX issues; write clear specs, and work with engineering to scope, build and verify fixes, product updates and new features

Requirements:

• Pedagogical expertise, especially in scaffolding literacy instruction for struggling readers, special education, and/or multilingual learner education
• Hands-on experience with AI evals, including human labeling, LLM-as-judge rubric design, AI metrics targets definition, and error analysis of LLM outputs
• AI product fluency: you use LLMs daily and have experience designing complex AI systems for products
• Experience defining new features and collaborating with engineers to deliver them: you can clearly communicate a problem and solution, and work through technical tradeoffs and UX questions
• Strong initiative, sharp judgment, and clear writing: you take action when you spot a problem, make swift decisions grounded in data and instincts, and communicate clearly to both technical and non-technical audiences
• and ideally from classroom teaching, intervention, instructional coaching or curriculum design
• Graduate degree and/or significant direct experience in special education, reading/literacy, ESL/ELD or bilingual education
• Familiarity with AI eval and observability tools (e.g., LangSmith, Braintrust, Arize, Langfuse) or annotation platforms
• Proficiency in a language other than English
• Prior work at an edtech company, or on content quality, assessment design or psychometrics
• Familiarity with accessibility standards (e.g., WCAG) and assistive technology