Task Development Engineer

METR is a nonprofit research organization focused on assessing AI capabilities and risks. The Task Development Engineer will contribute to the development of high-quality tasks that support the Time Horizons methodology, ensuring evaluations are robust and reliable for policymakers and other stakeholders.

Responsibilities:

• Developing difficult, novel tasks for models
• Quality assurance for existing tasks
• Baselining and scoring tasks
• Improving task development infrastructure

Requirements:

• Several years of experience working on complex projects and codebases
• Experience building hard (ideally agent-based) AI evaluations (e.g. RE-Bench, HCAST, SWE-bench Verified, Cybench, GPQA), ideally using the Inspect framework
• High attention to detail: You read closely, spot misspecifications and ambiguity, and pay attention to fiddly minutiae
• Familiarity with METR infrastructure: Prior experience with Hawk, and familiarity with the methodology behind our Time Horizons work, is a plus