AI Research Engineer

Dscout is hiring an AI Research Engineer for its AI Training & Evals unit. The role develops rigorous methods, rubrics, and long-horizon benchmarks to measure whether AI works for real people, from initial research through experiments and publication, and builds science that can run in live customer programs.

Responsibilities:

• Design rubrics and test whether they measure what they claim: validity, reliability, and sensitivity
• Measure how reliable human raters are, and when disagreement is noise versus real signal
• Calibrate AI judges against real human judgment, and study when they drift away from it
• Build quality signals that put evidence behind every data point we ship
• Design Dscout’s first benchmark: real users, in real context, over long horizons. Multi-week tasks across tools, not one-shot prompts
• Define the protocol, task design, scoring, and release. Work with our delivery team on fieldwork
• Work out how to keep a living benchmark honest over time: drift, contamination, difficulty, and what counts as a meaningful change
• Write the paper and own the public release
• Study when real-world human feedback improves a model and when it adds noise
• Explore how preference and judgment data from representative people differ from expert data
• Turn findings into methods our forward deployed engineers and delivery team can run in the field
• These areas are the core, not the whole list. You will have significant latitude to propose new programs, such as calibrating user simulations against real behavior, human-AI collaboration, and behavioral evaluation of agents
• Co-author with outside labs and academics on studies that need real-user data. Help run our paper-reading group