Applied AI Researcher, Benchmarking

Distyl is an applied AI technology company that partners with ambitious institutions to rearchitect critical operations for the frontier of AI. The Applied AI Researcher, Benchmarking will design evaluation frameworks and benchmarks for intelligent systems, explore new evaluation paradigms, and develop rigorous methodologies for measuring reasoning, reliability, interaction quality, and operational impact.

Responsibilities:

• The Benchmarking team defines how progress is measured. Researchers design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact. They construct benchmarks that reflect real-world complexity. Their systems become the standard by which new architectures, techniques, and releases are judged
• Researchers in Benchmarking explore new paradigms for evaluating intelligent systems: adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment. They investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capability. Their insights drive both Distyl’s internal research priorities and industry-wide standards