Member of Technical Staff - Pre-Training Evals Infra

Reflection is an AI research lab building open models. The Member of Technical Staff will build pre-training evaluation infrastructure, including evaluation authoring, distributed execution, performance optimization, correctness and reproducibility, researcher-facing tools, and operational monitoring.

Responsibilities:

• Design and evolve evaluation abstractions. Build clear APIs and data models for evaluation tasks, datasets, configurations, model outputs, and scores, making it straightforward to add new evaluations and extend existing workflows
• Build scalable, reliable execution infrastructure. Develop distributed systems for evaluating model checkpoints throughout training and ahead of release, with robust orchestration, observability, and recovery from failures
• Optimize evaluation and inference performance. Profile inference servers and distributed execution to improve throughput, GPU utilization, and time to results
• Establish evaluation correctness and reproducibility. Build validation and regression tests, version datasets and configurations, and preserve the provenance needed to reproduce results and investigate discrepancies
• Build researcher-facing tools. Create intuitive interfaces and dashboards for inspecting model outputs, comparing checkpoints and experiments, and debugging unexpected behavior
• Operate and monitor evaluations as training progresses. Track evaluation workloads and results, triage failures, and develop alerts and diagnostics that help distinguish changes in model quality from inference or evaluation issues
• Turn research needs into reusable systems. Partner directly with researchers to identify bottlenecks, prioritize high-impact improvements, and turn bespoke workflows into self-service tools that work across teams