Research Engineer, Benchmarking - Member of Technical Staff

Callosum is an intelligent systems company building infrastructure that unifies heterogeneous compute across the full stack. The Research Engineer will own and build a reproducible benchmarking system for evaluating agentic and algorithmic approaches, using execution-based grading, contamination controls, and stable baselines. The role will lead benchmark publications and use results to guide product decisions, customer engagements, and company-wide quality claims.

Responsibilities:

• Design and build a unified system for evaluating agentic and algorithmic solutions - task success, quality, and robustness across motifs, agent topologies, and decomposition strategies, on workloads that match what customers actually run. Cost per resolved task is an outcome you track, not the object of the exercise
• Mine real commits and traces, run sandboxed execution grading, and build task suites that reflect real agentic work: code search, code edit and repair, repository summarisation, tool use. Self-reported or model-graded success isn't enough on its own
• Enforce controls against contamination, overfitting to benchmarks, and metric gaming, and keep baselines stable over time - any result should be re-runnable to the same number, by us or by a reviewer
• Compare algorithmic and agentic approaches honestly, not models or chips - a motif that adds steps, latency, or cost has to earn it in resolved-task quality, and the system says clearly when it doesn't
• Lead external benchmark co-publications, held to a standard that survives peer and customer review
• Feed results directly into which approach ships, into the proof points behind customer engagements, and review quality claims across the company before they go out