RSI Evaluation & Agent Training Data Engineer

TELUS Digital AI Data Solutions is sourcing elite AI evaluation experts and benchmark engineers to support recursive self-improvement pipelines for frontier AI models. The role focuses on auditing multi-turn agent execution traces, diagnosing pipeline failures, evaluating benchmark quality, authoring hermetic agent task packages, and generating reproducible training trajectory data.

Responsibilities:

• Audit full multi-turn agent rollout trajectories to diagnose exact root causes of failure (attributing errors to defective task instructions, metric computation bugs, incorrect reference artifacts, environment drift, or data leakage)
• Evaluate benchmark item discrimination across model capability tiers (ensuring tasks avoid 0% floor or 100% ceiling saturation and separate monotonically)
• Inspect visible selfcheck verification loops, reward functions, and hidden evaluation test suites in sandboxed setups
• Author production-grade, hermetic agent task packages containing deterministic Docker setups (Dockerfile + pinned dependencies), explicit task specifications, and automated verifier/pytest suites
• Generate complete trajectory run data (tool calls, terminal logs, CoT reasoning steps, and final artifacts) for RSI training sets
• Validate task package reproducibility locally using digest-pinned base images and strict network/environment isolation

Requirements:

• English — Minimum Full Professional Proficiency (C1/C2 Required)
• Candidates must pass a strict technical gateway. Non-coding prompt engineers, generic QA annotators, or non-technical profiles will be automatically filtered out
• Language Requirement: English Full Professional Proficiency (C1/C2) or Native/Bilingual Proficiency is mandatory for precise technical reporting, bug attribution notes, and documentation
• Proven experience spinning up, configuring, and debugging local Docker container environments
• Daily operational proficiency with agentic coding CLI tools ( Claude Code, OpenAI Codex, Cursor ) and Model Context Protocol ( MCP )
• Fluent in writing and reading Python (pytest, AST parsing, data processing) and Shell/Bash automation scripts
• Candidates must demonstrate deep hands-on expertise in at least one of the following four scenario domains:

1. AI Models, Software & Agents: SWE-bench/Terminal-Bench tasks, multi-agent tool orchestration, repository refactoring, MCP tool servers, API integration.
2. Physical Sciences & Engineering: Computational physics, robotics/VLA simulation pipelines, circuit/embedded control diagnostics, control theory, mathematical reasoning.
3. Optimization Planning & Control: Operations research, linear programming, multi-agent scheduling, supply chain routing, state-space control logic.
4. Finance & Quantitative Research: Quantitative backtesting, risk model validation, time-series forecasting, market data pipeline debugging, financial RAG

• 100% Remote •(United States, Europe, India, and Philippines preferred)•