Forward Deployed Machine Learning Engineer

Protege is a company focused on solving the unmet need in AI by facilitating the exchange of AI training data. They are hiring a Forward Deployed Machine Learning Engineer to work on the Benchmarks and Evaluations vertical, responsible for building the technical foundation for evaluating AI models and improving their performance.

Responsibilities:

• Partner with the GM and early customers to define what constitutes strong evals in different domains
• Work with Protege researchers to design and build benchmarks
• Build the standards on how different modalities should be processed
• Build the backend the vertical runs on which includes data pipelines, execution environments, storage, and orchestration
• Stand up sandboxed environments for agentic evals, where models need tools, code execution, or multi-step tasks
• Find repeatable eval patterns, infrastructure gaps, and product opportunities from live engagements
• Partner with DataLab (our research team) on domain-specific data and research questions
• Build an understanding of the evals landscape, the GM's strategy, and customer demand
• Build an understanding of what our platform and data partners can support today, and where the gap is for eval building
• Identify the largest technical bets and ship multiple iterations of the eval infrastructure
• Own the engineering portion of customer engagements end to end

Requirements:

• 4+ years of engineering experience
• Hands-on ML work evaluating models
• Have previously owned backend and infrastructure
• High ambiguity tolerance and bias to action
• Comfort working with urgency to meet the pace and volume of the market demands
• Strong written communication
• Prior experience building benchmarks, evals, or human data pipelines for LLMs
• Time at a frontier lab, an eval-focused team, or a research org
• Founding or early engineer experience at a fast-moving startup
• Familiarity with agentic systems, RL environments, code-execution sandboxes, TEE/TREs