Freelance Agent Evaluation Engineer

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This role involves creating tasks and evaluation criteria for AI coding agents, ensuring that the tasks are challenging and solvable by AI, while also verifying the solutions provided by these agents.

Responsibilities:

• Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
• Design tasks from intermediate states of these environments - craft the prompt, define what 'solved' means, and ensure the task is solvable by an AI agent
• Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
• Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust