AI Quality Assurance Engineer

ThirdEye Data is seeking an AI Quality Assurance Engineer to own the evaluation process for its AI agent systems. The role involves building evaluation suites and infrastructure, defining quality metrics and release criteria, analyzing failures, and ensuring evaluation results reliably predict production behavior.

Responsibilities:

• Build and maintain evaluation suites for the full agent stack
• Test prompt behavior, tool selection, tool calls, multi-step plans, memory use, error recovery, and task results
• Define quality metrics for each capability. Examples: task success rate, trajectory correctness, tool-call accuracy, latency, and cost
• Build and maintain golden datasets, regression suites, and adversarial test sets. Put these datasets under version control
• Build automated evaluation pipelines. Connect them to CI/CD
• Make sure each change to a model, prompt, tool, or the harness must pass evaluation before release
• Use more than one grading method: rule-based checks, exact match, semantic match, human review, and LLM-as-judge
• Calibrate LLM judges against human labels. Monitor the judges for drift
• Add tracing and logs to agent runs. Record each step, each tool call, and each cost. Make failures easy to diagnose
• Define the KPI set for agent quality. Include accuracy, faithfulness, answer relevance, context precision, context recall, tool-call correctness, hallucination rate, latency, and cost
• Write standard definitions for each metric. Write standard rubrics and report formats
• Make sure teams can compare results across models, prompts, and releases
• Extend evaluation to more than one modality: text, image, audio, and structured documents. Select the correct grading method for each modality
• Do structured error analysis. Group failures by type
• Find the source of each failure: the model, the harness, or a tool
• Set sample sizes, pass thresholds, and confidence bounds. Agent runs are not deterministic. Plan for variance
• Work with engineers to reproduce failures and to verify fixes
• Find and prevent test contamination, overfitting to benchmarks, and metric gaming
• Keep a mix of offline evaluations, staged evaluations, and production monitoring
• Make sure evaluation results predict production behavior
• Define release criteria for agent changes. Approve or block releases with data
• Write clear documentation for the evaluation methods
• Report results to engineering and to leadership

Requirements:

• 4 or more years of hands-on work in evaluation of LLM systems and agents
• Hands-on skill with RAGAs, Promptfoo, and DeepEval
• Proof that you can create KPIs and metrics from zero: select the metrics, set the baselines, set the thresholds, and get agreement from stakeholders
• Proof that you can standardize evaluation across teams: shared metric definitions, reusable templates, versioned datasets, and one report format
• Experience with evaluation in more than one modality: text plus image, audio, video, or structured documents
• Knowledge of the agent evaluation problem space: trajectory evaluation, outcome evaluation, tool-use correctness, multi-turn behavior, non-determinism, and prompt sensitivity
• Experience with LLM-as-judge design: rubric design, calibration against human labels, and bias control
• Knowledge of statistics for evaluation: sampling, significance tests, inter-rater agreement, and variance across runs
• Strong Python skills. Experience with test harnesses, data pipelines, or evaluation tools that you built
• Skill with tracing and observability for agent systems. Example: OpenTelemetry-style traces and structured logs of tool calls
• Knowledge of QA fundamentals: test plans, test-case design, functional tests, regression tests, integration tests, and defect management
• Hands-on skill with test automation frameworks. Examples: pytest, Playwright, Selenium
• Hands-on skill with API tests. Examples: Postman, REST-assured
• Experience with test suites in CI/CD pipelines. Examples: GitHub Actions, Jenkins, GitLab CI
• Skill with bug tracking and test management tools. Examples: Jira, TestRail, Xray
• Ability to write clear defect reports that engineers can reproduce
• Knowledge of non-functional tests: performance, load, and reliability
• Skill with Braintrust, LangSmith, W&B Weave, or OpenAI Evals is a plus
• Experience with red-teaming, safety evaluations, or adversarial tests of AI systems
• Experience with human annotation work: labeling guidelines, calibration sessions, and inter-annotator agreement
• Background in CI/CD, infrastructure-as-code, or platform engineering
• Contributions to open-source evaluation tools. Published work on evaluation methods