AI Engineer - Data and Evals

Glean is hiring an AI Engineer - Data and Evals to define and measure the quality of its AI products. The role builds evaluation datasets, benchmarks, agentic verifiers, evaluation infrastructure, and enterprise simulations, translating findings into product improvements.

Responsibilities:

• Build evalsets and benchmarks from expert-curated examples, production data, synthetic data, and past failures, covering edge cases and key user slices
• Build and scale the eval platform and agent observability: experiment pipelines, trace capture and replay, and dashboards, with reproducible, affordable runs
• Build agentic LLM verifiers for correctness, groundedness, citation quality, and task success, and calibrate them against human labels using clear rubrics
• Run evals at every stage, from CI and pre-release suites to sampled production traffic, and report results with confidence intervals
• Build enterprise simulation environments with simulated users and realistic company data to test multi-step agent tasks
• Diagnose failures by slice and trace, and turn eval results into concrete fixes, weighing quality against latency and cost