Define and evolve accuracy metrics across the full shopping experience (retrieval, ranking, recommendations, outcomes)
Design and run experiments to measure improvements and regressions
Build and maintain evaluation datasets, benchmarks, and scoring frameworks
Improve the LLM judges that power our evaluation pipeline: prompting, calibration, and fine-tuning where it matters
Translate ambiguous product questions into clear, measurable hypotheses and analysis
Partner with ML Engineers to validate model changes and guide iteration
Identify failure modes and edge cases, and drive improvements through data
Make agent performance visible, trusted, and actionable across product and engineering

5+ years in Applied ML, AI Research, or Applied Science (PhD or equivalent depth strongly preferred)
Hands-on experience evaluating modern AI/ML systems: LLMs, agents, ranking, or recommendations
Direct experience with LLM-based systems: judge models, RAG, prompt engineering, fine-tuning, RLHF, or similar
Strong experimentation foundations: A/B testing, causal inference, statistical rigor
Proven ability to operate in ambiguity: defining problems, not just solving pre-defined ones
Clear, structured communication that influences across ML, engineering, and product.

AI Applied Scientist

Key skills