Product Manager, AI Evaluation Lead

Trimble Inc. is a global technology company that connects the physical and digital worlds, transforming the ways work gets done. They are seeking a Product Manager, AI Evaluation Lead to own and scale the AI evaluation program, design methodologies, and coordinate a network of subject matter experts to ensure high-quality AI performance.

Responsibilities:

• Own AI evaluation end to end as a dedicated program, with the mandate to create a defined, repeatable, and scalable system ("Eval Ops")
• Design the evaluation methodology — test sets and benchmarks spanning the dimensions that matter for our product (accuracy/factuality, instruction-following, robustness, multi-turn conversation behavior, and domain-specific edge cases), plus regression testing to catch when model or prompt changes break previously working behavior
• Build canonical evaluation data sets — reference projects spanning varied size, complexity, document volume, and project phase (preconstruction, execution, closeout), each with proportionate question/query banks
• Define rubrics and own ground truth — author clear, anchored scoring rubrics and calibration sets, and stand up the human-validation workflow that confirms whether answers are actually correct and keeps reviewers aligned
• Stand up a tiered, scalable eval pipeline — combine automated evaluation (e.g., LLM-as-a-judge) for fast iteration with expert human review for calibration and ground truth, so scarce SME time is spent where it matters most
• Design the eval workbench and contributor workflow — define the end-to-end tooling and process SMEs use to contribute (building on our existing internal eval tooling), so the system runs efficiently at scale
• Build and coordinate a flexible SME network — recruit, vet, onboard, schedule, and quality-control subject matter experts: Trimble SMEs where available, Document Crunch SMEs and customers where appropriate, and external/contract experts as needed
• Design the incentive model for the network — determine what motivates busy experts (especially Trimble SMEs) to lend scarce time, and build the structure that attracts and retains them
• Partner with Product and Engineering/QA — work with product managers to understand the features being built and the evals they require, and partner with engineering (including our existing eval tooling) to operationalize the process
• Establish a customer feedback loop — over time, incorporate real customer queries and chat history into evaluation data sets rather than relying solely on synthesized or SME-created data
• Measure and report quality — define metrics and dashboards that track evaluation coverage and answer quality over time, and help the company articulate, internally and to increasingly sophisticated customers, how we know our AI is high quality

Requirements:

• Demonstrated program and process ownership — a track record of taking an ambiguous, cross-functional problem, defining a process, rallying people around it, and driving it to measurable outcomes
• Strong people- and network-coordination skills — proven ability to navigate a large organization, recruit and motivate contributors, and orchestrate distributed work across many stakeholders. This is the heart of the role
• Network-building and incentive design — instinct for building a distributed contributor/expert community and designing incentives that attract and retain scarce expertise
• Hands-on fluency with AI / LLM evaluation — not just familiarity. You should be able to describe and write an eval and articulate what good eval coverage looks like, and have enough depth to design the system to requirements. You can lean on our engineering/QA and SME partners, but you need enough fluency to know what to ask them. Comfort with ground truth, rubrics, test sets, regression testing, LLM-as-a-judge, and human-in-the-loop review is expected
• Excellent communication and stakeholder management across Product, Engineering/QA, and SMEs; able to translate between technical and domain audiences
• Strong project- and resource-management skills, comfortable with the ebb-and-flow of evaluation work (a heavier ramp to establish a baseline, then flexing with demand)
• Detail-oriented and quality-obsessed, with sound judgment about coverage, prioritization, and when 'scoped-down but shipped' beats 'comprehensive but stalled.'
• Technical fluency — enough comfort with tooling to partner with engineering on (or extend) an internal evaluation workbench; light scripting or 'vibe-coding' is a plus, not a requirement
• Experience building or operating an expert network, annotation/labeling program, or human-feedback pipeline (e.g., RLHF-style review or a quality-operations function)
• Exposure to construction, AECO, or construction technology — enough domain awareness to queue work intelligently (deep construction expertise is not required; we can source it)
• Experience in B2B SaaS or AI product environments
• A track record designing metrics and dashboards to track evaluation coverage and quality over time