Senior AI Engineer

Procore Technologies, Inc. is hiring a Senior AI Engineer for its Evals & Goldsets team. The role builds and improves systems that evaluate AI agents for the construction industry, including production tracing, online LLM judging, and offline evaluation goldsets and benchmarks.

Responsibilities:

• Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production
• Build and maintain production trace infrastructure with Langfuse and Databricks
• Build and maintain data pipelines for the goldsets behind our evals, including annotation and quality checks
• Build agentic loops that mine production traces for failures and turn them into new eval cases and agent improvements
• Improve the runtime, cost, and reliability of our online and offline eval pipelines
• Contribute code and architecture, review designs, and mentor other engineers on the team