Applied AI Engineer — Evaluation & Reliability

15Rock is a financial intelligence firm that builds research using multi-agent pipelines on large language models. The Applied AI Engineer will build deterministic evaluation gates, fact-reconciliation and citation-integrity systems, regression harnesses, and production infrastructure to ensure AI-generated research is accurate and reliable.

Responsibilities:

• Deterministic evaluation gates that block a report when a defect is present — measured against thresholds set before the run, not a gut feel
• A fact-reconciliation layer that verifies a figure agrees with itself everywhere it appears
• Citation-integrity checks: every reference resolves, every quote is real, every source is live
• Golden-set and regression harnesses that catch quality drift before it ships
• The path from research to production: hardening a validated approach into the pipeline, behind CI, with the traces to prove it works