Senior AI Engineer

Touchstones is an AI-native hiring company building role-specific engineering work samples and evidence-based candidate evaluations. The Senior AI Engineer will build and evaluate LLM pipelines, develop AI features for human reviewers, and manage model selection, cost, latency, and fallback behavior.

Responsibilities:

• Build the LLM pipelines that evaluate captured work against each rubric criterion, with evidence citations and calibrated reasoning
• Design evaluation harnesses (golden sets, regression evals, recall/precision on evidence retrieval) so scoring quality is measured, not assumed
• Author and harden AI-engineering assessments: retrieval evaluation, agent tool-call guardrails, prompt-injection defenses
• Ship agentic features for reviewers: summarizing work trails, flagging gaps, drafting follow-up interview questions
• Own model choice, cost, latency and fallback across providers
• Keep the product's principles intact: AI advises, humans decide, no biometric signals

Requirements:

• 5+ years in software engineering, including 2+ years shipping LLM-powered features to production
• Strong Python and/or TypeScript; experience with LLM APIs, structured outputs, tool calling and RAG
• Hands-on experience building evals and judging model quality with data
• A product mindset and comfort with startup ambiguity
• Clear written communication
• Experience with hiring/HR tech, assessment design or psychometrics
• Background in agent safety, guardrails or red-teaming
• Postgres/Supabase or Next.js experience
• Open-source or public writing on AI evaluation