Member of Technical Staff, LLM Evaluation Infra

Inception develops fast, efficient diffusion-based AI models, including its Mercury large language model. The company is seeking an experienced engineer or scientist to build evaluation infrastructure and systems for frontier LLM performance, including scalable pipelines, benchmarks, statistical analyses, and quantitative metrics for quality, safety, reliability, and regression detection.

Responsibilities:

• Build scalable, automated evaluation pipelines that integrate into model training and deployment workflows
• Design, develop, and maintain robust evaluation frameworks and benchmarks for measuring LLM performance across diverse tasks and domains
• Conduct rigorous statistical analysis of model outputs to identify failure modes, biases, and performance gaps
• Partner with product and customer-facing teams to translate real-world use cases into meaningful evaluation criteria
• Define and implement quantitative metrics that capture model quality, safety, reliability, and regression detection