Agent Quality \/ Evals Engineer 1754

Softgic is a technology company seeking an Agent Quality / Evals Engineer to own the eval harness and quality gate for measurable agent quality. This role involves building and maintaining the MVP eval harness, integrating evals into CI, and defining release-gate thresholds in collaboration with the product and tech lead.

Responsibilities:

• Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs
• Wire evals into CI so quality regressions fail builds and releases
• Define and maintain release-gate thresholds with Product and the Tech Lead
• Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope

Requirements:

• Experience evaluating ML, LLM, or non-deterministic systems
• Strong test and benchmark design capability
• Comfort working with noisy metrics, thresholds, and probabilistic behavior
• Good scripting and automation skills