Key skills

LLM-Based Agent EngineeringAgent Orchestration and Tool UseAgent Evaluation and Regression TestingBackend and Distributed SystemsManaged Cloud Model Platforms and LLM Provider APIsProduction Observability and Distributed TracingAgent SecurityModel Context Protocol (MCP)Sandboxed Execution EnvironmentsAI Coding AgentsKubernetesGoAI

About this role

Cobalt builds and operates AI agents that test customer applications. The Staff AI Engineer will set technical direction and build shared foundations for reliable, secure, cost-efficient production agents, while improving how engineering teams use AI coding agents.

Responsibilities:

Set technical direction for agent systems across teams, including orchestration, tool use, context and memory management, human oversight and failure recovery
Design, build and operate shared foundations: evaluation frameworks, agent observability and tracing, execution controls, and model access with routing and failover where needed
Establish evaluation practices that measure task completion, accuracy, false positives, security and cost per successful task. Work with domain experts to build representative test cases and turn production failures into regression tests
Make agent workflows resilient to tool failures, timeouts, rate limits and partial execution, with clear limits on retries, runtime and spend
Work with security and platform teams to build in scoped permissions, customer data isolation, audit trails and safe execution of tools and code
Establish practical engineering standards: evals in CI, prompt and model versioning, regression testing for non-deterministic systems, cost and latency budgets, and staged rollouts with rollback paths
Lead model selection and cost optimization. Benchmark models against real workloads and make informed trade-offs between quality, latency and cost
Partner with teams on production readiness, incident investigation and improvements to agent reliability
Help teams use AI coding agents effectively across planning, implementation, review and testing. Measure the impact on delivery speed and software quality
Mentor engineers and technical leads through design reviews, pairing, internal talks and written guidance. Help teams adopt shared capabilities and improve them based on feedback
Work with product, research and security leaders to turn business priorities into a technical roadmap with clear outcomes