Principal, Agentic Reliability Engineer

Slalom is hiring a Principal Agentic Reliability Engineer for its Agentic Managed Services practice. The role designs and deploys agentic solutions with clients, owns their reliability and performance in production, and helps develop reusable tooling and standards for the practice.

Responsibilities:

• Partner with clients to shape the opportunity and design the agent workflow — what the agents should do, how their performance will be judged, and what guardrails keep them safe
• Build the first working agents: stand up the initial workflow, evaluation harness, and guardrails, and demonstrate that the solution meets the client's acceptance criteria under real conditions
• Define the reliability baseline the solution must hold in production including the SLOs, evaluation criteria, and autonomy posture that govern it
• Own root-cause analysis and ongoing performance of the agent workflow itself: why it behaves as it does, and how to make it measurably better over time
• Own the evaluation harness: configure, run, and interpret agent evaluations against workflow-specific acceptance criteria
• Monitor drift signals (e.g. output-quality degradation, latency trends, evaluation pass-rate decline, cost anomalies) and remediate before they breach SLOs
• Run A/B and shadow evaluations when agent behavior changes are proposed
• Capture operational findings, incident lessons, and evaluation-calibration notes so the pod's knowledge compounds over time rather than resetting
• Define and track the KPIs that describe how well the agents are performing and investigate when they move in unexpected directions
• Translate telemetry into narratives the team and client can act on
• Develop and communicate clear plans to improve agent performance and reporting on their impact once shipped
• Serve as the final technical authority for AI agent incidents including owning root-cause analysis of the agent workflow, remediation, and postmortem documentation
• Determine whether an incident reflects a transient error, agent drift, workflow-logic failure, or an underlying model-behavior change
• Manage change control for agent workflow updates: version the change, run the evaluation suite, document the expected behavioral delta, and propose the go-live schedule
• Stage rollouts with shadow-deployment patterns before promoting to full release; monitoring the stability window and rolling back per runbook if thresholds are breached
• Manage identity, access, and integrations: app registrations and credentials, access control and permissions, and the authentication, authorization, and contracts that govern how agents communicate with other tools and systems
• Manage cost and infrastructure: right-sizing the compute agents run on, setting token and budget caps per agent, optimizing workflow steps for efficient resource use, and reporting spend — and the cost impact of the changes you make — to the team and client
• Contributing reusable agents, evaluation templates, and runbook patterns so each new engagement starts further ahead than the last
• Acting as a primary production signal for Slalom's AMS platform team — surfacing the gaps whose closure most improves how the work gets done
• Helping define the standards and playbooks for a new and fast-growing engineering discipline

Requirements:

• 5+ years of engineering experience in a production environment (software engineering, data engineering, platform/DevOps, or SRE)
• Strong Python or TypeScript; ability to read, modify, and evaluate agent orchestration code without extensive scaffolding
• Hands-on experience with at least one production AI/ML system (model serving, pipeline orchestration, or an LLM-based application)
• Familiarity with observability tooling (distributed tracing, structured logging, alerting) and incident-response workflows
• Analytical rigor: comfortable defining what correct looks like for an AI agent decision, and building a test to verify it
• Client-facing or solutioning experience — the ability to scope and shape technical engagements with clients, not only operate them
• Direct experience with LLM-based agent frameworks — LangChain, LangGraph, CrewAI, AutoGen, AWS Bedrock Agents, or equivalent
• Evaluation frameworks for AI systems (Langfuse, Braintrust, RAGAS, DeepEval, or custom harnesses)
• Familiarity with agentic memory architectures — episodic, semantic, and procedural stores; vector databases; RAG
• AI governance concepts: safety guardrails, NIST AI RMF, or OWASP LLM Top 10