Staff Applied AI Engineer, Product & Agent Performance

Arcadia is a healthcare platform that transforms complex healthcare data into trusted intelligence for providers, payers, and life sciences organizations. The Staff Applied AI Engineer will own product-layer decisions shaping agent behavior, including prompting, retrieval, context, memory, evaluation, and escalation, while improving the safety, reliability, transparency, and performance of agentic workflows.

Responsibilities:

• Design and iterate on agent behavior across real, live workflows, including long-horizon, multi-turn agentic tasks
• Design retrieval and context architecture so the right source data reaches a model in the right structure and agents remain grounded in real data rather than filling gaps with assumptions
• Design memory and state handling across multi-turn and multi-agent flows, determining what is carried forward, summarized, or dropped and why
• Create context and prompt templates that combine few-shot examples, structured formatting, and reasoning scaffolding for consistent agent behavior
• Improve performance through prompting, tool-use strategy, and context construction, validated through direct experimentation rather than guesswork
• Build and run evaluations against real production conditions to measure performance, regressions, failure modes, and edge cases
• Author evaluation rubrics, quality heuristics, and thresholds that weight failures by severity and cost, not just frequency, and monitor those measures against production behavior
• Design and validate escalation paths that route agents to human review based on confidence and uncertainty while preserving safety and consistency under adversarial and edge-case conditions
• Design for cost-aware performance alongside latency, reliability, and accuracy through efficient context construction and tool-call economy
• Evaluate and sign off on model changes by baselining current behavior, running comparative evaluations, and making the go/no-go call before a change reaches a customer
• Maintain product-level AI documentation, including model cards, intended use, limitations, and known failure modes, so customer-facing teams work from actual agent behavior
• Partner closely with Product and product managers to ensure agents are not just capable, but steerable, trustworthy, and ready to scale