Senior AI Product Engineer

Health Note is building AI-powered digital assistants for healthcare workflows. The Senior AI Product Engineer will build and improve production AI product systems for the Access Agent, using evaluations, analytics, experiments, and operational data to measure performance, identify issues, and drive reliable improvements.

Responsibilities:

• Own improvements to the Access Agent from investigation through production rollout, measurement, and follow-up
• Build practical ways to measure agent performance, including evals, dashboards, review workflows, logs, and production metrics
• Analyze transfer patterns, containment rates, task success, and operational outcomes to identify improvement opportunities
• Build tooling and workflows that connect prompts, configurations, interventions, customer workflows, and outcomes
• Create checks that help us catch regressions before and after changes go live
• Partner with Product, Operations, and Engineering to prioritize and validate improvements
• Design lightweight experiments and measurement approaches for model, STT, prompt, workflow, and configuration changes
• Build visibility into agent performance through dashboards, analytics, and reporting
• Use AI-assisted development tools and agentic workflows to accelerate experimentation and delivery
• Own a product improvement from problem identification through production rollout and measured impact
• Establish practical checks for evaluating changes before production deployment
• Create visibility into key Access Agent performance metrics and breaking points
• Ship measurable improvements with documented before-and-after impact
• Help Health Note build a disciplined, scalable system for continuous AI improvement