Define and measure conversation quality KPIs that align with product, support, and business goals.
Conduct systematic quality audits of AI conversations using both quantitative metrics and qualitative assessment, flagging conversations that went well and those that need improvement.
Analyze conversation data to identify failure patterns, drop-off points, conversion blockers, and untapped sales opportunities.
Partner with product and engineering to translate analytical insights into specific improvements: prompt refinements, conversation flow redesigns or updates to logic for how our AI responds to queries.
Establish quality assurance processes and frameworks that scale with conversation volume.
Support the integration of conversational quality tools and frameworks into existing workflows and tooling.
Present findings and recommendations to leadership with clear narratives connecting AI performance to business outcomes.
Lead cross-functional initiatives.
Own the AI conversation quality testing strategy, define standards, performance benchmarks, and a long-term improvement roadmap.

5+ years of experience with a track record of business impact in analytics, business systems, conversational AI, UX, or related roles.
Strong analytical skills with hands-on experience in data interpretation, trend analysis, and reporting.
Familiarity with conversational AI principles, metrics (e.g., intent accuracy, resolution rate), and tools (e.g., Claude, ChatGPT, Gemini).
Excellent written and verbal communication skills with the ability to translate data insights into narrative recommendations.
Experience with SQL and BI tools (e.g., Looker or Tableau) and comfort leading self-serve analysis.
Strong curiosity, attention to detail, and a data-informed mindset.
Ability to thrive in a remote, cross-functional, collaborative environment.

AI Conversation Analyst

Key skills