Product Analyst

Location - Phoenix, AZ is preferred but remote can be considered

Required Qualifications

• 3+ years experience drafting detailed Jira user stories, maintaining product backlogs, and defining acceptance criteria based on business requirements in an Agile product team or equivalent environment.
• 2+ years experience in data analysis with the ability to interpret AI performance metrics and translate findings into actionable product recommendations.
• 1+ year experience working with LLM-based applications, including understanding of hallucination patterns, output quality issues, grounding techniques, bias detection, and mitigation strategies.
• 1+ year hands-on experience executing structured AI agent evaluation plans, tracking iterations, maintaining golden datasets, and performing regression testing.
• 1+ year experience supporting the development, evaluation, and continuous improvement of AI-powered products, AI agents, copilots, or generative AI solutions.
• Experience translating AI product requirements into prompts, agent workflows, business rules, and measurable success criteria.
• Experience defining AI product success metrics and measuring improvements in output quality, user satisfaction, task completion, or productivity.
• Experience partnering with engineering, UX, data science, and business stakeholders to prioritize, evaluate, and refine AI product capabilities.

Preferred Qualifications

• Experience in HR technology, learning technology, content creation platforms, or knowledge management solutions as a Business Analyst, Product Analyst, Product Owner, or equivalent role.
• Hands-on experience building and executing specialized AI test cases, evaluating LLM outputs, prompt variations, and agentic workflows.
• Hands-on experience with prompt engineering, prompt optimization, and evaluation of AI-generated content quality, accuracy, completeness, and consistency.
• Experience designing evaluation rubrics, benchmark datasets, golden datasets, and quality measurement frameworks for AI-generated outputs.
• Experience conducting side-by-side model comparisons, experimentation, or A/B testing to improve AI product performance.
• Familiarity with Retrieval Augmented Generation (RAG), grounding strategies, hallucination mitigation, and content sourcing concepts.
• Familiarity with automated AI evaluation tools, frameworks, and testing methodologies.
• Experience with human-in-the-loop workflows and designing review, validation, and approval processes for AI-generated outputs.
• Experience working with AI governance, responsible AI practices, bias detection, content safety, and model quality evaluation.
• Experience evaluating content quality against defined business, compliance, instructional design, or organizational standards.
• Strong written and verbal communication skills, with the ability to document evaluation methodologies, findings, recommendations, and trade-offs for both technical and business audiences.