AI Quality Assurance, Prompt Team

Role Overview

• Define, execute, and continuously improve quality assurance standards for AI agent and prompt outputs, ensuring generated responses meet expectations for accuracy, consistency, completeness, and explainability.
• Oversee QA activities throughout the AI lifecycle, including pre‑deployment validation and post‑deployment output monitoring (regression testing).
• Ensure AI outputs support informed decision-making by validating clarity, applicability, and alignment with intended business value.
• Serve as an advocate for high-quality AI outputs, promoting shared understanding of output standards and expectations across the organization.
• Lead and oversee QA Analysts responsible for reviewing and evaluating AI agent and prompt outputs.
• Provide direction, coaching, and prioritization to ensure consistent application of output QA standards and timely completion of QA activities.
• Forecast QA timelines and capacity in coordination with AI leadership to support delivery commitments and ensure deadlines are met.
• Coordinate cross team to align QA activities with delivery timelines and ensure prompt and agent readiness.
• Assist with the development of repeatable AI output evaluation approaches that can be applied across use cases to drive efficiency and consistency.
• Document QA findings, trends, and recurring issues related to AI agent and prompt outputs.
• Prepare reports, summaries, and presentations for management outlining QA results, risks, and improvement opportunities.
• Identify opportunities to improve QA processes, tools, and standards based on observed output behavior and delivery feedback.

Requirements

• Bachelor’s degree in a relevant field (such as Business, Computer Science, Data Science, Engineering, or related discipline), or equivalent combination of education and experience.
• Minimum of 6+ years of industry and/or relevant experience, typically with 1+ years in a Senior Associate level role or external equivalent.
• 8+ years of relevant professional experience, with demonstrated involvement in AI, analytics, content evaluation, automation, or quality assurance disciplines.
• Prior QA management experience preferred, including oversight of analysts or reviewers responsible for evaluating outputs or deliverables.
• Prior leadership experience preferred, with the ability to coordinate work, forecast timelines, and drive accountability across teams.
• Demonstrated experience evaluating AI-generated outputs, including prompts, agent responses, or decision-support content.
• Strong understanding of how prompt design, instructions, and context influence AI agent behavior and output quality.
• Excellent written and verbal communication skills, with the ability to explain quality findings and recommendations to both technical and non-technical stakeholders.
• Strong analytical and critical thinking skills, with attention to patterns, edge cases, and systemic quality issues.
• Experience working in Agile or delivery-oriented environments, coordinating priorities across multiple stakeholders.
• Real estate, mortgage, or financial services experience preferred, but not required.

Benefits

• PTO and paid holidays
• Health insurance
• Dental insurance
• Vision insurance
• Life insurance
• Disability insurance
• 401K