AI Business Analyst

Allied Benefit Systems is focused on enhancing their AI capabilities and is seeking an AI Business Analyst to ensure the reliability and safety of their AI models and integrations. The Analyst will collaborate with developers and product owners to design tests, validate AI outputs, and contribute to the overall test strategy for trustworthy AI solutions.

Responsibilities:

• Test AI features integrated into core systems (e.g., Dynamics CRM, QicLink claims platform, Docuvantage) to ensure they function correctly in end-to-end workflows
• Write and automate test cases for AI outputs and traditional software features; define clear pass/fail criteria (using statistical sampling when appropriate) to validate AI results
• Own regression testing for AI capabilities: continuously re-run critical test scenarios as models are updated or data changes, to catch issues early
• Contribute to the overall test strategy and perform adversarial or edge-case testing (e.g., stress tests, prompt injections) to evaluate the robustness and security of AI components
• Document and recommend solutions for issues found; coordinate with the AI Product Owner and SMEs to clarify requirements for fixes and improvements
• Collaborate closely with developers and the Product Owner to report bugs, verify fixes, and improve requirements. Emphasize shared accountability for quality across the team

Requirements:

• Bachelor's degree in related field or equivalent work experience required
• 7+ years in software Quality Assurance/testing roles, with significant experience in enterprise systems or complex integrations required
• Proven expertise in test automation (UI and API) and building regression test suites in CI/CD environments
• Familiarity with AI/ML system testing principles. Comfortable validating outputs that may vary statistically and determining acceptable accuracy thresholds
• Strong analytical skills to identify edge cases and pinpoint defects across interconnected systems (frontend, backend, data, model)
• Excellent documentation and communication skills for writing test plans, reporting issues, and guiding teams on quality best practices
• Test automation tools such as Selenium or Cypress for web interface testing, and Postman or REST-assured for API testing
• Testing frameworks like PyTest or JUnit, integrated with CI/CD pipelines (Azure DevOps, Jenkins) for continuous testing
• Version control (Git) for maintaining test scripts, and monitoring/logging tools (Azure Monitor, Splunk) to investigate issues
• Performance testing or security testing tools for evaluating AI systems under load or threat scenarios
• Experience in industries like insurance or healthcare is a plus (provides domain context for Allied's processes)