Software Quality Engineer – AI

Role Overview

• Design and execute comprehensive test plans for backend services, APIs, and microservices architectures.
• Partner with Backend Engineers, AI Engineers, and Product teams to understand requirements and identify quality risks early in the development cycle.
• Develop automated testing frameworks using tools like Pytest, Playwright, unittest, and integration testing libraries.
• Build end-to-end test suites that validate the integration between backend systems, AI agents, databases & services.
• Advocate for quality-first practices, influencing architectural decisions and embedding testing into the development lifecycle.
• Design and implement automated evaluation frameworks for Generative AI features, including LLM / SLM model testing, prompt/ output validation, and behavioral assessment of agentic workflows.
• Develop quality metrics and evaluation methodologies for LLM-based applications, assessing accuracy, consistency, reliability, quality of AI/ML models.
• Create and maintain curated evaluation datasets and synthetic test data that cover edge cases, adversarial scenarios, and real-world variability.
• Leverage AI evaluation tools and frameworks & establish observability and monitoring requirements to quality assessment.
• Establish observability and monitoring requirements including structured logging, metrics collection, and tracing for both backend services and AI agent behavior.
• Implement quality gates and acceptance criteria and create test documentation including test plans, automation architecture, quality reports, and runbooks.
• Build strategies for testing in cloud environments & containerized execution.
• Implement performance testing, load testing, and reliability testing for production backend services and AI inference pipelines.
• Define and champion quality standards, best practices, and testing methodologies for both traditional backend systems and AI applications.
• Conduct code reviews with a focus on testability, quality patterns, and maintainability.
• Mentor junior quality engineers, sharing expertise in automation frameworks, AI testing approaches, and quality engineering principles.
• Communicate quality insights, risk assessments, and test results effectively to technical and non-technical stakeholders using tools like Jira and Confluence.

Requirements

• 2.5+ years of experience in software quality engineering, test automation, or software development with strong QA focus.
• Deep expertise in test automation frameworks and tools such as pytest, unittest, Selenium, Playwright etc.
• Strong programming skills in Python or Java with ability to write production-quality test code and automation frameworks.
• Proven track record building automated testing infrastructure for backend services and APIs.
• Experience with various testing methodologies: unit testing, integration testing, end-to-end testing, contract testing, performance testing, and test-driven development (TDD).
• Strong understanding of CI/CD pipelines and experience integrating automated tests into deployment workflows.
• Comfortable working in Linux/Unix environments for test execution and debugging.
• Familiarity with testing AI/ML applications or LLM-based systems, with understanding of unique quality challenges in non-deterministic systems.
• Knowledge of Generative AI concepts, agentic frameworks (ReAct, chain-of-thought, tool use), and LLM capabilities/limitations.
• Exposure to agentic AI patterns and knowledge of how to test tool orchestration, multi-turn conversations, and RAG systems.
• Excellent written and verbal communication skills with ability to document findings clearly and comprehensively.
• Strong collaboration skills working across cross-functional teams in Agile/Scrum environments.
• Experience with collaboration tools like Jira, Confluence, and Git for team workflows.

Tech Stack

• Cloud
• Java
• Linux
• Microservices
• Python
• React
• Selenium
• Unix

Benefits

• Health insurance
• 401(k) matching
• Flexible work hours
• Paid time off
• Remote work options