QA Engineer

Role Overview

Duties and Responsibilities:

· Design, build, and maintain a regression test suite covering both frontend and backend functionality

· Perform structured manual testing across new features and existing workflows

· Develop and maintain automated test suites using industry-standard frameworks

· Build and execute automated tests for AI agent response quality, including accuracy, tone, and flow adherence

· Produce regular test reports summarizing coverage, pass/fail rates, and outstanding defects

· Collaborate with Engineering and Product teams to define acceptance criteria and test cases prior to development

· Identify, document, and track defects through to resolution

· Contribute to CI/CD pipeline integration for automated test execution

Requirements

Must-have Skills / Qualification:

· 5+ years of experience in software QA, with a mix of manual and automation testing

· Hands-on experience building automated test suites using Playwright

· Basic scripting experience using JavaScript, Python, or Java

· Experience with API and backend testing (REST, Postman, or similar tools)

· Familiarity with testing AI or LLM-generated outputs, including evaluation of response quality, prompt adherence, and edge case handling

· Experience with prompt testing and LLM observability tools such as Langfuse, LangTrace, or similar platforms for tracing, evaluating, and monitoring AI responses

· Ability to write clear, traceable test cases aligned with acceptance criteria

· Strong written communication skills for test reporting and defect documentation

· Experience working in Agile development environments

Advantageous or Nice-to-Have Skills/Experience:

· Experience testing conversational AI or chatbot platforms

· Familiarity with healthcare or SaaS environments is a plus

· Experience integrating automated tests into CI/CD pipelines (e.g., GitHub Actions, CircleCI, or similar tools)

· Must have own equipment (laptop and stable internet connection)

Tech Stack

• Java
• JavaScript
• Python