Key skills
Quality EngineeringPythonNode.jsCI/CD integrationAutomation frameworksE2E testingPerformance testingMentoring QA teamsCommunication skillsAILLMCI/CDLeadershipMentoringCommunication
About this role
Keystone Recruitment is seeking a hands-on Quality Engineering Lead to define, own, and execute the end-to-end quality strategy for full-stack and LLM-powered products. This role blends technical depth with leadership, ensuring robust automation, release readiness, and measurable quality standards across distributed systems.
Responsibilities:
- Own quality strategy for complex, distributed systems; define quality metrics such as SLOs, defect escape rate, DORA metrics, and coverage benchmarks
- Design and implement automation frameworks and evaluation pipelines using Python and Node.js
- Drive testing across all layers: E2E, frontend/UI, backend/API, and performance (load and stress testing)
- Build and operate quality systems for LLM-powered products, including:
- AI agents and multi-step agent workflows
- MCP tools, servers, and integrations
- LLM-as-a-Judge automated evaluation harnesses
- Define golden datasets, regression prompts, judge calibration strategies, and validation processes for probabilistic/non-deterministic behavior
- Integrate automated quality systems into CI/CD pipelines with release gates, orchestration, and dashboards
- Own release readiness processes and quality sign-off
- Lead root-cause analysis, defect triage, and continuous improvement initiatives
- Mentor QA engineers and SDETs; establish clear ownership boundaries and quality standards
- Collaborate closely with Engineering, Product, and Research stakeholders
Requirements:
- 7+ years of experience in Quality Engineering, SDET, or Software Engineering roles
- 2–3+ years leading quality initiatives or owning quality strategy for complex systems
- Strong development skills in Python and Node.js for automation and evaluation tooling
- Proven experience defining end-to-end quality strategy for full-stack applications
- Experience testing across E2E, UI, backend/API, and performance layers
- 2+ years building automated evaluation systems for LLM-powered products, including agents, workflows, MCP components, and LLM-as-a-Judge pipelines
- Experience working with golden datasets, regression prompts, and judge calibration
- Experience integrating QA systems into CI/CD pipelines with release gates
- Experience mentoring QA/SDET teams
- Excellent written and verbal communication skills in English