Senior Software Development Engineer in Test

Qualia is hiring a Senior AI/Agentic Quality Infrastructure SDET to build production-grade test infrastructure and AI-driven validation systems. The role focuses on agentic testing, AI/ML evaluation, observability, automation frameworks, and engineering tools that help teams ship complex products reliably.

Responsibilities:

• Build and maintain test infrastructure, AI/ML evaluation systems, agentic validation harnesses, release orchestration tooling, observability, and performance testing platforms as production-grade software rather than scripts
• Design agentic test harnesses that let AI agents interact with applications, browsers, APIs, databases, logs, telemetry, and CI/CD systems to execute and validate complex workflows, reasoning about application state rather than relying solely on predefined test scripts
• Build AI/ML evaluation frameworks for validating non-deterministic systems, including deterministic assertions, semantic and model-based evaluation, regression datasets, and human-in-the-loop review, defining measurable quality signals such as correctness, reliability, latency, cost, and hallucination/error rates
• Build observability and telemetry into our automation and agentic infrastructure—structured logs, metrics, distributed tracing, and execution timelines—so engineers can understand why an automated or agentic workflow succeeded or failed
• Develop systems that automatically distinguish product, test, infrastructure, and AI/model reasoning failures, correlating them with logs, traces, deployments, and commits to accelerate root-cause analysis
• Design and build our test automation frameworks, including the migration from Cypress to Playwright, and own core infrastructure such as test environment provisioning, parallelization, and CI/CD integration at scale
• Apply AI to engineering-quality problems including test generation, flaky-test triage, failure clustering, and release-risk analysis, while building guardrails so agent conclusions are backed by observable evidence
• Partner with Software Engineers, Quality Engineers, SRE, and Platform teams to make these capabilities reusable platforms, and mentor engineers on test architecture, observability, and AI-assisted engineering

Requirements:

• 5+ years of experience as an SDET or software engineer building test infrastructure, developer tooling, platform engineering, observability, or release engineering systems—not primarily writing test cases or automation scripts
• Strong programming ability in JavaScript/TypeScript, including designing backend services, libraries, frameworks, and production-quality infrastructure from scratch
• Strong software engineering fundamentals including system design, distributed systems, APIs, asynchronous systems, and production operations
• Experience building or scaling CI/CD systems, release orchestration, test infrastructure, or developer platforms, and designing observability/telemetry systems using structured logging, metrics, and distributed tracing
• Hands-on experience with LLMs, AI-assisted engineering, or agentic systems beyond simply using an AI coding assistant, including tool/function calling, agent workflows, and multi-step AI workflows
• Experience or familiarity with LLM and agent evaluation, including evaluation datasets, deterministic and model-based evaluators, and the unique challenges of testing probabilistic/non-deterministic systems
• Strong understanding of the testing pyramid and experience determining when validation belongs at the unit, integration, API, UI, performance, or agentic layer
• Comfortable mentoring engineers and raising the technical quality bar across teams