Quality Assurance Engineer

Trilon is building a supercharged, technology-enabled future for their people and partners. The QA Engineer plays a critical role in ensuring that every AI-powered tool meets high standards for quality, reliability, and usability, while also being responsible for the full quality lifecycle and collaborating closely with engineers and product managers.

Responsibilities:

• Own the full quality lifecycle for your product pod, from test planning through production sign-off
• Define acceptance criteria in partnership with engineers and product managers
• Design comprehensive test strategies that cover functionality, performance, and reliability
• Establish and maintain quality standards for all deliverables
• Design, build, and maintain automated test suites for APIs, services, and user interfaces
• Develop regression tests to detect issues as systems evolve
• Integrate automated testing into CI/CD pipelines to ensure continuous validation
• Write clean, maintainable test code that scales with the product
• Design and implement testing approaches for AI-generated outputs, including prompt evaluation and RAG validation
• Define evaluation criteria and benchmarks for accuracy, consistency, and usefulness
• Monitor for quality drift as models, prompts, or data sources change
• Partner with engineers to improve output reliability and performance
• Identify, document, and track bugs and quality issues across the stack
• Work closely with engineers to troubleshoot and resolve defects early in the development cycle
• Prioritize issues based on impact and risk to end users
• Validate that all features meet acceptance criteria and quality standards before release
• Assess production readiness, including performance, reliability, and edge cases
• Serve as the final sign-off authority for releases within the pod
• Partner with Lead Engineers, Applied AI Engineers, and Product Managers to improve development and testing practices
• Engage with operating company users to understand real-world expectations and feedback
• Continuously improve testing methodologies, tools, and frameworks

Requirements:

• 4+ years of experience in QA engineering, software testing, or a related role
• Strong experience with test automation frameworks and tools for API, backend, and UI testing
• Proficiency in at least one programming language such as Python, JavaScript, or similar
• Experience building and maintaining automated test suites and integrating with CI/CD pipelines
• Experience testing APIs and working with REST or similar architectures
• Familiarity with modern software development practices, including version control and agile methodologies
• Experience testing AI-enabled applications, including LLM outputs, prompt behavior, or RAG pipelines
• Ability to define evaluation criteria for non-deterministic systems and assess output quality
• Strong analytical and problem-solving skills, with attention to detail
• Ability to clearly communicate quality risks, tradeoffs, and recommendations
• Experience collaborating closely with engineers, product managers, and cross-functional teams
• Comfort working in a fast-paced environment with evolving requirements
• Curiosity and interest in understanding engineering workflows within the AEC industry