Key skills
Test automationPythonAI model evaluationCI/CDTest automation toolsAI infrastructureJavaScript/Node.jsFairness testingResponsible AISoft skillsJavaScriptAIMLLLMRAGAgenticNode.jsOpenTelemetryPostmanOpenAPIVersion ControlOWASP
About this role
Alight is a company that champions people and helps organizations build a healthy workforce through innovative benefits solutions. The AI Automation Engineer will play a key role in developing and maintaining automated test frameworks for AI models, collaborating with various teams to ensure quality and efficiency in AI automation processes.
Responsibilities:
- Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
- Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
- Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
- Author and validate OpenAPI/JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
- Instrument observability metrics (latency, success rate, cost, drift) using OpenTelemetry and export to dashboards
- Support data QA, lineage tracking, and drift monitoring (Evidently, OpenLineage)
- Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes
- Contribute to red-team playbooks and security testbooks mapped to OWASP/ATLAS risks
Requirements:
- Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
- Proficiency in Python, JavaScript/Node.js, or similar programming languages
- Familiarity with CI/CD, version control, and cloud-based deployment environments
- Experience with test automation tools (promptfoo, Ragas, Postman)
- Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
- Basic knowledge of fairness, safety, and security testing for AI systems
- Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
- Interest in Responsible AI, bias mitigation, and regulatory compliance
- Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience