Role: AI/ML QA Engineer

Location: McLean, VA 22102 (100% Onsite)

Primary Skills: AI/ML Testing, Python, SQL, ETL Testing, Snowflake, Databricks, Azure, Healthcare Domain, GenAI Testing.

Job Summary: We are seeking a skilled AI/ML QA Engineer to ensure the quality, reliability, performance, and governance of AI and Machine Learning solutions in the healthcare domain. The role involves validating AI models, testing data pipelines, evaluating model outputs, identifying bias and performance issues, and supporting the end-to-end quality assurance lifecycle for AI-driven applications.

The ideal candidate should have strong QA expertise combined with exposure to AI/ML technologies, data validation, automation testing, and healthcare data environments.

Key Responsibilities

Develop and execute test strategies for AI/ML applications and data-driven systems.
Validate AI model outputs for accuracy, consistency, bias, fairness, and reliability.
Design and execute data quality testing for structured and unstructured datasets.
Test ETL/ELT pipelines supporting AI model development and deployment.
Validate training, testing, and inference datasets used in AI solutions.
Perform functional, integration, regression, performance, and user acceptance testing.
Work with Data Scientists, Data Engineers, Product Owners, and Business Stakeholders to define AI testing requirements.
Create automated test frameworks for AI and analytics solutions.
Monitor model performance and identify drift, degradation, and anomalies.
Verify compliance with healthcare data quality and governance requirements.
Support defect triage, root cause analysis, and resolution tracking.
Prepare QA documentation, test cases, test reports, and validation evidence.

Required Technical Skills

QA & Automation
Manual Testing
Test Planning & Execution
Test Automation (Selenium, API Testing with Postman)
JIRA, TestNG / PyTest
Data Testing (SQL, ETL Testing, Data Validation)
AI/ML Testing (AI Model Validation, LLM Testing, Bias & Fairness Testing)
Programming (Python, PySpark preferred)
Cloud & Data Platforms (Azure, Databricks, Snowflake)

Expertise You'll Bring:

Experience testing Generative AI and LLM-based applications.
Familiarity with GitHub Copilot, Microsoft 365 Copilot, and enterprise AI tools.
Understanding of prompt engineering and prompt validation techniques.
Experience evaluating AI output quality, hallucinations, and response consistency.
Knowledge of Responsible AI principles, fairness, explainability, and governance.

Preferred Qualifications

Experience validating AI/ML models in production environments.
Exposure to MLOps processes and model deployment frameworks.
Knowledge of healthcare compliance and data privacy requirements.
Strong analytical and problem-solving skills.
Excellent communication and stakeholder management capabilities.

Education

Bachelor's Degree in Computer Science, Information Technology, Engineering, Data Science, or related field.

Nice to Have

Healthcare domain experience
Snowflake Certification
Azure Certification
Databricks Certification
AI/ML Testing Certification
Experience with GenAI and LLM evaluation frameworks

QA Engineer with AI/ML experience

Key skills

About this role