Senior PowerShell Software Engineer (AI Evaluation)

The UVA VEC is focused on a high-impact AI initiative aimed at enhancing the reasoning accuracy and reliability of conversational systems in software engineering. The Senior PowerShell Software Engineer will evaluate AI-generated scripts, validate system-level behaviors, and ensure technical precision to improve model performance.

Responsibilities:

• Evaluate AI-generated PowerShell scripts for correctness, clarity, and completeness
• Execute and validate scripts in controlled environments to verify outputs
• Identify logical flaws, inefficiencies, and edge case failures in automation workflows
• Annotate model responses with detailed, structured feedback
• Assess script quality, maintainability, and adherence to best practices
• Verify technical accuracy using trusted documentation and references
• Apply standardized evaluation frameworks and scoring methodologies
• Ensure alignment with expected system behavior and conversational guidelines

Requirements:

• 5+ years of experience in software engineering, automation, or related technical roles
• Advanced expertise in PowerShell scripting and Windows-based systems
• Strong understanding of system administration, automation, and scripting best practices
• Ability to independently solve complex algorithmic and scripting challenges
• Experience executing, debugging, and validating scripts across environments
• High attention to detail in reviewing technical outputs and reasoning
• Fluent English communication skills (written and technical)
• Experience using LLMs in development workflows and understanding their limitations
• Bachelor's, Master's, or PhD in Computer Science or a related field
• Experience contributing to open-source projects with accepted pull requests
• Familiarity with model evaluation, RLHF, or annotation workflows
• Background in competitive programming or technical problem solving
• Experience reviewing scripts or code in production environments
• Exposure to multiple programming languages or automation ecosystems
• Ability to clearly explain technical concepts to non-technical audiences