Senior Ruby Software Engineer (AI Evaluation)

The UVA VEC is focused on improving the reliability and clarity of conversational systems through structured AI evaluation initiatives. They are seeking a Senior Ruby Software Engineer to evaluate AI-generated Ruby code, validate outputs, and provide detailed feedback to enhance system performance.

Responsibilities:

• Evaluate AI-generated responses to coding and software engineering problems
• Execute and validate Ruby code to ensure correctness and performance
• Identify logical errors, inefficiencies, and edge case failures
• Annotate outputs with detailed feedback on strengths and weaknesses
• Assess code readability, maintainability, and architectural soundness
• Perform fact-checking using reliable technical references
• Apply standardized evaluation frameworks and scoring criteria
• Ensure outputs align with expected engineering and conversational standards

Requirements:

• 5+ years of professional experience in software engineering or related fields
• Strong expertise in Ruby programming language
• Ability to solve medium to hard algorithmic problems independently
• Experience executing, testing, and debugging production-level code
• Strong understanding of data structures, algorithms, and system design principles
• High attention to detail in reviewing technical reasoning and outputs
• Fluent English communication skills
• Experience using LLMs in coding workflows and understanding their limitations
• Ability to follow structured evaluation frameworks and guidelines
• Bachelor's degree or higher in Computer Science or related discipline
• Experience contributing to open-source projects with accepted contributions
• Familiarity with additional programming languages or ecosystems
• Experience in model evaluation, RLHF, or data annotation
• Background in web frameworks such as Ruby on Rails or similar ecosystems
• Experience reviewing code in production environments
• Ability to explain complex technical concepts clearly to varied audiences