The UVA VEC is focused on improving the reliability and clarity of conversational systems through structured AI evaluation initiatives. They are seeking a Senior Ruby Software Engineer to evaluate AI-generated Ruby code, validate outputs, and provide detailed feedback to enhance system performance.
Responsibilities:
- Evaluate AI-generated responses to coding and software engineering problems
- Execute and validate Ruby code to ensure correctness and performance
- Identify logical errors, inefficiencies, and edge case failures
- Annotate outputs with detailed feedback on strengths and weaknesses
- Assess code readability, maintainability, and architectural soundness
- Perform fact-checking using reliable technical references
- Apply standardized evaluation frameworks and scoring criteria
- Ensure outputs align with expected engineering and conversational standards
Requirements:
- 5+ years of professional experience in software engineering or related fields
- Strong expertise in Ruby programming language
- Ability to solve medium to hard algorithmic problems independently
- Experience executing, testing, and debugging production-level code
- Strong understanding of data structures, algorithms, and system design principles
- High attention to detail in reviewing technical reasoning and outputs
- Fluent English communication skills
- Experience using LLMs in coding workflows and understanding their limitations
- Ability to follow structured evaluation frameworks and guidelines
- Bachelor's degree or higher in Computer Science or related discipline
- Experience contributing to open-source projects with accepted contributions
- Familiarity with additional programming languages or ecosystems
- Experience in model evaluation, RLHF, or data annotation
- Background in web frameworks such as Ruby on Rails or similar ecosystems
- Experience reviewing code in production environments
- Ability to explain complex technical concepts clearly to varied audiences