Key skills
PythonJavaC++CPHPGoSwiftSQLObjective-CAIMLNLPLLMGitVersion ControlCommunication
About this role
Oak Tree Software is seeking experienced Software Engineers to evaluate, benchmark, and improve the coding capabilities of frontier AI models. In this role, you will assess AI-generated code, validate solutions against real-world software engineering tasks, and contribute to the development of high-quality evaluation datasets and benchmarks.
Responsibilities:
- Review and evaluate AI-generated code for correctness, efficiency, maintainability, and adherence to requirements
- Analyze software engineering tasks and validate whether proposed solutions meet expected outcomes
- Debug code, reproduce issues, and verify fixes across different programming environments
- Assess model-generated explanations, reasoning, and implementation approaches for technical accuracy
- Create, refine, and maintain evaluation datasets, benchmarks, and grading rubrics for coding tasks
- Identify edge cases, failure modes, and areas where AI systems struggle with software engineering problems
- Document findings clearly and provide structured feedback to improve evaluation quality and consistency
- Collaborate with project teams to establish quality standards and evaluation methodologies
Requirements:
- Bachelor's or Master's degree in Computer Science, Software Engineering, or a related technical field
- 3+ years of professional software engineering experience
- Strong proficiency in one or more of the following languages: Python, Java, C/C++, Go, Swift, Objective-C, PHP, or SQL
- Strong understanding of data structures, algorithms, software design principles, and debugging methodologies
- Experience performing code reviews and evaluating code quality in production or large-scale codebases
- Ability to analyze complex technical problems and assess solution correctness with minimal supervision
- Familiarity with version control systems (e.g., Git) and modern software development workflows
- Strong written communication skills and attention to detail
- Experience with AI/ML data annotation, NLP, prompt engineering, model evaluation, or LLM-related projects
- Experience evaluating AI-generated code, benchmark creation, or software quality assessment