Senior Software Engineer – LLM Evaluation (Remote)

Keystone Recruitment is working with a global AI research client focused on developing advanced evaluation datasets for large language models. The Senior Software Engineer will curate high-quality datasets and collaborate with researchers to assess AI-generated code and enhance model reliability in software engineering workflows.

Responsibilities:

• Curate and develop realistic software engineering tasks across languages such as Python, JavaScript (including React), C/C++, Java, Rust, and Go
• Review, evaluate, and refine AI-generated code for efficiency, scalability, correctness, and maintainability
• Collaborate with cross-functional research teams to enhance AI-driven coding solutions against industry performance benchmarks
• Design verification mechanisms to automatically validate software engineering solutions
• Analyze stages of the software development lifecycle (architecture design, API design, prototyping, production deployment, monitoring, and maintenance) and evaluate model performance across these stages
• Build internal tools or agents to detect code quality issues and error patterns

Requirements:

• Several years of professional software engineering experience
• At least 2 years of continuous full-time experience at a product-focused technology company
• Strong expertise in building and deploying scalable, production-grade applications
• Deep understanding of software architecture, debugging, performance optimization, and code review standards
• Experience working with modern development workflows and tooling
• Strong written and verbal communication skills for documenting structured evaluation feedback