Senior Software Engineer — AI Evaluation & Benchmarks

Alignerr is seeking an experienced software engineer to build coding benchmarks and data pipelines for evaluating frontier AI models. The role involves analyzing model-generated code, creating rigorous evaluation scenarios, and contributing to evaluation frameworks and systems that improve AI software development capabilities.

Responsibilities:

• Design and implement coding benchmarks used to evaluate frontier AI models across real-world programming tasks
• Build and maintain scalable data pipelines for AI evaluation workflows
• Analyze AI-generated code for correctness, reliability, and edge-case failures
• Create structured evaluation scenarios that rigorously test reasoning, debugging, and code quality
• Work with large code repositories and multi-language environments
• Collaborate on systems that improve how AI models understand and generate software
• Provide detailed technical feedback on model performance and failure patterns
• Contribute to the design of evaluation frameworks that set industry standards

Requirements:

• 4+ years of professional software engineering experience
• Experience working at a high-growth tech company or top-tier software organization
• Expert proficiency in Python — you write clean, performant, well-tested Python code
• Hands-on experience with code repositories and working in large, complex codebases
• Proven experience designing and implementing LLM coding benchmarks and data pipelines
• Track record of working in high-performance engineering environments with large-scale products or platforms
• Strong command of version control systems (Git) and modern development workflows
• Bilingual or native English speaker with strong written communication skills
• Self-directed, technically rigorous, and comfortable operating with autonomy
• Senior or Lead-level engineering profiles with a history of technical ownership
• Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field — or equivalent professional experience
• Proficiency in one or more additional languages: JavaScript, Go, C++, or other relevant languages
• Experience with CI/CD pipelines and writing robust unit tests (pytest, Mocha, JUnit)
• Background in security engineering or significant open-source contributions
• Familiarity with AI/ML evaluation methodologies or model benchmarking