Remote | Python Software Engineer — $55–$85/hour

24-MAG is offering a specialized full-time consulting opportunity for experienced software engineers with strong Python development skills. The role involves designing and evaluating software engineering tasks for AI models, requiring expertise in Python, technical documentation, and debugging.

Responsibilities:

• Create realistic, multi-step software engineering challenges based on practical development workflows
• Design technically demanding problems that require implementation, debugging, environment configuration, and analytical reasoning
• Define clear requirements, constraints, expected outputs, and acceptance criteria
• Ensure tasks assess genuine software engineering capability rather than superficial code generation
• Build complete and verifiable reference solutions in Python
• Create the supporting setup, dependencies, tests, and validation checks required for each task
• Write clean, readable, and maintainable code
• Confirm that solutions run reliably within the intended technical environment
• Document implementation decisions and expected behaviour clearly
• Use AI coding assistants and agent-based development tools within practical engineering workflows
• Evaluate how frontier models approach complex coding and debugging tasks
• Identify implementation errors, unsupported assumptions, inefficient approaches, and incomplete solutions
• Analyse where AI agents succeed, struggle, or exploit unintended shortcuts
• Document failure patterns and provide evidence supporting evaluation conclusions
• Review tasks and reference solutions created by other software engineering specialists
• Assess clarity, correctness, difficulty, reproducibility, and technical fairness
• Identify ambiguous instructions, hidden assumptions, grading gaps, and environment issues
• Provide actionable feedback that improves task quality and benchmark reliability
• Collaborate closely with researchers and fellow task authors

Requirements:

• At least 1 year of experience in software engineering, research engineering, or a related coding-intensive role
• Strong hands-on Python scripting, implementation, and debugging skills
• Experience developing clean, readable, and maintainable software
• Everyday fluency with Git, IDEs, repositories, and standard software development workflows
• Comfort configuring environments, dependencies, tooling, and validation processes
• Strong technical writing and documentation skills
• Ability to work independently through ambiguous and open-ended engineering problems
• Reliable availability for approximately 35 hours per week
• An MSc or PhD in computer science, software engineering, another STEM discipline, or a related technical field is highly relevant
• Equivalent practical experience in a research-intensive or engineering-intensive role may also be considered
• Academic or professional work involving significant coding, data analysis, or technical experimentation may strengthen an application
• Open-source contributions, technical projects, publications, or substantial software development work may also be valuable
• Experience using AI coding assistants, prompt engineering methods, or agent-based workflows
• Previous work in AI training, model evaluation, or benchmark development
• Background authoring technical tasks, reference solutions, or grading criteria
• Familiarity with automated testing, CI/CD workflows, containers, or reproducible environments
• Experience reviewing code or technical assignments created by other engineers
• Knowledge of agentic AI systems and multi-step coding evaluations
• Strong ability to identify edge cases, unintended shortcuts, and subtle implementation issues
• Experience collaborating with AI research or evaluation teams