Remote | Software Engineer — $90–$140/hour

24-MAG LLC connects experienced professionals with remote consulting opportunities across technical and evaluation workstreams. The Software Engineer will evaluate AI-generated technical content and system outputs for correctness, clarity, relevance, and usefulness, while providing detailed written rationales and feedback. The role also involves reviewing technical documentation, applying consistent quality standards, and collaborating asynchronously in structured remote workflows.

Responsibilities:

• Evaluate and compare AI-generated responses to technical prompts for correctness, clarity, tone, and usefulness
• Review outputs involving technical documentation, error messages, code comments, architectural explanations, instructions, and related software-engineering content
• Determine whether generated responses accurately address the underlying technical question or requirement
• Identify outputs that appear fluent or convincing but contain factual, logical, or technical inaccuracies
• Assess whether technically correct responses are communicated clearly and appropriately for their intended audience
• Review technical writing for precision, completeness, structure, and instructional value
• Evaluate documentation, design documents, API references, specifications, and comparable technical materials
• Identify unclear explanations, ambiguous wording, misleading statements, inconsistencies, and missing information
• Assess whether technical content is appropriately tailored to developers, technical stakeholders, or other intended users
• Apply professional software-engineering judgement when reviewing the quality of technical communication
• Assign analytical quality scores to AI-generated technical responses
• Write detailed rationales explaining the reasoning behind each evaluation
• Clearly distinguish between factual correctness, communication quality, relevance, and overall usefulness
• Document subtle errors, inconsistencies, or ambiguities with precision and consistency
• Provide actionable written feedback that can support improvements in AI model performance
• Work independently within structured remote evaluation workflows
• Collaborate asynchronously with project managers, reviewers, and other expert contributors
• Follow established evaluation guidelines, rubrics, and project-specific quality standards
• Apply consistent judgement across repeated technical assessment tasks
• Complete assigned work within agreed project requirements and defined timeframes

Requirements:

• Professional experience as a Software Engineer working with a modern technology stack
• Native-level English fluency and exceptional written communication skills
• Proven experience authoring or reviewing technical documentation, design documents, API references, specifications, or similar materials
• Experience conducting code reviews, design reviews, or providing structured feedback on technical writing
• Strong ability to assess technical communication for correctness, audience appropriateness, and instructional value
• Excellent analytical skills and meticulous attention to detail
• Ability to identify subtle factual inaccuracies, logical inconsistencies, ambiguous explanations, and misleading technical language
• Strong independent-working skills and comfort operating within remote, asynchronous project environments
• Familiarity with AI model evaluation, annotation, RLHF, or related structured evaluation workflows is advantageous but not required