ML Systems Engineer - Model Training and Infrastructure (SWE-focused LLMs)

Cosine is building autonomous AI engineers and Lumen models that can understand, write and maintain software within real development workflows. The ML Systems Engineer will build model-training environments, data-generation pipelines, reinforcement-learning infrastructure and evaluation systems for software-engineering models, working across machine learning, software engineering, data and infrastructure.

Responsibilities:

• Contribute to the end-to-end training of software-engineering models
• Implement supervised fine-tuning pipelines using curated code and conversation datasets
• Build reinforcement-learning loops in which models write code, run tests and use development tools
• Develop custom PyTorch dataloaders, training objectives and evaluation workflows
• Run and analyse fine-tuning and RL experiments across large, modern open-source models
• Develop synthetic data-generation pipelines for future RL and fine-tuning runs
• Design systems for producing, filtering, transforming and sampling large-scale datasets
• Work with object storage, dataset sharding and data-quality checks
• Investigate which examples, tasks and sampling strategies lead to better model behaviour
• Turn model failures into concrete improvements to training data and future experiments
• Design, build and deploy containerised services that support model training and evaluation
• Use Docker and orchestration platforms such as Kubernetes to operate RL infrastructure
• Build environments where agents can modify repositories, run tests, use tools and receive meaningful feedback
• Improve the reliability, reproducibility and observability of large-scale training runs
• Work across Python, Go and the surrounding infrastructure required to run these systems
• Help maintain and extend evaluation suites for code models
• Build evaluations around unit tests, benchmark suites, repository-level tasks and real engineering workflows
• Analyse model failure modes, reward-hacking risks and regressions
• Develop better ways to measure code quality, maintainability, correctness and architectural fit
• Feed evaluation results back into model, data and infrastructure decisions
• Work with research, infrastructure and product teams to identify the highest-value training problems
• Turn broad goals such as “make Lumen better at X” into clear experiments and measurable outcomes
• Develop opinionated reward functions for software-engineering agents
• Document decisions, communicate trade-offs and help the wider team understand what the results mean
• Take ownership of projects from initial idea through to deployment and iteration