ML & Agentic Systems Engineer [IC4]

Sourcegraph builds tools that help engineering teams understand and manage large codebases. The Staff ML & Agentic Systems Engineer will lead model, evaluation, and agent-system direction for the Code Understanding team, improving the quality, cost, and latency of its AI-powered products.

Responsibilities:

• Agentic systems. You'll design and harden the multi-step, tool-using agent loops behind current and new agentic experiences, turning research and experiments into reliable, observable, and affordable products at enterprise scale
• Pragmatic use of evaluations. Crafting agentic products means needing to tell when a change actually helped, which is hard when agents keep changing and the product keeps shifting. You'll bring judgment about where evaluations earn their keep, when to use targeted smoke tests and metrics, and how to avoid noise dressed up as rigor, so we can move fast with confidence
• Models: selection, upgrading, and training. You'll decide which models we run where, drive upgrades, and fine-tune our own when that's the right call
• Retrieval and context engineering. You'll push on how we ground models in a customer's code - retrieval, ranking, context windows, citations - to make answers more accurate and verifiable
• Cost and latency. Every surface has a per-user economic budget. You'll treat cost and latency as product features, and profile, distill, cache, and right-size models so we can ship ambitious features sustainably
• Get the Code Understanding products and their model/agent pipelines running end-to-end locally, and land your first improvements to a model, prompt, retrieval path, or eval
• Build a clear picture of where the AI engineering pain is, and the product surfaces most constrained by them
• Get to know the team and our customers, and start forming your own opinions about where our agentic products should go next
• Join the team's on-call support rotation
• Own a meaningful agentic slice of the product end-to-end, driving it from problem-framing through rollout and measurement
• Establish how the team ships model and prompt changes responsibly: the evals, dashboards, and guardrails that make quality and cost regressions visible before customers feel them
• Begin up-leveling teammates in building with models by pairing with them, reviewing their code, and modeling good agent engineering instincts
• Be the recognized technical authority for agent engineering and agentic systems on Code Understanding. You will be the person teammates, and increasingly the wider department, defer to on model, eval, and agent-design decisions
• Have measurably moved the products: better answer quality, lower cost/latency, or new agentic capabilities that weren't feasible before
• Be setting the direction of the team's roadmap where it intersects agents, bringing conviction, backed by evidence, about which bets are worth making, and pulling other engineers up to execute on them

Requirements:

• You are a staff engineer and technical leader with hard-won skills across production machine learning, evaluation, and agent systems
• You're equally comfortable reasoning about an eval harness, a fine-tuning run, a retrieval pipeline, and the multi-step agent loop that ties them together, and you make everyone around you better at all of it
• You operate at staff scope: you own the most ambiguous, highest-risk problems in your domain, go into whatever codebase a problem requires, set standards and patterns others adopt, and translate fluidly between engineering goals and business objectives
• You influence direction beyond your immediate team
• You lead through technical excellence and mentorship
• You have personally owned a production model lifecycle
• You have trained or fine-tuned at least one model and taken it from dataset construction through evaluation, production rollout, and monitoring
• You can explain how you chose between training or fine-tuning and prompting or retrieval, how you chose baselines and metrics, what error analysis revealed, and how production evidence affected the next version
• You've designed multi-step agentic systems and made them reliable, observable, and cost-bounded
• You have a point of view on where agents shine and where deterministic code or human judgment is required
• You build representative datasets, meaningful baselines, useful error taxonomies, and release criteria that connect offline measurements to production behavior
• You know where rigorous evaluations earn their keep, where lightweight smoke tests or qualitative review are enough, and where a precise-looking metric is misleading
• You make measured quality, latency, and cost tradeoffs and use the appropriate combination of model selection, prompting, retrieval, caching, distillation, and fine-tuning rather than reaching reflexively for a more complex model
• Given a rough product idea, a few customer quotes, and a Slack thread, you come back with a plan, a prototype, milestones, and a point of view on tradeoffs, without it being pre-scoped
• You own high-technical-risk projects end-to-end
• As a senior IC, you go into whatever part of the codebase a problem requires, recognize issues beyond your immediate area, and translate between engineering goals and business objectives
• You mentor by pairing on hard problems, providing substantive design and code reviews, and spreading agent engineering literacy across the team
• You're comfortable on customer calls and in feedback threads, you turn raw signals into requirements, scopes, and milestones, and you push back when feedback would lead the product astray
• You ship the smallest correct thing, prefer robust solutions over complicated ones, and keep a high-quality bar with simplicity
• You're a strong software engineer who can ship production services
• You're comfortable across our stack - Go on the backend, TypeScript on the frontend, GraphQL, Postgres, Docker - or you're clearly able and eager to get there
• You're fluent with agentic coding tools, and you understand and own every line it submits
• You've shipped an LLM-powered or agentic developer-facing product you can speak about opinionatedly: what worked, what didn't, what you'd do differently
• You've fine-tuned, distilled, or trained models to meet cost, latency, or quality targets in production
• Experience with retrieval, ranking, embeddings, or search relevance
• Experience working directly with enterprise customers and translating their needs into a product
• Experience mentoring or up-leveling engineers, especially raising a team's agent engineering fluency