Senior Machine Learning Engineer

The hiring client is building production-grade agentic software for engineering teams working with machine learning workloads at scale on its GPU platform. The role architects and delivers Python agents that automate cluster-failure analysis, telemetry and profiling interpretation, performance diagnosis, and operational workflows.

Responsibilities:

• Agent Architecture & Frameworks: Design and implement modular Python agent runtimes, orchestration loops, custom tool adapters, state/session management, and robust CLI/service interfaces
• LLM Orchestration & Reliability: Build deterministic, production-grade workflows using frameworks such as LiteLLM, DSPy, LangChain, or LlamaIndex. Enforce strict JSON Schema outputs, budget/token caps, allowlisted tool calling, and grounded, evidence-backed root cause reports
• Client Ecosystem Integration: Connect intelligent agents into the Client’s ROCm™ ecosystem, PyTorch, distributed training/serving runtimes, telemetry feeds, cluster log collectors, and hardware/software profiling utilities
• Evaluation & CI/CD Pipelines: Design comprehensive evaluation infrastructure, including golden fixture datasets, regression suites, LLM-as-a-judge benchmarks, and automated validation gates before rollout
• Cluster Operations & Diagnostics: Develop agents that autonomously triage multi-node distributed training and inference issues—such as collective communication hangs (NCCL/RCCL), OOMs, silent numerical divergence, host-device bottlenecks, and Slurm/Kubernetes job crashes
• Packaging & Security: Deliver production-ready artifacts utilizing Docker and Kubernetes, complete with comprehensive runbooks, defense-in-depth egress filtering, and secure credential handling suitable for customer-isolated environments

Requirements:

• Open exclusively to US-based candidates currently authorized to work in the US without current or future sponsorship
• requires a minimum of 10+ years of professional software experience (5+ years in production Python)
• Mandatory hands-on experience debugging multi-node GPU cluster jobs, rank communication failures, and production proficiency with PyTorch alongside large-scale frameworks (Megatron-LM, DeepSpeed, TorchTitan, or vLLM)
• Proven track record delivering production LLM agentic/tool-calling systems combined with direct experience in GPU profiler/trace analysis and cluster failure diagnosis
• Bachelor's degree or higher in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical industry experience
• 10+ years of professional software engineering experience, with 5+ years dedicated to building and scaling production Python distributed systems
• Hands-on experience debugging and running multi-node training or high-throughput inference across GPU clusters, including log/metric correlation across nodes and ranks
• Deep hands-on proficiency with PyTorch and large-scale acceleration/serving stacks (e.g., DeepSpeed, Megatron-LM, TorchTitan, vLLM, TensorRT-LLM, or TGI)
• Demonstrated track record resolving real-world cluster anomalies: throughput degradation, memory leaks/fragmentation, collective communication deadlocks, launch misconfigurations, and profiler trace analysis (PyTorch Profiler, Perfetto)
• Proven history of shipping agentic workflows, autonomous tool-use systems, or complex LLM chains into production or near-production environments
• High standards for clean design, test-driven development, typing (mypy), CI automation, and clear technical documentation
• Direct experience with the CLIENT ROCm™ software platform and ROCm profilers/debuggers (Omnitrace, Rocprof)
• Open-source contributions to major ML or LLM orchestration repositories (e.g., vLLM, PyTorch, Hugging Face, DSPy)
• Advanced performance tuning background: Model Flops Utilization (MFU) optimization, 3D parallelism strategies (TP/PP/DP/EP), roofline modeling, and custom kernel/communication overlap analysis
• Experience with HPC cluster orchestrators (Slurm, Kubernetes/KubeFlow) and high-speed interconnect telemetry (InfiniBand, RoCE)
• Knowledge of agent security posture: prompt injection mitigation, network isolation, least-privilege tool execution, and local VPC execution