Senior Site Reliability Engineer, DGX Cloud

NVIDIA is advancing artificial intelligence and high-performance computing through DGX Cloud, a fully managed AI platform on major cloud providers. The Senior Site Reliability Engineer will maintain and optimize large-scale DGX Cloud clusters, improve reliability and observability, operate GPU workloads across cloud environments, and lead incident response and automation efforts.

Responsibilities:

• Build, implement and support operational and reliability aspects of large-scale Kubernetes clusters with focus on performance at scale, real-time monitoring, logging, and alerting
• Define SLOs/SLIs, monitor error allowances, and streamline reporting
• Support services before they launch through system creation consulting, developing software tools, platforms and frameworks, capacity management, and launch reviews
• Maintain services once they are live by measuring and supervising availability, latency, and overall system health
• Operate and optimize GPU workloads across AWS, GCP, Azure, OCI, and private clouds
• Scale systems sustainably through mechanisms like automation and evolve systems by pushing for changes that improve reliability and velocity
• Lead triage and root-cause analysis of high-severity incidents
• Practice balanced incident response and blameless postmortems
• Participate in on-call rotation to support production services

Requirements:

• BS in Computer Science or related technical field, or equivalent experience
• 8+ years of experience operating production services
• Expert-level knowledge of Kubernetes administration, containerization, and microservices architecture
• Experience with infrastructure automation tools (e.g., Terraform, Ansible, Chef, Puppet)
• Proficiency in at least one high-level programming language (e.g., Python, Go)
• In-depth knowledge of Linux operating systems, networking fundamentals (TCP/IP), and cloud security standards
• Solid grasp of SRE principles, such as SLOs, SLIs, error budgets, and incident management
• Experience building and operating comprehensive observability stacks (monitoring, logging, tracing) using tools like OpenTelemetry, Prometheus, Grafana, ELK Stack, Lightstep, Splunk, etc
• Operating GPU-accelerated clusters with KubeVirt in production
• Applying generative-AI techniques to reduce operational toil
• Experience with workflow orchestration platforms such as Temporal, Cadence, Airflow, Argo Workflows, or Step Functions
• Experience operating and resolving problems in production AI inference workloads across the model-to-GPU stack, including vLLM, SGLang, PyTorch, TensorRT-LLM, NVIDIA Dynamo, CUDA, NCCL, and GPU performance analysis