LLM Inference Systems Engineer

Hippocratic AI is building a safety-focused healthcare artificial intelligence platform. The company is seeking an LLM Inference Systems Engineer to build and operate distributed LLM serving, high-performance networking, GPU inference, benchmarking, observability, and Kubernetes production systems.

Responsibilities:

• Design, implement, and operate disaggregated LLM serving architectures, including independent prefill and decode pools, KV-cache transfer, routing, batching, and failure recovery
• Optimize multi-node inference across modern GPU systems using NVLink, NVSwitch, NVLS, InfiniBand, RoCEv2, GPUDirect RDMA, NCCL, UCX, and related communication paths
• Build and tune serving systems using SGLang, vLLM, NVIDIA Dynamo, TensorRT-LLM, or comparable frameworks
• Profile end-to-end performance across compute, memory, network, and storage; improve time to first token, inter-token latency, throughput, tail latency, and cost per token
• Develop repeatable benchmark and capacity-planning workflows across model architectures, GPU types, parallelism strategies, and concurrency levels
• Productionize serving on Kubernetes with health checks, autoscaling, safe rollouts, metrics, tracing, and actionable diagnostics
• Diagnose complex distributed failures such as collective timeouts, topology mismatches, packet loss, congestion, KV-transfer stalls, GPU OOMs, and uneven load
• Partner with model researchers and platform engineers to launch new models, serving features, and hardware generations safely