Staff Software Engineer, LLM Inference Platform

Zenlayer is hiring a Staff Software Engineer to lead inference performance for its LLM platform. The role focuses on benchmarking and optimizing inference engines, kernels, KV transfer, and parallelism to improve token throughput and cost while meeting latency objectives.

Responsibilities:

• Establish a unified Token Performance metrics framework (TTFT, TPOT, Goodput, GPU utilization, per-token cost), and build load-testing benchmarks that reflect real production traffic (high concurrency, long context, multi-turn Agent workloads)
• Perform bottleneck attribution for models such as DeepSeek, Kimi, GLM, Qwen, and MiniMax across compute, memory bandwidth, communication, scheduling, and queuing, and let the data determine which layer is the dominant constraint
• Dive deep into the source code of vLLM / SGLang / TensorRT-LLM, customizing and contributing upstream on critical paths such as the scheduler, Continuous Batching, Chunked Prefill, CUDA Graph, and KV block management
• Clearly define the responsibility boundary between the inference engine and orchestration layers such as Dynamo / llm-d, and reduce tail latency through request routing and load balancing
• Lead the core design of Prefill-Decode disaggregation: the KV transfer path (NIXL/RDMA/UCX), transfer-compute overlap, and phase-aware autoscaling
• Define the failure domains and recovery semantics of cross-instance KV transfer, and answer the key question: under which workloads does PD disaggregation deliver gains, and under which does it make things slower
• Design hybrid TP/PP/DP/EP parallelism strategies and quantify the trade-offs between EP and TP
• Tackle expert load imbalance and All-to-All communication overhead, and drive Wide-EP into production
• Write and tune critical CUDA (and ROCm/HIP) operators: memory access optimization, kernel fusion, and Attention / MoE / communication-related kernels
• Build a closed-loop "quantization – accuracy – throughput" evaluation: the real-world gains and accuracy regressions of FP8/FP4/INT4/AWQ/GPTQ across different engines and hardware