Senior Deep Learning Communication Architect

Santa Clara, Utah, United States of America

Full Time

20 hours ago

$184,000 - $288,000 USD

H1B Sponsor Likely

Key skills

Deep neural networks (DNN) training and inferenceLarge language model (LLM) training and inference optimizationParallelism techniques for deep learning Data ParallelismParallelism techniques for deep learning Pipeline ParallelismParallelism techniques for deep learning Tensor ParallelismParallelism techniques for deep learning Expert ParallelismParallelism techniques for deep learning FSDPDisaggregated Serving architecturesInference servers DynamoInference servers TritonPyTorchTensorRT-LLMvLLMSGLangC++ programmingPython programmingGPU computingCUDAOpenCLInfiniBand networksRoCE networksCommunication protocols for deep learningHigh-speed interconnects NVLinkHigh-speed interconnects InfiniBandHigh-speed interconnects SPC-XCommunication libraries MPICommunication libraries NCCLCommunication libraries UCXCommunication libraries UCCCommunication libraries NVSHMEMDeep LearningCommunication

About this role

NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. They are seeking a Senior Deep Learning Communication Architect to optimize communication performance and design efficient communication protocols for deep learning workloads.

Responsibilities:

The software architecture group at NVIDIA has openings for a Deep Learning Communication Architect. We scale the DNN models and training/inference frameworks to systems with hundreds of thousands of nodes
Optimizing communication performance: Identify and eliminate bottlenecks in data transfer and synchronization during distributed deep learning training and inference
Designing efficient communication protocols: Develop and implement communication algorithms and protocols tailored for deep learning workloads, minimizing communication overhead and latency
Hardware and software co-craft: Collaborate with hardware and software teams to craft systems that effectively apply high-speed interconnects (e.g., NVLink, InfiniBand, SPC-X) and communication libraries (e.g., MPI, NCCL, UCX, UCC, NVSHMEM)
Exploring innovative communication technologies: Research and evaluate new communication technologies and techniques to enhance the performance and scalability of deep learning systems
Developing and implementing solutions: Build proofs-of-concept, conduct experiments, and perform quantitative modeling to validate and deploy new communication strategies