Key skills
GPU cluster bringupInfiniBandRoCEEthernet fabricsServer architecturePCIeNUMAMemory hierarchyPerformance debuggingAutomation frameworksDistributed storage systemsLustreCephNVMe-oFLiquid coolingHigh-density deploymentsAILeadership
About this role
Nscale is building AI infrastructure for frontier-scale workloads, focusing on high-density, high-performance GPU clusters. They are seeking a Principal Systems Engineer to lead the deployment and validation of large-scale GPU clusters, ensuring performance and operational excellence.
Responsibilities:
- Define the technical standards for node, rack, and full-cluster bringup
- Lead large-scale GPU cluster deployments (multi-rack, multi-pod environments)
- Architect high-performance network fabrics (IB, RoCE, Ethernet) optimized for AI workloads
- Establish cluster-level acceptance criteria and validation frameworks
- Tune and validate NCCL, RDMA, GPUDirect, and collective operations at scale
- Identify and eliminate performance bottlenecks across hardware, topology, and firmware layers
- Drive congestion control and fabric optimization strategies
- Define performance benchmarking methodology for AI training workloads
- Design repeatable deployment models for multi-site expansion
- Build automation frameworks for provisioning and cluster validation
- Establish deployment SLAs, quality gates, and operational readiness standards
- Reduce time-to-capacity while increasing reliability
- Serve as the escalation point for complex bringup and performance issues
- Mentor senior engineers and shape infrastructure best practices
- Influence hardware selection, rack topology, and data center design decisions
- Partner with executive leadership on infrastructure scaling strategy
Requirements:
- 10+ years of experience in large-scale infrastructure or HPC environments
- Proven experience bringing up large GPU clusters (hundreds+ GPUs)
- Deep expertise in high-speed networking (InfiniBand, RoCE, Ethernet fabrics)
- Strong understanding of server architecture (PCIe, NUMA, memory hierarchy)
- Experience debugging performance issues across compute and network layers
- Strong automation and systems-level thinking
- Experience scaling AI training clusters for frontier models
- Experience with liquid cooling or ultra-high-density deployments
- Knowledge of distributed storage systems (Lustre, Ceph, NVMe-oF)
- Experience defining infrastructure standards in a fast-growing organization