AI Infrastructure Engineer

Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. They are seeking an AI Infrastructure Engineer to own the software stack of GPU clusters, ensuring efficient operation and management of high-performance infrastructure for model training and serving.

Responsibilities:

• OS Bring-Up & Node Lifecycle Engineering
• Golden Images & Automated Bring-Up: Own the node software definition — versioned OS images, kernel tuning (NUMA, hugepages, IRQ affinity, cgroups), GPU/NIC driver stacks — and the automated pipeline that takes a node from base OS to production-ready
• Validation & Burn-In: Build automated acceptance suites (DCGM diagnostics, nccl-tests/RCCL tests, bandwidth and topology checks, HPL) that gate every node before it enters a scheduler pool
• Fleet Maintenance: Execute rolling kernel/driver/toolkit upgrades with minimal disruption to running workloads; enforce configuration consistency, detect drift, and maintain the driver ↔ CUDA/ROCm ↔ framework compatibility matrix across the fleet
• Self-Healing Operations: Automate detection of unhealthy nodes (Xid/ECC errors, link flaps, thermal throttling), with cordon/drain/reboot/re-image workflows and clean handoff to Hardware Operations for physical repair or RMA
• Configuration Management & Automation
• Infrastructure as Code: Manage all node and cluster configuration through Ansible/SaltStack playbooks in Git, with peer-reviewed changes, CI validation, and canary rollouts before fleet-wide deployment
• Provisioning Pipelines: Build and maintain image/provisioning tooling (PXE, MaaS, Packer, or similar) so new or re-imaged nodes are reproducible, not hand-crafted
• Operational Tooling: Develop Python/Bash tooling for cluster operations, health reporting, and workflow automation
• Orchestration & Scheduling (Kubernetes & Slurm)
• Kubernetes for Serving: Deploy and operate GPU-enabled Kubernetes for inference workloads — NVIDIA GPU Operator, device plugins, node feature discovery, topology-aware scheduling, and MIG/MPS partitioning where appropriate
• Training Schedulers: Operate Slurm (or Run:AI) for multi-node training — partitions, QoS, preemption, accounting, and container integration (enroot/pyxis)
• Container Platform: Maintain base images, registries, and the NVIDIA Container Toolkit / ROCm container stack; keep training and serving images lean, current, and reproducible
• GPU Driver & ML Stack Engineering
• Driver & Runtime Lifecycle: Build, deploy, and debug the full accelerator stack — NVIDIA (CUDA toolkit, cuDNN, NCCL, Fabric Manager) and AMD (ROCm, RCCL) — including kernel modules (DKMS), GPUDirect RDMA/Storage, and the RDMA software stack (MOFED/DOCA)
• Framework Environments: Maintain curated, optimized PyTorch and JAX environments with sane dependency and version management for researchers and production services
• Distributed Performance: Tune NCCL/RCCL across NVLink/NVSwitch and InfiniBand/RoCE fabrics, ensure topology-aware job placement, and run continuous communication/throughput benchmarks to catch regressions
• Advanced Debugging & Observability
• Escalation Point: Own the hard problems — NCCL hangs and timeouts, CUDA memory leaks, ROCm kernel crashes, straggler nodes, and unexplained throughput drops
• Observability: Own software-layer monitoring (DCGM exporter, Prometheus/Grafana, alerting) plus job-level GPU utilization and cluster efficiency reporting