AI Infrastructure Operations Engineer - Compute & Storage - Highrise AI

Highrise AI, a subsidiary of Hut 8, operates GPU infrastructure for AI workloads. The AI Infrastructure Operations Engineer provides Tier 2 support for production GPU compute and storage platforms, troubleshooting incidents, restoring service, and maintaining reliable cluster operations.

Responsibilities:

• Own and manage escalated incidents, service requests, and problem tickets from initial technical triage through resolution, escalation, or documented handoff
• Diagnose compute and storage faults using system logs, monitoring dashboards, management interfaces, and approved diagnostic tools
• Assess severity and customer impact; coordinate with the NOC and site teams to restore service within operational and SLA targets
• Produce clear troubleshooting records, timely ticket updates, shift handovers, and actionable Tier 3 or OEM escalations
• Troubleshoot Linux servers, GPU health alerts, boot failures, firmware/driver issues, hardware faults, and resource availability problems
• Support node provisioning, reimaging, configuration changes, routine maintenance, and return-to-service validation using established automation and runbooks
• Investigate Kubernetes and Slurm node or workload failures, working with platform engineers when application or scheduler-level escalation is required
• Perform first-level analysis of connectivity or GPU communication symptoms and engage network or Tier 3 engineers for deeper fabric and performance investigation
• Monitor and troubleshoot distributed and high-performance storage systems, including capacity, throughput, availability, and client connectivity
• Investigate filesystem mount failures, storage-node or disk alerts, degraded services, and access or performance complaints
• Execute approved recovery, maintenance, and component replacement workflows with storage vendors and data hall technicians
• Support operational tasks across file, object, and block storage platforms as deployed at Highrise sites
• Assist with upgrades, planned maintenance, cluster expansions, and validation activities following established change controls
• Maintain accurate asset, configuration, and ticket records; improve runbooks, troubleshooting checklists, and knowledge articles
• Identify recurring incidents and automate repeatable operational work with scripts and existing infrastructure management tools
• Collaborate with remote engineering teams, on-site technicians, OEMs, and vendors to resolve issues safely and efficiently

Requirements:

• Bachelor's degree in computer science, engineering, or related discipline, or equivalent hands-on experience
• 3–6 years of experience supporting Linux servers, HPC, cloud, or data center infrastructure in a production environment
• Practical troubleshooting experience with enterprise compute hardware, BMC interfaces (iDRAC/iLO or equivalent), operating systems, and monitoring tools
• Working familiarity with storage technologies such as NFS, distributed filesystems, object storage, and storage performance fundamentals
• Experience using ticketing systems, change management procedures, and Bash or Python scripting for routine diagnostics and automation
• Structured, evidence-driven troubleshooting with sound judgment on when to remediate, collaborate, or escalate
• Strong ownership of tickets, attention to operational detail, and commitment to reliable customer service
• Ability to work effectively across NOC, SRE, hardware, network, storage, and Tier 3 engineering teams
• Clear written and verbal communication in English
• Exposure to NVIDIA GPU systems, InfiniBand/RDMA, Kubernetes, Slurm, or AI/HPC storage platforms is a strong advantage