Staff Software Engineer, AI Inference Gateway

Salad is building a sustainable cloud computing platform powered by distributed consumer and workstation GPUs. The Staff Software Engineer will own the AI Inference Gateway, designing and operating request routing, scaling, observability, billing telemetry, and inference infrastructure while evaluating models, quantizations, and GPU hardware.

Responsibilities:

• Own the AI Gateway system end to end: request routing, streaming, batch/async job handling, and horizontal scaling across multiple gateway servers
• Build and tune the fleet-efficiency algorithms — autoscaling nodes, scoring performance, and evicting underperformers so the network converges on peak throughput per dollar
• Operate and scale the underlying inference nodes running vLLM, llama.cpp, and similar servers
• Design and maintain OpenTelemetry-based observability across the gateway and the fleet; pay-per-token and subscription billing already rides on this telemetry, so you'll keep that integration accurate
• Spend roughly 25% of your time benchmarking new models, quantizations, and GPU hardware, and work with the product and marketing teams to make the call on what goes into production
• Collaborate on cross-system design and integrations with the rest of Salad engineering; explain trade-offs clearly to both technical and non-technical teammates