AI Benchmark Engineer, Inference Performance

Silicon Data builds data infrastructure, benchmarking, and market indices for the global compute economy. The AI Benchmark Engineer will build and run reproducible inference performance benchmarks, define measurement methodology, investigate anomalies, and translate results into defensible technical and economic metrics.

Responsibilities:

• Build and maintain the inference benchmarking harness. Automated, containerized, reproducible runs across inference engines (vLLM, TensorRT-LLM, SGLang, TGI), model families, and hardware targets
• Own the inference metrics that matter. Time-to-first-token, inter-token latency, sustained and peak throughput, latency percentiles under concurrency, goodput under SLO constraints, and their expression per dollar and per watt
• Design workload profiles that reflect real usage. Input and output length distributions, concurrency patterns, streaming versus batch, prefill-heavy versus decode-heavy — rather than synthetic best-case runs
• Capture and validate the configuration fingerprint. Engine version, model revision and quantization, driver and firmware, kernel and runtime versions, tensor and pipeline parallelism, scheduler settings. A result without its full configuration is not a result
• Establish run repeatability and statistical rigor. Define warmup and steady-state criteria, variance thresholds, minimum run counts, and outlier handling. Know when a run is invalid and be willing to throw it out
• Investigate performance anomalies. When a cohort underperforms, determine whether the cause is thermal, network, contention, misconfiguration, or a genuine silicon difference — and produce the evidence
• Partner with our data and GPU benchmarking engineers on cohort definition, cross-suite consistency, and how inference results align with hardware-level benchmarks
• Write the methodology. Public-facing write-ups, versioned methodology documents, and technical documentation that a skeptical customer can audit and reproduce
• Work with our pricing products so measured inference performance can be expressed in economic terms — cost of capability per measured token per second
• Track the field. New engines, new serving techniques, new model architectures, and emerging public benchmarks worth incorporating or explicitly rejecting