NVIDIA AIPerf Ends Vanity Throughput: TTFT, ITL, Truth

NVIDIA released AIPerf on September 19, a benchmarking tool for LLM inference at scale — multiprocess load generation that prevents client-side bottlenecks, tracking time-to-first-token, inter-token latency, and throughput across 15+ endpoint types with Poisson and gamma traffic patterns. Vanity throughput numbers just lost their hiding place.

The release matters because most published inference numbers measure the benchmark harness, not the server. Single-process clients saturate before the endpoint does, reporting client limits as server performance. Three facts anchor AIPerf:

• Multiprocess architecture isolates server performance by removing client-side bottlenecks from the measurement.
• TTFT, ITL, and throughput are tracked separately instead of collapsing into one tokens-per-second figure.
• Configurable Poisson and gamma traffic patterns simulate realistic production arrivals instead of polite sequential requests.

This is the measurement rigor my serving evaluations follow, the same batch-realistic discipline as my speculative-decoding analysis. Same honesty standard, now as a shipping tool.

Why tokens-per-second lies

A single tokens-per-second figure hides three independent behaviors. TTFT — time to first token — measures prefill: how fast the server digests your prompt and emits token one. It decides interactivity; past 500ms chat feels broken. ITL — inter-token latency — measures decode smoothness: the gap between consecutive tokens across the whole stream, where p99 tails decide whether output stutters. Throughput measures sustained concurrent capacity: how many users the server holds before queues explode. A server can top throughput charts while delivering 4-second first tokens — unusable for chat, invisible in the headline. A server can top throughput charts while delivering 4-second first tokens — unusable for chat, invisible in the headline.

Here's the catch. Most teams benchmark with a script that sends one request at a time and reports the average — measuring an idle server under zero contention. Production arrives as bursts with three distinct shapes. Poisson arrivals model independent users trickling in — the chat baseline. Gamma shapes model batch jobs with clustered, bursty submissions — nightly pipelines, eval harnesses, agent fleets looping. Flash crowds add a third shape no steady-state benchmark captures: sudden 10x spikes that detonate queueing. Concurrent sessions then compete for KV cache and batch slots, so latency degrades non-linearly past saturation — the knee that sequential scripts never reach because they never approach it. An idle-server benchmark predicts nothing about the 9 AM peak except optimism.

That matches my KV-cache findings: capacity planning lives or dies on realistic concurrency, and sequential benchmarks systematically overstate headroom.

What AIPerf measures that others skip

Don't do this: signing inference contracts on vendor tok/s. My rule is throughput-at-SLO: maximum sustained load while TTFT stays under 500ms and ITL p99 under 100ms. Any number without the latency bound attached is marketing with units.

The pattern: benchmark the contract, not the demo

flowchart TD
    SLO[Define SLO: TTFT, ITL p99, error budget] --> LOAD[AIPerf multiprocess load]
    LOAD --> SHAPE[Poisson user + gamma batch traffic]
    SHAPE --> MEASURE[TTFT, ITL, throughput at SLO]
    MEASURE --> VERDICT{Meets SLO at target load?}
    VERDICT -->|yes| SIGN[Sign, attach report]
    VERDICT -->|no| TUNE[Retune: batching, parallelism, cache]

Every inference decision — provider, GPU shape, parallelism config — re-runs the same battery. Reports attach to contracts so capacity claims stay falsifiable. My per-task cost ledger prices each configuration per thousand agent requests, turning throughput deltas into dollar deltas.

Step 1: Define SLOs before load

config.py

from pydantic import BaseModel

class ServeSLO(BaseModel):
    ttft_p50_ms: int = 500
    itl_p99_ms: int = 100
    target_rps: float = 40.0
    traffic: str = "poisson"
    endpoints: int = 15
    duration_min: int = 10
    client_procs: int = 8

CONFIG = ServeSLO()

SLOs come from product requirements, not vendor sheets — chat needs TTFT, agents need sustained throughput, background jobs need cost per million. The traffic shape mirrors production mix: Poisson for user arrivals, gamma for batch bursts. Eight client processes minimum, or the harness becomes the ceiling.

Step 2: Run the battery, read the tails

run_battery.py

async def battery(endpoint, cfg=CONFIG) -> dict:
    try:
        res = await aiperf.run(endpoint, processes=cfg.client_procs,
                               traffic=cfg.traffic, rps=cfg.target_rps,
                               minutes=cfg.duration_min)
    except HarnessError as e:
        logger.warning("harness fault", extra={"err": str(e)})
        raise
    ok = (cfg.ttft_p50_ms > res.ttft_p50
          and cfg.itl_p99_ms > res.itl_p99)
    return {"pass": ok, "ttft": res.ttft_p50,
            "itl_p99": res.itl_p99, "rps": res.sustained_rps,
            "client_cpu": res.client_cpu_max}

Client CPU is a first-class result: if the load generators saturate, the numbers describe the test rig, not the server. My protocol pins generator CPU under 70% — above that, add processes and re-run rather than recording. Token accounting includes warmup and discarded tokens, verifier-style aggregation overhead where applicable, and the full ten-minute window rather than the best sixty seconds. Equal request counts never imply equal compute: batching, prefix caching, and parallelism settings change the cost of the same traffic, so every report carries the server configuration alongside the metrics. My first AIPerf-style run reported a ceiling that moved 40% upward the moment I doubled client processes — a year of capacity planning had rested on harness limits.

requirements.txt

httpx==0.28.1
pydantic==2.8.0
numpy==2.1.0
structlog==24.4.0
python-dotenv==1.0.1

Pydantic v2.8 needs extra="allow" on result schemas or nested metric payloads fail validation. I lost an afternoon to that exact error before pinning it.

Step 3: Re-run on every change that matters

Model swaps, quantization changes, parallelism retunes, traffic mix shifts — each re-runs the ten-minute battery in CI. Two cases proved the habit: an FP8 quantization that held quality benchmarks while doubling ITL p99 tails on long decodes, caught Wednesday instead of reaching Friday traffic; and a tensor-parallelism change from 4 to 8 that lifted throughput 30% while adding idle-GPU cost the bill review caught the same week. Quality gates and serving gates run side by side, and either can block a release. My terminal-bench harness gates quality; this battery gates serving. A quantization that holds quality while breaking ITL tails ships nowhere until retuned.

The client-ceiling war story: 40% phantom headroom

For a year our capacity model said the endpoint saturated at 28 RPS. AIPerf-style multiprocess loading showed 39 — the old single-process harness had capped out, and we had provisioned 40% extra GPUs against a phantom ceiling. Six figures of annual GPU spend traced to an unmeasured client bottleneck. Measure the measurer first, always.

When NOT to run the battery

Let's be clear. Serverless inference with autoscaling needs spot checks, not batteries — the provider owns capacity. Tiny prototypes never repay ten-minute runs; trust defaults until volume justifies measurement. And single-user local serving answers to a stopwatch, not a load generator.

Skip it for serverless, prototypes, and laptops. Run it where you sign capacity contracts, provision GPUs, or promise latency SLOs — everywhere a phantom ceiling costs real money.

AIPerf makes honest inference measurement shippable: multiprocess load, tail metrics, realistic traffic, client-CPU receipts. Benchmark the contract, and the whole class of phantom-ceiling spend disappears.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.