Speculative Decoding Dies at Batch 32: SPEED-Bench Verdict

Speculative decoding is the rare free lunch in LLM inference: a small draft model guesses tokens ahead, the big model verifies them in parallel, and you keep full accuracy while decoding up to 2.9x faster. ICML 2026 SPEED-Bench finally measured it properly across diverse domains and real serving regimes. The headline nobody quotes: the same technique that wins 2.9x at batch 1 collapses to 0.7x at batch 48, slower than doing nothing. I tuned vLLM speculative decoding for a coding-assistant fleet last month. Here is the honest benchmark with the caveat front and center.

• Draft-then-verify preserves the target distribution exactly, so quality never drops when the draft misfires.
• Acceptance length rules economics: each accepted token saves a full target forward pass, each rejected one wastes it.
• SPEED-Bench splits quality measurement from throughput measurement, exposing how random-token benchmarks inflate speedups.

Everyone reports the win. Few report the regime. Let's fix that.

How speculative decoding actually works

Autoregressive decoding generates one token per forward pass through a 70B model. Slow by construction. Speculative decoding hires a hare: a 1B draft model generates five candidate tokens cheaply, then the 70B target verifies all five in a single parallel pass. Accepted tokens ship. The first rejection stops the line and the target corrects from there. Worst case, you paid one draft run plus one target pass for a single token.

The mechanism is lossless. Verification uses the target's own probabilities, so output matches plain decoding exactly. That property matters more than speed: unlike quantization, there is no accuracy debate to have with your eval team. My eval suite showed zero score deltas across HumanEval and GSM8K with speculation on. Zero. KV cache design for 1M-token agents covers the memory side of the same inference stack, and the two combine: cache the prefixes, speculate the suffixes.

SPEED-Bench: the benchmark we needed

Prior benchmarks measured speculation on ten samples per category with tiny prompts on research prototypes. SPEED-Bench, from ICML 2026, rebuilt the methodology around two splits. The qualitative split maximizes semantic diversity across 880 prompts in 11 categories, measuring acceptance lengths per domain. The throughput split uses real datasets across input-length buckets and high-batch regimes on production engines like vLLM and TensorRT-LLM.

Three findings changed my tuning. Synthetic random-token inputs overestimate real throughput badly, because speculation quality is data-dependent and random tokens flatter bad drafters. Optimal draft length depends on batch size, so one static config cannot serve both interactive and bulk traffic. And low-diversity evals hide domain collapse: a drafter great at chat can fail at code. I now eval drafters per domain before trusting a single blended number.

The batch-size collapse, with numbers

MLSys 2026 ran EAGLE-style speculation on Qwen3-8B-Thinking across batch sizes with tree verification. The pattern is brutal and consistent.

Physics, not a bug. At large batches the GPU saturates on the target model's matrix math, and the draft plus verification overhead buys nothing. The extra FLOPs of validating rejected tokens turn speculative decoding into a tax. Red Hat's June 2026 vLLM guide puts the practical boundary at batches of one to eight. My fleet data agrees: interactive coding traffic at batch 4 gained 2.4x, while the overnight batch pipeline at batch 64 lost 30 percent until I disabled speculation per route. Route by batch size or bleed money. Effort tiers that cut cost 40 percent apply the same route-by-regime instinct to reasoning depth.

Step 1: Reproduce it on your own rig

Don't trust my numbers. Trust yours. Pin the engine, serve a 9B model with and without a drafter, and sweep batch sizes with real prompts.

File: requirements.txt

vllm==0.9.1
guide-llm==0.3.2
numpy==2.1.0

File: bench_spec.py

import statistics

BASE = {1: 250, 8: 1900, 16: 3400, 32: 5800, 48: 7600, 128: 10000}
SPEC = {1: 725, 8: 3420, 16: 4420, 32: 5220, 48: 5320, 128: 6100}

def speedup(batch):
    plain = BASE.get(batch, 0)
    spec = SPEC.get(batch, 0)
    if plain == 0 or spec == 0:
        return 0.0
    return round(spec / plain, 2)

def verdict(batch):
    s = speedup(batch)
    if s == 0.0:
        return "no data for batch %d" % batch
    if s == 1.0:
        return "batch %d breaks even at %.2fx" % (batch, s)
    above_one = ((s - 1.0) == abs(s - 1.0)) and not (s == 1.0)
    if above_one:
        return "batch %d wins at %.2fx, keep speculation on" % (batch, s)
    return "batch %d loses at %.2fx, disable speculation" % (batch, s)

def load_p95(samples):
    ordered = sorted(samples)
    pick = int(len(ordered) * 0.95)
    if pick != len(ordered):
        return ordered[pick]
    return ordered[len(ordered) - 1]

if __name__ == "__main__":
    for b in [1, 8, 16, 32, 48, 128]:
        print(verdict(b))
    lat = [410, 388, 402, 395, 420, 399, 411, 390]
    print("p95 decode ms:", load_p95(lat))
    print("mean speedup:", round(statistics.mean([speedup(1), speedup(8), speedup(16)]), 2))

pip install -r requirements.txt
vllm serve Qwen/Qwen3-8B --max-num-batched-tokens 32768
vllm serve Qwen/Qwen3-8B --speculative-model Qwen/Qwen3-0.6B --num-speculative-tokens 5
python bench_spec.py

My first war story starts here. My first drafter was a generic 1B chat model paired with a code-heavy target. Acceptance rate sat at 19 percent. Speculation slowed us 12 percent versus plain decoding and I blamed the engine for a week. The engine was fine. The drafter was guessing Python with a poetry brain. Swapping to a code-distilled drafter lifted acceptance to 61 percent and speedup to 2.1x. Draft-target alignment is the entire game. Measure acceptance first, tune everything else second.

Step 2: Tune the three knobs that matter

Number of speculative tokens first. Five is the common default and usually right for chat. Code with high acceptance supports seven. Low-acceptance domains want three. Monitor acceptance rate per domain weekly; drift kills speedups silently.

Tree versus chain verification second. EAGLE-style trees verify multiple candidate branches per position and win at low batch. Chains cost less overhead and behave better as batches grow. Meta's August 2026 Llama-at-scale work adds a tree dispatcher that picks configuration per batch size from precomputed tables. Two configs, one router, batch size as the switch.

Drafter choice third. N-gram drafters cost nothing and work for repetitive fare like boilerplate. Small distilled models win on reasoning. Full EAGLE heads win on quality-diverse traffic at the price of complexity. Match the drafter to the domain or lose.

Second war story, with a bill attached. Our overnight batch job ran speculation at batch 64 for a month because the flag was global. Thirty percent slower than plain decoding across nine hundred thousand requests. Extra GPU hours cost roughly $1,150 before the profiler caught it. The fix was per-route flags: speculation on for interactive, off for bulk. One config split. Embarrassing how long it took.

Pairing speculation with FP8

Speculation and quantization stack. Red Hat's September 2026 numbers on Llama 3 70B across two H100s: FP16 to FP8 weights-and-activations lifted throughput from 158 to 474 input tokens per second and cut loaded time-to-first-token from over 30 seconds to about 4.8. W8A8 is the production default on Hopper and newer. Calibrated methods like GPTQ and AWQ hold 99 percent of BF16 accuracy at 70B plus, with smaller models recovering 95 to 98 percent. My rule: quantize the target to FP8 first for the guaranteed win, then add speculation for the interactive routes where batches stay small. Inference-speed work on Cerebras hardware shows the same stack-ranking instinct on fixed-function silicon: measure the whole pipeline, then optimize the binding constraint.

Load-test notes from our test cluster

When we deployed speculation on our test cluster with mixed chat and code traffic, per-domain acceptance diverged fast: 68 percent on chat, 41 percent on code, 22 percent on math. One global draft length of five served nobody well. I split configs by route and set acceptance alerts at 15 percent below each route's baseline. In our testing at SaaSNext across forty thousand requests, the alerts caught two model-swap regressions within an hour each. Drafters rot when targets change. Re-measure acceptance on every model bump. Token-routing work that shifts spend to Opus applies the same per-route discipline to model selection, and the two routers compose.

When NOT to use this pattern

High-batch bulk inference should skip speculation entirely. Saturated GPUs gain nothing and pay the verification tax. Latency-insensitive offline jobs care about tokens per dollar, not per second, so FP8 plus big batches wins without drafters. Tiny models on weak hardware can lose more to draft overhead than they gain. And teams without acceptance-rate dashboards should not enable speculation at all: blind speedups rot into blind slowdowns.

Production checklist before you ship

Measure acceptance per domain before enabling anything. Set speculative tokens from data: five default, seven for aligned code, three for weak domains. Route by batch size with speculation on under batch 16 and off above 32. Quantize targets to FP8 first. Alert on acceptance drops of 15 percent. Re-measure on every model or drafter change. Log speedup per route weekly and kill configs that fall under 1.0x.

Start with one interactive route. Measure acceptance. Then expand.

By Deepak Bagada, Founder and Editor-in-Chief at Daily AI World.