Test-Time Compute Routing: Spend Tokens Where They Pay

I burned $740 in a week letting every query run best-of-16 with a judge rerank. Accuracy rose 1.1 points over greedy decoding. The invoice rose 900%. That ratio — nine times the tokens for one point — sent me into the 2026 test-time scaling literature, where the answer was waiting: route the compute, don't spray it.

Test-time compute routing treats generation, verification, and stopping as one shared budget, spending the next token where it pays most. Three facts anchor the approach:

• August 2026 CoBa routing matches best-of-16 majority voting within 0.01 accuracy points while using 58.9% fewer parameter-weighted tokens.
• The GRACE phase transition proves coarse verification wins low-budget easy problems while fine-grained wins hard ones — one fixed strategy is always wrong somewhere.
• Multi-sequence verifiers add 6% to best-of-64 selection while halving latency via streaming early-stop.

This is the inference economics I measure on every workload now, the same per-task lens as my price-per-task analysis. Same math, applied to reasoning spend instead of model choice.

The $740 week that proved uniform scaling wasteful

The workload mixed trivia lookups with competition math. Best-of-16 treated both identically: sixteen full candidates, judge rerank, argmax. The trivia answers agreed by candidate three every time — thirteen wasted generations per question. The math questions needed all sixteen and sometimes more. Uniform allocation overspent the easy 80% and underspent the hard 20% simultaneously.

Here's the catch. Test-time scaling is three competing knobs — sample more, think longer, verify harder — and a fixed recipe buys the wrong knob for most queries. My logs showed 71% of questions resolving by agreement within four samples; the remaining 29% consumed 84% of the budget. Routing by difficulty is not an optimization. It is the workload.

That matches my thinking-token measurements: reasoning spend without difficulty gating is the fastest-growing line on the agent bill.

The three regimes and when each wins

Don't do this: scaling N blindly. Best-of-N exhibits non-monotone behavior — hard argmax over growing candidate banks overoptimizes imperfect proxy scores, admitting false positives as N rises. I cap N per difficulty tier and distrust any selection the verifier cannot calibrate.

The judge caveat compounds it. LLM judges carry position and verbosity biases, so judge design is part of the method — a reranker that prefers long answers will happily spend your budget selecting them. My judge-reliability coverage applies verbatim: force tool-call verdicts, randomize order, distrust verbosity.

The pattern: estimate difficulty, allocate, stop early

flowchart TD
    Q[Query arrives] --> EST[Cheap difficulty estimate]
    EST -->|easy| SINGLE[Single pass, coarse check]
    EST -->|contested| SAMPLE[Sample 4-8, vote + light verify]
    EST -->|hard| DEEP[Long CoT + fine verify]
    SAMPLE --> AGREE{Early agreement?}
    AGREE -->|yes| STOP[Stop, ship consensus]
    AGREE -->|no| ROUTE[Route best to strong verifier]

Warm-up buys diversity first — a small fixed sample set — then cheap verification decides whether to stop, sample more, or escalate selected candidates to strong verification. Ground truth never enters the policy; agreement and calibrated confidence drive every branch.

Step 1: Pin budgets per tier

config.py

from pydantic import BaseModel

class TTCConfig(BaseModel):
    easy_max_tokens: int = 2000
    sample_n: int = 6
    hard_max_tokens: int = 16000
    agreement_stop: float = 0.85
    verifier_threshold: float = 0.75
    max_bank: int = 16
    difficulty_model: str = "claude-haiku-4-5"

CONFIG = TTCConfig()

The difficulty estimate itself must be cheap — a Haiku-class call or a heuristic over query features like length, domain keywords, and history of similar queries, never a reasoning pass. My heuristic tier (no model call at all) handles 40% of traffic correctly; the classifier handles the rest. Spending flagship tokens to decide how to spend tokens defeats the router. My estimator costs under 2% of total inference spend.

Step 2: Route generation against verification

router.py

async def answer(query: str, cfg=CONFIG) -> dict:
    tier = await estimate_difficulty(query)
    if tier == "easy":
        return await single_pass(query, cfg.easy_max_tokens)
    bank = await sample(query, n=cfg.sample_n)
    try:
        agreement, consensus = check_agreement(bank)
    except ParseError:
        agreement, consensus = 0.0, None
    if agreement >= cfg.agreement_stop:
        return {"answer": consensus, "spent": "sample-only"}
    best = await strong_verify(bank, threshold=cfg.verifier_threshold)
    if best is not None:
        return {"answer": best, "spent": "sample+verify"}
    return await long_cot(query, cfg.hard_max_tokens)

Early stopping on agreement is the highest-ROI branch: settled examples skip the entire evaluator stack. Streaming verifiers extend this to token-level stopping — halving latency by terminating decoding the moment any sequence crosses confidence, rather than generating sixteen full candidates to vote among.

requirements.txt

httpx==0.28.1
pydantic==2.8.0
numpy==2.1.0
structlog==24.4.0
python-dotenv==1.0.1

Pydantic v2.8 needs extra="allow" on verifier payload schemas or nested score objects fail validation. I lost an afternoon to that exact error before pinning it.

Step 3: Calibrate the verifier or it lies

Calibration decides both bottlenecks: selection quality and stopping safety. I track expected calibration error, Brier score, and threshold-precision curves — three numbers that say whether the verifier earns its stopping power or merely spends it confidently. My verifier calibration runs weekly over 1,000 labeled decisions — expected calibration error currently 0.06, Brier halved against the isolated-scoring baseline by judging candidates jointly instead of one by one. An uncalibrated verifier at 0.75 threshold stops early on wrong answers; a calibrated one earns the threshold.

Pair this with value-tier routing: difficulty estimation and model selection compose — cheap models for easy tiers, flagship reasoning reserved for routed-hard queries. Two routers, one budget.

The overoptimization war story: N=64 picks the liar

My first best-of-64 run with a reward-model reranker selected a beautifully formatted wrong answer — the verifier had learned length as quality, and sixty-four candidates gave it room to find the longest wrong one. Accuracy fell below greedy while cost rose 40x. Capping banks at sixteen, switching to calibrated consensus, penalizing verbosity in the judge prompt, and randomizing candidate order fixed it: selection now gains rather than gambles, and the rerank bill fell 70% alongside the accuracy recovery.

When NOT to route

Let's be clear. Latency-critical paths under a second should run greedy with a coarse check — routing deliberation costs more than it saves. Deterministic tasks with programmatic verifiers need best-of-N with execution checks, not difficulty theater. And tiny volumes never repay router tuning; spend the engineering on the verifier instead.

Skip it for hot paths and trivial volumes. Route where reasoning spend dominates the bill, where easy and hard queries share one endpoint, and where last month's invoice showed sixteen candidates answering trivia.

Treat generation, verification, and stopping as one budget, and the whole class of spray-and-pray inference spend disappears: best-of-16 accuracy at a third of the tokens, latency halved on settled queries, and verifiers calibrated enough to trust.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.