Serve 1M-Token Agents Without Melting GPUs: KV Cache Design

Every agent turn replays context. Without cache engineering, a 1M-token loop recomputes a million positions per decode step and your GPUs melt while finance watches. With it, DeepSeek compressed KV footprints from 389KB per token in V1 to 890 bytes in V4.1 Flash, a 437x fall across four generations.

I run Daily AI World and serve long-context agents at SaaSNext. Direct answer:

• Paged attention ends fragmentation waste, the vLLM-era baseline every server needs
• Prefix-cache routing sends same-prefix requests to the same GPUs, turning rereads into hits
• MLA plus FP8 compresses what gets stored, halving prefill compute on new architectures

Here is the systems design that makes million-token agents affordable.

The KV math that rules serving cost

Attention stores keys and values per token per layer. A 70B-class dense model at BF16 needs roughly 1MB per token across layers. One million tokens means one terabyte of KV per concurrent sequence. No single GPU holds that, so serving splits, offloads, or recomputes, and each path costs latency or hardware.

Four techniques stack. Paged attention manages KV in blocks like virtual memory, killing fragmentation that once wasted 60% of capacity. Prefix caching reuses computed KV for shared prompt prefixes across turns and users. Multi-head latent attention compresses KV into learned latents, the design behind DeepSeek's 8x generation-over-generation cuts. Quantization to FP8 halves bytes with tuned quality gates. Combined, they move 1M-context agents from exotic to routine.

The CED architecture breakdown with 890-byte caches shows the frontier: encoder-only prefill at 8B active parameters with decoder KV projected from encoder states. Prefill complexity roughly halves. That is architecture doing what infrastructure used to.

Production war story 1: the round-robin router that cached nothing

In our first 1M-context deployment we load-balanced round-robin across 8 GPUs. Hit rate sat 11%. Every turn recomputed full prefixes on cold GPUs while cached blocks expired unused elsewhere. P99 prefill hit 14 seconds and users called the agent frozen. The GPUs were fine. The routing was the bug.

We switched to prefix-hash routing: hash the shared system plus repo prefix, pin those requests to the same GPU group. Hit rate jumped to 83% overnight. P99 prefill fell to 2.1 seconds. Same hardware, same model. Lesson: cache without affinity is decoration. My thinking-effort analysis showing 60 to 70% thinking share compounds this: reasoning loops replay the most context, so they gain the most from prefix hits.

Production war story 2: the timestamp that poisoned every prefix

When we added wall-clock timestamps to system prompts for freshness, hit rates collapsed from 83% to 6%. One changing token at position 40 invalidated every cached block after it. The team blamed the model upgrade that shipped the same week. Two days of bisecting proved the timestamp guilty. A single dynamic prefix token cost roughly $1,400 in extra compute before we caught it.

Fix: move dynamic fields after the static bulk, or keep timestamps out of cached regions entirely. Structure prompts as static system, stable repo context, then dynamic turn data last. Shared prefixes must be byte-identical to hit. Audit prompt templates for clocks, UUIDs, and counters weekly. The spend-routing discipline from the Fable analysis applies: measure cache-hit rate on the same dashboard as dollars, because one drives the other.

Runnable production code: prefix-aware router

Hash shared prefixes, pin to GPU groups, monitor hits.

File 1: config.py

from pydantic_settings import BaseSettings
from pydantic import Field

class Settings(BaseSettings):
    gpu_groups: int = Field(default=4, alias="GPU_GROUPS")
    prefix_len: int = Field(default=4000, alias="CACHE_PREFIX_LEN")
    hit_target: float = 0.80
    alert_below: float = 0.60

    class Config:
        extra = "allow"

settings = Settings()

File 2: router.py

import hashlib, logging
from config import settings

log = logging.getLogger("kv-route")

def prefix_key(system: str, context: str) -> str:
    # Only the stable head participates in affinity
    head = (system + context)[:settings.prefix_len]
    return hashlib.sha256(head.encode()).hexdigest()[:16]

def assign_group(key: str) -> int:
    return int(key, 16) % settings.gpu_groups

def check_hits(hits: int, total: int) -> dict:
    rate = hits / max(total, 1)
    status = "healthy"
    if settings.alert_below > rate:
        status = "ALERT: affinity broken, audit prompt templates"
    elif settings.hit_target > rate:
        status = "watch: below target"
    return {"hit_rate": round(rate, 3), "status": status}

if __name__ == "__main__":
    k = prefix_key("system v7", "repo snapshot 8841")
    print("group", assign_group(k))
    print(check_hits(830, 1000))
    print(check_hits(410, 1000))

File 3: requirements.txt

vllm==0.9.0
pydantic==2.8.0
pydantic-settings==2.5.0
redis==5.2.1

Run it:

uv pip install -r requirements.txt
python router.py

Step 1: enable prefix caching in vLLM with paged attention defaults. Step 2: route by stable-prefix hash, never round-robin for stateful agents. Step 3: alert below 60% hits and audit templates for dynamic prefix poison. Voice workloads gain identically: the Gemini Live background-tool pattern replays dialogue context every turn, which is pure prefix-cache fuel.

Eviction and compaction policy for endless runs

Caches fill, then choices hurt. FIFO eviction drops the system prompt head that everything shares, collapsing hit rates to zero in one sweep. LRU keeps recent turns but starves the stable head under bursty multi-user load. Priority pinning wins for agents: lock the static system plus repo head, apply LRU only to the dynamic tail. Our 1M-token support agent pins 4k head tokens and pages the rest, holding 81% hits at 40 concurrent sessions where naive LRU managed 34%. Compaction runs nightly, merging fragmented blocks and dropping dead sessions older than 7 days. Monitor pinned bytes per GPU. Pinned heads that grow past 8k start starving diverse traffic, so cap and page overflow. The rule is simple: pin what repeats, evict what ages, measure both.

Quantization validation protocol before FP8

FP8 halves bytes but shifts numerics. Validate in four steps. First, replay 200 golden turns at BF16 and record per-turn scores. Second, repeat at FP8 with identical seeds. Third, diff per-turn deltas and flag any turn dropping over 2 points. Fourth, gate dtype promotion on zero flagged turns in the hardest class. Our FAQ fleet passed in one round. The 38-hour math chain flagged 6 turns with 3 to 5 point drops, all in late-chain symbolic steps where error compounds. That fleet stays BF16 until per-layer mixed precision lands. Document the dtype beside the model id in every artifact. Silent precision changes are silent quality changes, and evals cannot catch what they are not told to compare.

When NOT to chase cache rates

Do not cache across trust boundaries. Shared prefixes between tenants leak through timing. Isolate caches per tenant even at hit-rate cost.

Do not quantize KV without per-workload validation. FP8 holds on FAQ and coding, degrades on 38-hour math chains in our tests. Gate dtype changes on your hardest evals.

Do not let prefixes grow unbounded. Million-token shared heads pin GPUs and starve diversity. Cap stable heads near 4k tokens and page the rest.

Verdict for September 2026 serving teams

Page it, prefix it, route it, compress it. The models went long-context. The infrastructure decides who affords it.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World. I serve long-context agent fleets at SaaSNext and watch hit rates like revenue. More at https://deepakbagada.in.