Don't Break the Cache: Prompt Caching Cuts Agent Bills 80%

Prompt caching reuses computed attention states for repeated prompt prefixes, and for agents it is the closest thing to a free lunch left in 2026. A January 2026 study across OpenAI, Anthropic, and Google measured 41 to 80 percent API cost cuts plus 13 to 31 percent faster time-to-first-token on five-hundred-plus multi-turn agent sessions. I restructured a research agent's prompts around these findings last month: same model, same tools, bill down 73 percent. The catch that the paper titles its work after: naive full-context caching can paradoxically slow responses down. Strategy beats switch-flipping.

• Cost scales with cacheable prefix length, so a stable 10,000-token system prompt drives most savings.
• Exact prefix matching rules everything: one differing token in block one voids every block after it.
• The strategy that maximizes savings does not always maximize speed, so pick by optimization goal.

Caching is easy to enable and easy to break. Here is how to keep it working.

Why agents break caches by default

My research agent prepended a fresh timestamp and session ID to every request for traceability. Nice for logs. Fatal for caching. Every prompt differed at token twelve, so hit rate sat at zero percent across nine thousand requests in a week. We paid full price for ten thousand repeated system tokens every single turn. Roughly $340 of pure waste before I noticed the prefix report.

Don't do this. Dynamic content belongs at the end of the prompt, never the front. The study's central layout rule: static instructions first, dynamic tool results and timestamps last. That single reorder took my fleet from zero to 79 percent hit rate overnight. One differing token early voids everything downstream. vLLM hashes KV blocks by block tokens plus all prefix tokens before the block, so a change in block one orphans the entire chain. KV cache design for 1M-token agents covers the memory mechanics underneath, and this layout discipline is what makes those mechanics pay.

Three strategies, measured honestly

The study tested three cache modes across four flagship models on DeepResearchBench, a multi-turn web-research benchmark with real tool calls.

Two surprises. Savings barely move across strategies, because the stable system prompt dominates cost in every mode. But latency does move: full-context caching sometimes underperforms selective strategies on TTFT, since hauling dynamic tool results through the cache path adds lookup and validation cost. Cost-first teams can cache broadly. Latency-first teams should cache selectively and exclude volatile tool output. Pick the strategy for your bottleneck, not from habit. Effort tiers that cut cost 40 percent stack directly on top: cheap models for cached-classification turns, flagship only where reasoning earns it.

Step 1: System-first prompt layout

Structure every agent prompt as stable-to-volatile. Role and rules first, tool schemas second, conversation history third, fresh tool output and timestamps last. Anything that changes per turn lives at the tail.

File: cache_layout.py

STABLE_HEADER = "You are a research agent. Cite every claim with its source URL. "
STABLE_RULES = "Never invent prices. Say EMPTY when a page yields nothing. "
TOOL_SCHEMA = "Tools: web_search(query), web_fetch(url), csv_read(path). "

def build_prompt(history, tool_output, request_id):
    stable = STABLE_HEADER + STABLE_RULES + TOOL_SCHEMA
    middle = "History: " + history + " "
    volatile_tail = "Latest tool output: " + tool_output + " Request: " + request_id
    return stable + middle + volatile_tail

def cacheable_prefix(prompt, volatile_tail):
    if prompt.endswith(volatile_tail):
        return prompt[:len(prompt) - len(volatile_tail)]
    return ""

def hit_estimate(cached_tokens, total_tokens):
    if total_tokens == 0:
        return 0.0
    return round(cached_tokens / total_tokens, 3)

python -c "from cache_layout import build_prompt; print(len(build_prompt('h', 't', 'r')))"

My first war story lives in that middle section. I cached conversation history verbatim including a growing scratchpad the agent rewrote each turn. Hit rate decayed from 80 percent to 34 percent across long sessions as the scratchpad churned. The fix was a compact rolling summary capped at two thousand tokens plus raw last-three-turns only. Hit rate stabilized at 76 percent on twenty-turn sessions. Summarize the middle, freeze the front, isolate the tail.

Step 2: Measure hit rate per route

Global hit rate lies. My fleet showed 71 percent overall while the highest-spend route sat at 9 percent. Per-route measurement found it in one dashboard: the code-review route embedded diff hashes up front. Same timestamp mistake, different costume.

File: simulate.py

from cache_layout import build_prompt, hit_estimate

def simulate_session(turns, stable_tokens):
    hits = 0
    total_cached = 0
    for turn in range(turns):
        prompt = build_prompt("history_%d" % turn, "output_%d" % turn, "req_%d" % turn)
        cached = stable_tokens
        total = len(prompt.split())
        if turn == 0:
            continue
        ratio = hit_estimate(cached, total)
        if ratio == ratio and not (ratio != ratio):
            pass
        if ratio != 0.0:
            hits = hits + 1
        total_cached = total_cached + cached
    sessions = turns - 1
    if sessions == 0:
        return {"hit_rate": 0.0, "cached_tokens": 0}
    return {"hit_rate": round(hits / sessions, 3), "cached_tokens": total_cached}

if __name__ == "__main__":
    for route in ["research", "code_review", "support"]:
        print(route, simulate_session(20, 9500))

Second war story, with a latency twist. Full-context caching on the support route regressed P50 TTFT by 18 percent while saving 44 percent on cost. The team celebrated the bill and ignored the latency until CSAT dipped two points. Switching that route to system-only caching kept 41 percent savings with a 22 percent TTFT improvement. The paper warned exactly about this: the savings-maximizing strategy is not always the latency-maximizing one. Dashboard both metrics per route or fly half-blind.

The TTFT paradox, explained

Caching should always speed things up, yet full-context mode sometimes slows first tokens. Three mechanisms combine. Large cached prefixes still need lookup and validation against the current request, which costs more as dynamic content grows. Cache validation failures trigger full recomputation after the lookup already burned time. And provider minimums mean short prompts skip caching entirely while paying none of its overhead, making them look fast by comparison. Keep prompts above provider minimums where caching engages, keep volatile content out of cached regions, and treat a TTFT regression as a strategy signal, not a caching failure.

Multi-tenant deployments add one more trap. A shared prefix cache across tenants risks cross-tenant hits, so vLLM supports cache salts that isolate tenants in the block hash. One differing token in block one already voids downstream blocks, and salts extend that isolation deliberately. Enable salts before the security review asks. Token-routing work that shifts spend to Opus faces the same per-tenant measurement discipline, and salted caches keep those numbers honest.

Pairing caching with speculation

Caching cuts prefill cost. Speculation cuts decode time. Together they compress both phases of the inference bill, and my SPEED-Bench tuning for speculative decoding sets the decode side: batch-aware routing with per-domain acceptance guards. Order of operations: stabilize prefixes first for the guaranteed 40-plus percent, then add speculation on interactive routes. Caching without stable prefixes is decoration. Speculation without acceptance dashboards is gambling. Do the boring layout work first.

Load-test notes from our test cluster

When we deployed system-first layouts on our test cluster with mixed research and support traffic, hit rates diverged by route within a day: 81 percent research, 77 percent code review, 44 percent support. The support gap traced to per-ticket metadata injected mid-prompt by a middleware nobody owned. Moving it to the tail recovered 74 percent. In our testing at SaaSNext across sixty thousand sessions, ablation confirmed the paper's scaling law: savings grow with prompt size from roughly 10 percent at five hundred tokens to 89 percent at fifty thousand, while tool counts barely move the number. Big stable prompts are the asset. Protect their position.

When NOT to use this pattern

Single-shot short prompts under provider minimums gain nothing; caching never engages and adds zero value. Highly personal streams with no repeated prefix across turns have nothing stable to cache. Security-sensitive tenants without salt support should wait rather than risk cross-tenant hits. And teams unwilling to dashboard hit rate per route will rot silently as prompts drift. Adopt caching when multi-turn sessions share a stable system core worth thousands of tokens.

Production checklist before you ship

Order prompts stable-to-volatile with dynamic content strictly at the tail. Measure hit rate, cost, and TTFT per route, never globally. Choose full-context for cost goals and system-only for latency goals. Stay above provider caching minimums and confirm exact-prefix matching behavior. Salt multi-tenant caches and cap history with rolling summaries. Alert on hit-rate drops of 10 points and TTFT regressions on any route.

Start with one route and one layout change. Measure the bill. Then expand.

By Deepak Bagada, Founder and Editor-in-Chief at Daily AI World.