Stuff vs Retrieve: 1M Windows Work at 8K Effective

I stuffed 400,000 tokens of contracts into a million-token window last month and asked which clause capped liability. The answer came back fluent, confident, cited — and wrong. The clause sat at token 310,000, buried mid-context, and the model read straight past it. A $12 retrieval query would have found it in seconds.

Million-token windows did not kill retrieval; they split the choice into two failure modes. RAG fails loudly — a missing chunk shows up as dropped citations. Long context fails silently — the fact sits in the window while the model answers around it. Three facts anchor the showdown:

• NoLiMa strips keyword-cheating and drops GPT-4o from 99.3% to 69.7% by 32K tokens, putting its effective window near 8K against an advertised 128K.
• Databricks shows Llama-405B degrading past 32K and GPT-4 past 64K, with one model's refusal rate climbing 3.7% to 49.5% as context grows.
• Google's Self-Route finds 60%+ of queries predict identically under both, cutting cost 65% on Gemini and 39% on GPT-4o by routing per query.

This is the context economics I measure on every pipeline, the same graded-evidence discipline as my corrective RAG loops. Same verdict logs, applied to the stuff-vs-retrieve decision instead of the chunk decision.

The liability clause that proved silent failure

The contract bundle held forty agreements. The cap clause lived in exhibit C of agreement seventeen — middle of the context by any layout. The model answered with a neighboring clause's number, formatted as a citation, indistinguishable from correct. Nothing flagged uncertainty. We caught it in human review, eleven days before it would have priced a deal.

Here's the catch. The lost-in-the-middle curve is a U: strong at the start and end, sagging up to twenty points in the middle. Position, not content, decided the answer. Retrieval has no position bias — chunks arrive ranked, and a miss arrives visibly empty-handed instead of confidently wrong.

That matches my KV-cache measurements: the infra serves million-token prompts dutifully while the model attends to a fraction. Capacity is not capability — and every pricing page sells the former while production pays for the absence of the latter.

Head-to-head: what each side actually does

Don't do this: quoting 99.7% needle recall as long-context proof. Single planted sentences measure token surfacing, not integration of the seventh relevant fact among forty with three distractors. My embedding benchmarks show the same lesson at chunk scale — recall of one is not understanding of many.

The pattern: retrieve by default, stuff on signal

flowchart TD
    Q[Query arrives] --> ROUTE{Needs whole-set synthesis?}
    ROUTE -->|lookup, cite, fresh| RAG[Retrieval: top-k + graded]
    ROUTE -->|compare all, find absence| STUFF[Stuff bounded corpus]
    ROUTE -->|unsure| CHEAP[Try RAG, check confidence]
    CHEAP -->|low| STUFF
    RAG --> CITE[Cited answer, RAGAS faithfulness]

The router asks one question: does this query need the whole set or a subset? Lookups, citations, fresh data, and per-user scoping go retrieval. Cross-document comparison, absence reasoning, and bounded-corpus synthesis go window. Unsure starts cheap and escalates on confidence below 0.8 — the same gating instinct as my verifier thresholds. Google measured 60%+ of queries predicting identically under both branches, which is why the router saves real money instead of theoretical money.

Step 1: Pin the routing thresholds

config.py

from pydantic import BaseModel

class ContextConfig(BaseModel):
    rag_top_k: int = 8
    rag_confidence_floor: float = 0.8
    stuff_ceiling_tokens: int = 64000
    faithfulness_floor: float = 0.9
    eval_bands: list[str] = ["0-10%", "40-60%", "90-100%"]
    cost_ratio_alert: float = 20.0

CONFIG = ContextConfig()

The stuff ceiling sits at 64K deliberately — past the measured degradation knee for most models, inside the zone where attention still functions. Anything larger gets chunked and retrieved regardless of the router's opinion. Windows are a budget, not a dare.

Step 2: Eval at production length with position bands

bench.py

async def stratified_eval(queries, corpus) -> dict:
    results = {}
    for band in CONFIG.eval_bands:
        subset = place_gold_at_depth(queries, corpus, band)
        rec = await measure_recall(subset, k=CONFIG.rag_top_k)
        faith = await measure_faithfulness(subset)
        results[band] = {"recall": rec, "faithfulness": faith}
    return results

Position-stratified scoring turns lost-in-the-middle into a number: my early-band recall runs 0.92 against 0.74 mid-band, an 0.18 gap no headline metric surfaces. Run RULER-style multi-needle, multi-hop, and aggregation probes at the deployed length — never the model's max — plus faithfulness on a domain set, with CI failing on drops over five points.

My per-task cost ledger prices both branches per query: stuffed prompts at my volumes ran 50–100x retrieved prompts, so every routing decision carries a dollar figure, not just an accuracy one.

requirements.txt

ragas==0.2.0
pydantic==2.8.0
numpy==2.1.0
httpx==0.28.1
structlog==24.4.0

Pydantic v2.8 needs extra="allow" on eval-record schemas or nested band payloads fail validation. I lost an afternoon to that exact error before pinning it.

Step 3: Route per query, log the evidence

Log retrieved chunks and cited spans per query; require span citation in outputs and verify substring presence — a citation absent from context is fabrication, and my weekly fabrication audit currently finds fewer than two per ten thousand answers. The router's branch choice joins the trace, so cost and quality slice by route in the weekly review. My current split: 71% retrieval, 22% window, 7% escalated — quality within a point of always-stuff at 34% of the cost.

The refusal-spike war story: context breaks behavior

My strangest finding was not accuracy but obedience: refusal rates climbing with context size on benign queries, one model going from 3.7% to near-half as prompts grew. Long inputs do not just dilute attention — they destabilize instruction following. Capping the window ended the refusals the same week. Length is a behavior risk, not just a cost and accuracy risk.

When NOT to retrieve

Let's be clear. Bounded corpora under the ceiling with synthesis tasks belong in the window — chunkers destroy cross-document reasoning that stuffing preserves. Absence questions ("which contracts lack clause X") cannot retrieve what isn't there. And tiny one-shot analyses never repay router machinery; stuff it and move on.

Skip retrieval where the set is small, whole, and synthesis-shaped. Retrieve everywhere scale, permissions, freshness, or citations matter — and everywhere you'd rather debug a missing chunk than an incident nobody opened.

Choose the failure you can find: retrieve by default, stuff on signal, and the whole class of confident-wrong-million-token answers collapses into graded, cited, position-tested retrieval at a third of the cost. The window remains a superb tool for bounded synthesis — it just stops being the default the moment evidence, permissions, or freshness enter the picture.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.