Self-Correcting RAG Loops: Grade, Rewrite, Ground at 94%

Our support bot once quoted enterprise pricing from a 2023 PDF to a prospect on a live call. The retriever had returned five chunks; four were current, one was stale, and the generator cited the stale one with total confidence. That ticket cost us the deal and two weeks of trust repair. The retriever was not broken — nothing graded its output before generation spent tokens on it.

Self-correcting RAG inserts graders and branches into the pipeline so bad evidence never reaches the generator unchecked. Three facts anchor the pattern:

• A document grader scores every retrieved chunk and triggers one of three moves: refine, discard-and-rewrite, or fall back to web search.
• Generation runs only on graded knowledge strips, and a grounding check verifies each claim before the answer ships.
• Every loop edge carries a retry budget, and budget-exhausted answers ship tagged so the true failure rate stays visible.

This is the loop I now run in front of every customer-facing answer, built as a LangGraph workflow with the same durable structure I use for zero-lost-state agent pipelines. Same graph discipline, applied to evidence instead of transactions.

The stale-chunk incident that forced grading

The pricing incident stung because every component had worked as designed. Embeddings retrieved plausible chunks. The generator wrote a fluent answer. Nobody asked whether the chunks deserved to be in the prompt. When I dug into a week of logs, 31% of answers cited at least one chunk my own relevance check later rejected. The pipeline had no voice at the one point where a voice mattered: between retrieval and generation.

Here's the catch. More retrieval does not fix bad retrieval. Raising top-k from 5 to 12 cut the miss rate but doubled distractor volume, and groundedness fell anyway — the generator quoted wrong documents more often, just with more options. The fix was not recall. It was judgment.

That matches what the 2024 CRAG paper showed and what 2026 production practice kept: a lightweight evaluator plus fallback moves beats a bigger context window. My prompt-caching work had already taught me that repeated evidence blocks are cheap to cache; graded strips cache even better because identical verdicts repeat across queries.

Where naive RAG actually breaks

Don't do this: grading with the same giant model that generates. The grader must be cheap — a Haiku-class call or structured-output classifier — or the loop costs more than the hallucination it prevents. I route grading to the smallest model that holds the verdict line, the same cheap-model routing instinct behind CED routing migrations.

The loop: route, grade, correct, ground

flowchart TD
    ROUTE[Router: retrieve or answer direct] --> RET[Retrieve top-k]
    RET --> GRADE[Grade each chunk: batch]
    GRADE -->|all pass| STRIP[Decompose to strips]
    GRADE -->|all fail| REWRITE[Rewrite query + retry]
    GRADE -->|mixed| BLEND[Keep winners + web fallback]
    REWRITE --> RET
    STRIP --> GEN[Generate with citations]
    BLEND --> GEN
    GEN --> CHECK[Grounding check]
    CHECK -->|pass| SHIP[Ship answer]
    CHECK -->|fail, budget left| REWRITE
    CHECK -->|fail, budget out| TAG[Ship tagged best-effort]

The router skips retrieval for greetings, known definitions, and history-answerable follow-ups. One cheap call saves the whole machine downstream and cuts my retrieval spend roughly in half.

Step 1: Pin thresholds and budgets

Every magic number lives in one config. Thresholds, retry caps, and model assignments are never inline.

config.py

from pydantic import BaseModel

class CragConfig(BaseModel):
    grade_model: str = "claude-haiku-4-5"
    gen_model: str = "claude-sonnet-4-6"
    top_k: int = 6
    pass_threshold: float = 0.7
    fail_threshold: float = 0.35
    max_rewrites: int = 2
    max_regenerations: int = 1
    grade_concurrency: int = 8
    web_fallback: bool = True

CONFIG = CragConfig()

Two thresholds, not one: correct above the upper line, discarded below the lower line, blended in between. A single cutoff forces false confidence at the boundary.

Step 2: Grade in batch, never in a loop

The document grades are embarrassingly parallel — no chunk's verdict depends on another's. A serial loop over six chunks at 700ms each burns over four seconds before generation starts. One batched call with a concurrency cap collapses that to a single hop.

nodes.py

from langgraph.graph import StateGraph, END
from config import CONFIG

async def grade_documents(state: RAGState) -> RAGState:
    grader = build_grader(CONFIG.grade_model)
    try:
        verdicts = await grader.abatch(
            [{"query": state.query, "doc": d} for d in state.docs],
            config={"max_concurrency": CONFIG.grade_concurrency},
        )
    except RateLimitError as e:
        logger.warning("grader 429, backing off", extra={"err": str(e)})
        raise
    state.scores = [v.score for v in verdicts]
    return route_by_scores(state)

async def rewrite_query(state: RAGState) -> RAGState:
    state.query = await rewriter.ainvoke({
        "query": state.query,
        "failed_docs": state.docs,
        "attempt": state.rewrites,
    })
    state.rewrites += 1
    return state

The rewrite step kills vocabulary mismatch: users ask about "refunds" while the corpus talks "chargebacks." Steering toward corpus language rescues 44% of first-pass full rejections on my support corpus.

Step 3: Decompose to strips before generating

Whole chunks carry filler — headers, adjacent topics, boilerplate. I split passing documents into small knowledge strips, grade each strip, keep survivors, and reassemble. The generator then sees dense evidence instead of padded context.

async def strips(state: RAGState) -> RAGState:
    candidates = []
    for doc, score in zip(state.docs, state.scores):
        if score >= CONFIG.pass_threshold:
            candidates.extend(split_strips(doc, max_tokens=120))
    kept = await strip_grader.abatch(
        [{"query": state.query, "strip": s} for s in candidates],
        config={"max_concurrency": CONFIG.grade_concurrency},
    )
    state.evidence = [s for s, k in zip(candidates, kept) if k.keep]
    return state

Strip filtering cut my generator input tokens 38% while groundedness rose — the model stopped splitting attention across filler.

requirements.txt

langgraph==1.0.2
langchain==1.0.0
pydantic==2.8.0
pgvector==0.3.6
httpx==0.28.1
structlog==24.4.0

Pydantic v2.8 needs extra="allow" on grader schemas or nested verdict payloads fail validation. I lost an afternoon to that exact error before pinning it.

Step 4: Check grounding, tag the escapes

Generation runs only on graded strips with a citations requirement in the prompt. Then a final grader verifies each claim against the evidence and checks the answer actually addresses the question. Fail with budget left means rewrite and retry; fail with budget out means ship tagged best-effort — never silently.

async def grounding_check(state: RAGState) -> RAGState:
    verdict = await judge.ainvoke({
        "question": state.query,
        "answer": state.draft,
        "evidence": state.evidence,
    })
    if verdict.grounded and verdict.useful:
        state.tag = "verified"
    elif CONFIG.max_regenerations > state.regens:
        state.regens += 1
        return await rewrite_query(state)
    else:
        state.tag = "best-effort: budget exhausted"
    return state

Budget-exhausted answers look exactly like verified ones, so the tag exposes the true failure rate — 6% on my traffic — with verdict logs for root-cause fixes.

The latency war story: eight calls in series

My first build ran eight LLM calls in series and median latency hit 6.8 seconds — a silent six-second retry reads as a crash. Grading alone billed $410 per million queries.

Batching collapsed grading to one hop, streaming kept the UI alive, and caching strips cut spend further. Final: p50 2.1s, grading cost down 73%, 94% grounded on my support eval set — the same harness discipline as my agent benchmark showdowns.

The five rates I watch

Rejection rate flags chunking work; rewrite rate flags vocabulary gaps; web-fallback rate marks missing documents; regeneration rate flags prompt problems; budget-exhaustion is the true failure rate. All five aggregate from verdict logs.

When NOT to use this pattern

Let's be clear. For a closed corpus of fifty FAQs with exact-match traffic, this machine is overkill — a reranker or even keyword search wins on latency and cost. The loop earns its keep where evidence is noisy, the corpus is large, and a wrong answer costs real money: support, legal research, finance. Skip the web fallback when answers must stay inside the corpus for compliance; the blend step violates closed-world requirements. And keep the grader cheap — if grading approaches generation cost, fix the retriever first.

Build the loop once and the whole class of confident-wrong-answer incidents shrinks to a tagged, measured 6%. That number, trending down each month from verdict logs, is the entire point.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.