Build a Multi-Model Semantic Caching Router Workflow with Redis & LangGraph in 2026

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

The most expensive token in an enterprise AI budget is the one you pay for twice. Identical or near-identical requests — the same support question from three customers, the same summarization job re-run on slightly different timestamps, the same classification call from a retry loop — are the hidden tax on every agent fleet. Most teams attack it with exact-match caching, which catches exactly nothing in practice, because real traffic is approximately the same, not byte-identical. A customer asking "refund status?" and another asking "where is my refund?" produce two cache misses and two full model calls for one unit of work.

This workflow fixes that with semantic caching: it embeds every request, stores vectors in Redis, and serves any request within a similarity threshold from cache — turning approximate duplicates into cache hits. It is built as a multi-model router too: cache misses route to the cheapest model that meets the quality bar for the request class, and LangGraph orchestrates the lookup, the routing decision, the cache write-back, and the fallback chain. The result is a 40-60% reduction in inference spend on high-duplication workloads, measured and attested, not hoped for. It is the same cost-discipline pattern we run across our AI workflows library, and the model endpoints you will route to are catalogued in the MCP directory.

Architecture Overview

graph TD
  R[Incoming Request] --> E[Embed Request]
  E --> V[Redis Vector Search]
  V --> H{Cosine >= Threshold?}
  H -- yes --> C1[Serve Cached Answer]
  H -- no --> R1[Route by Request Class]
  R1 --> M1[Cheap Model]
  R1 --> M2[Mid Model]
  R1 --> M3[Frontier Model]
  M1 --> W1[Write-back to Cache]
  M2 --> W1
  M3 --> W1
  W1 --> O1[Return + Log Hit/Miss]
  C1 --> O1

The flow is simple to read and subtle to build: embed the request, search the vector index, and on a miss route intelligently instead of defaulting to the most expensive model. The write-back policy matters as much as the lookup — you cache high-confidence, deterministic outputs, and you never cache anything that depends on volatile context.

Part 1 — Embedding and cache primitives

.env

EMBED_MODEL=text-embedding-3-small
EMBED_DIM=1536
SIMILARITY_THRESHOLD=0.92
REDIS_URL=redis://cache.internal:6379
CACHE_TTL_SECONDS=86400
MODEL_ROUTES={"qa":"gpt-5.6-flash","extract":"claude-fable-5","hard":"claude-opus-5"}
MAX_CACHE_RETRIES=1

cache.py

import redis, numpy as np, json

r = redis.Redis.from_url(env("REDIS_URL"), decode_responses=False)

def embed(text: str) -> list[float]:
    resp = embed_client.embeddings.create(
        model=env("EMBED_MODEL"), input=text)
    return resp.data[0].embedding

def vec_bytes(v: list[float]) -> bytes:
    return np.asarray(v, dtype=np.float32).tobytes()

def semantic_lookup(text: str) -> dict | None:
    vec = embed(text)
    res = r.execute_command(
        "FT.SEARCH", "semantic_cache",
        f"*=>[KNN 1 @vec $B AS score]",
        "PARAMS", "2", "B", vec_bytes(vec),
        "RETURN", "3", "answer", "score", "id",
        "SORTBY", "score", "ASC", "DIALECT", "2",
    )
    if not res or len(res) < 2:
        return None
    doc = res[1]
    score = float(doc[1][3])
    if score >= float(env("SIMILARITY_THRESHOLD")):
        return {"answer": doc[1][1], "score": score}
    return None

def cache_write(text: str, answer: str, score: float = 1.0):
    vec = embed(text)
    r.execute_command(
        "FT.ADD", "semantic_cache", f"req:{abs(hash(text))}",
        "FIELDS", "vec", vec_bytes(vec),
        "answer", answer, "score", score,
        "NOSAVE")

The Redis Vector Similarity (RediSearch) configuration is the load-bearing part. semantic_lookup embeds the incoming request and runs a KNN search for the single closest vector, then applies the similarity threshold in the application layer — the threshold is policy, not engine. The cosine threshold of 0.92 is deliberately strict: for Q&A and extraction, a wrong cached answer is worse than a model call, so the cache only serves near-duplicates. The write-back path stores the embedding with the answer so future approximate duplicates hit without ever calling the model.

Part 2 — The routing graph

router.py

from pydantic import BaseModel, Field

class RequestClass(str):
    qa = "qa"
    extract = "extract"
    hard = "hard"

class RouteDecision(BaseModel):
    request_class: RequestClass
    model: str
    reason: str = Field(max_length=200)

async def classify_request(text: str) -> RouteDecision:
    # Cheap classifier: length + structure heuristics, model-assisted fallback
    if len(text) > 600 or "compare" in text or "explain" in text:
        return RouteDecision(request_class="hard",
                             model=env("MODEL_ROUTES")["hard"], reason="complex")
    if "json" in text.lower() or "extract" in text.lower():
        return RouteDecision(request_class="extract",
                             model=env("MODEL_ROUTES")["extract"], reason="structured")
    return RouteDecision(request_class="qa",
                         model=env("MODEL_ROUTES")["qa"], reason="default")

Routing is deliberately coarse: three buckets (qa / extract / hard), each mapped to a model tier. The magic is not in the classifier — it is in making the model choice explicit and logged. Every request records which tier served it and why, which is what turns the router into a cost-optimization instrument instead of a black box. Over time, the classify request can itself be a small model call, but start with heuristics: they are free, deterministic, and easy to audit.

Part 3 — The LangGraph orchestration

graph.py

from langgraph.graph import StateGraph, END
from typing import TypedDict

class RouterState(TypedDict):
    text: str
    cache_hit: bool
    answer: str
    model: str
    source: str  # "cache" | "model"

def lookup(state: RouterState) -> RouterState:
    hit = semantic_lookup(state["text"])
    if hit:
        state["cache_hit"] = True
        state["answer"] = hit["answer"]
        state["source"] = "cache"
    return state

def route(state: RouterState) -> RouterState:
    if state.get("cache_hit"):
        return state
    decision = asyncio.run(classify_request(state["text"]))
    state["model"] = decision.model
    state["answer"] = asyncio.run(call_model(decision.model, state["text"]))
    state["source"] = "model"
    cache_write(state["text"], state["answer"])
    return state

def route_cache(state: RouterState) -> str:
    return "end" if state.get("cache_hit") else "route"

g = StateGraph(RouterState)
g.add_node("lookup", lookup)
g.add_node("route", route)
g.set_entry_point("lookup")
g.add_conditional_edges("lookup", route_cache, {"route": "route", "end": END})
g.add_edge("route", END)
app = g.compile()

Retry rules: cache lookups never retry — a cache miss is a free event, just call the model. Model calls retry once with exponential backoff (1s, then 4s) for transient API errors, then fail over to the next tier in the route chain (hard → mid → cheap is never the direction; the fallback is cheap → mid → hard, so degraded days cost less, not more). Cache write-back failures are swallowed and logged: a missed cache write costs a future hit, never a correctness error. And because the cache is bounded by TTL (24h default), stale answers expire naturally — the same bounded-cache discipline we recommend across our AI workflows library.

Part 4 — Observability and the cost ledger

metrics.py

import json, time
from collections import defaultdict

ledger = defaultdict(lambda: {"hits": 0, "misses": 0, "cost": 0.0})

def record(state: RouterState, cost_per_call: float):
    key = state["source"]
    ledger[key]["hits" if state["source"] == "cache" else "misses"] += 1
    if state["source"] == "model":
        ledger[key]["cost"] += cost_per_call

def hit_rate() -> float:
    total = sum(l["hits"] + l["misses"] for l in ledger.values())
    hits = sum(l["hits"] for l in ledger.values())
    return hits / max(total, 1)

The metrics layer is what separates a cache from a cost center: it tracks hit rate, per-tier spend, and served-from-cache fraction on every request. With a 40-60% hit rate on duplicated workloads, the ROI math is immediate — each cache hit saves the full inference cost of the routed model call. Teams that deploy this workflow consistently report blended cost-per-request dropping by half on support, extraction, and classification lanes, which is the same unit-economics discipline we track on the latest AI news desk and apply across the MCP directory integrations.

Production checklist

1. Strict threshold first. Start at 0.92+ similarity and tune down only after you measure false-hit rate on your own traffic.
2. Never cache volatile context. Cache outputs only when the input alone determines the answer — no timestamps, user state, or live data baked in.
3. Fallback cheap → hard, never the reverse. Degraded days should cost less, not more.
4. Log the ledger. Hit rate, per-tier spend, and source are metrics, not trivia — the router is a cost instrument.
5. Bound the cache. TTL and eviction keep answers fresh and the vector index sized.

Frequently Asked Questions

Q: How is semantic caching different from exact-match caching?

A: Exact-match caching requires byte-identical requests and catches almost nothing in real traffic. Semantic caching embeds the request and serves any near-duplicate above a similarity threshold, so "refund status?" and "where is my refund?" hit the same cache entry.

Q: What similarity threshold should I start with?

A: Start strict at 0.92 cosine and tune down only after measuring false-hit rates on your own traffic. For Q&A and extraction, a wrong cached answer costs more than a model call, so err conservative.

Q: Why route cache misses to different models?

A: Because not every request needs the frontier model. Classifying requests into qa/extract/hard tiers and mapping each to the cheapest adequate model cuts blended cost per request on top of the cache savings.

Q: What happens when a cache write fails?

A: It is swallowed and logged. A missed write costs a future cache hit, never a correctness error — the cache is an optimization, and optimizations fail silently by design.

Q: Does the cache go stale?

A: Yes, by design: entries expire after the TTL (24 hours default), so answers refresh naturally. Never cache outputs that depend on volatile context, and the TTL bounds the worst-case staleness.