Build a Model-Routing Gateway Workflow for 1M-Token Agentic Models: Routing to NVIDIA Nemotron 3.5 Lightning

In August 2026, NVIDIA made the economics of long-context agentic work suddenly awkward. Nemotron 3.5 Lightning is an open-weight agentic model — 30B total parameters, only 3B active, running a hybrid Mixture-of-Experts architecture with up to 1M tokens of context — shipped in full-precision BF16 and quantized variants, deployable on OpenRouter, build.nvidia.com, NeMo Switchyard, and AWS SageMaker JumpStart. Read it the way platform teams read it: this is basically a local, cheap, 1M-context agentic worker. The moment a frontier 1M-context API call costs real money per prompt, the question stops being "can Lightning handle it?" and becomes "why are we sending everything to the expensive model?"

This article answers that question with a model-routing gateway: a LangGraph orchestration graph wrapped as a FastMCP server in Python that routes every agent subtask to the cheapest eligible model — long-context, repository-scale, cheap-parallel reasoning to Nemotron 3.5 Lightning; genuinely hard, multi-step, novel reasoning to a frontier model. You will get routing keys (cost, latency, context usage), A/B guardruns, a fallback chain, and hard budget caps, plus ownership of the surprises you will meet in production.

Why the 1M-Token Frontier Bill Is the Killer

Long-context agentic work has a cost curve that is brutal in arithmetic terms. A 1M-token request costs a fixed large sum per million input tokens — and agents loop. Every repository-scale task re-reads files, re-embeds symbol tables, and re-consults the same 1M-token corpus dozens of times. The wasted spend is not the single big call; it is the repeated toll on the same corpus.

Nemotron 3.5 Lightning changes the unit economics in three ways that matter operationally:

1. 1M-token context on your own iron. The 3B-active MoE keeps per-token cost low, so the full 1M-token window becomes an affordable working set instead of a risk line on the invoice.
2. Open weights mean predictable routing. You can serve it locally, or via NeMo Switchyard, which makes cost, latency, and throughput your variables instead of a vendor's.
3. Quantized variants for worker lanes. BF16 for the planner, 4-bit quant for bulk file-summarization workers; each lane pays only for the precision it needs.

The gateway below exploits exactly those three properties. Only genuinely hard reasoning — novel algorithm design, cross-cutting architecture decisions, adversarial debugging — should touch an expensive frontier model. Everything else, including almost all of the 1M-token heavy lifting, rides the cheap lane.

Architecture

The routing gateway sits inside the agent runtime: every tool call, plan step, and summarization request passes through it.

graph TD
    AG[Agent Runtime / LangGraph supervisor] -->|subtask + metrics| GW[Routing Gateway / FastMCP server]
    GW -->|routing keys / cost + latency + context| RT{Router / policy + guardrails}
    RT -->|long-context / cheap-parallel| NE[Nemotron 3.5 Lightning / BF16 1M ctx]
    RT -->|hard reasoning / novel| FR[Frontier Model / opus-class]
    RT -->|generative fallback| FB[Fallback Chain / BF16 -> quant -> frontier]
    NE -.stream tokens + usage.-> GW
    FR -.cost meters.-> GW
    FB -.health.-> GW
    GW -.every decision.-> LEDG[Routing Ledger / decisions + costs]
    LEDG -.budget caps / hard stop.-> GW
    subgraph AA[A/B Guardrun / shadow compare]
        RT -.shadow sample.-> NE2[Nemotron BF16 / challenger policy]
    end

Two structural choices matter. First, the router is a sidecar — agents do not bypass it, because their only path to a model endpoint is the gateway. Second, the gateway keeps a routing ledger: every decision, the keys that drove it, the cost it implied, and whether the choice would have won under the last A/B guardrun. That ledger is the audit trail for both money and model-quality claims.

Routing Keys

The router classifies each subtask along three keys:

The tunable rule of thumb: a subtask that wants more than roughly 120k context with a repository-scale reasoning profile (summarize, map symbols, diff, plan within known APIs) goes Lightning. A subtask under 40k context but with novel, multi-step reasoning (design a new cache strategy, resolve a subtle race) still gets the frontier router for one hop. Everything mid-tier is measured against the guardruns.

Configuration

# .env — routing gateway
NEMO_LIGHTNING_ENDPOINT=http://nemo-infer.internal:8443
NEMO_LIGHTNING_QUANT=http://nemo-quant.internal:8443
FRONTIER_ENDPOINT=https://api.frontier.example/v1
FRONTIER_KEY=sk-fr-2026-4f9a1c7e      # stored only in the gateway secrets file
FALLBACK_ORDER=nemo-bf16,nemo-quant,frontier
MAX_CONTEXT_ESTIMATE=1048576
LONG_CONTEXT_THRESHOLD=120000
HARD_REASON_FRONTIER=true
DAILY_BUDGET_CAP=120.00
ROUTING_LEDGER=./ledger/routing.jsonl
GUARDRUN_SAMPLE_RATE=0.05
BUDGET_HARD_STOP=true

Decision Schema

# schemas.py
from enum import Enum
from pydantic import BaseModel, Field


class Lane(str, Enum):
    NEMO_BF16 = "nemo.lightning.bf16"
    NEMO_QUANT = "nemo.lightning.quant"
    FRONTIER = "frontier"


class RoutingRequest(BaseModel):
    request_id: str
    agent: str
    task_type: str                     # plan | summarize | diff | design | debug
    context_estimate: int              # tokens, computed by the tokenizer
    reasoning_hardness: str = Field(...)  # low | medium | high | frontier
    declared_cost: float


class RoutingDecision(BaseModel):
    lane: Lane
    reason: str
    keys: dict[str, float]
    estimated_cost: float
    guardrun_id: str | None = None

The decision is itself a first-class event — the ledger rows are exactly RoutingDecision records serialized to JSONL, so budget accounting and A/B evaluation read the same bytes the router wrote.

Tool Layer: Model Endpoint Adapters

# tools.py
import httpx


class NemoAdapter:
    def __init__(self, endpoint: str, quant: bool = False):
        self.base = endpoint
        self.quant = quant
        self._long_read_timeout = 900   # genuine 1M-token reads take minutes

    def generate(self, messages: list[dict], context_total: int) -> dict | None:
        body = {
            "messages": messages,
            "context_budget": min(context_total, 1048576),
            "precision": "int4" if self.quant else "bf16",
        }
        try:
            with httpx.Client(timeout=self._long_read_timeout) as c:
                r = c.post(f"{self.base}/v1/chat/completions", json=body)
                r.raise_for_status()
                data = r.json()
                return data or None            # empty result triggers fallback
        except httpx.TimeoutException:
            return None


class FrontierAdapter:
    def __init__(self, endpoint: str, key: str):
        self.endpoint, self.key = endpoint, key

    def generate(self, messages: list[dict]) -> dict | None:
        try:
            with httpx.Client(timeout=600) as c:
                r = c.post(f"{self.endpoint}/v1/chat/completions",
                           headers={"Authorization": f"Bearer {self.key}"},
                           json={"messages": messages})
                r.raise_for_status()
                return r.json() or None
        except httpx.TimeoutException:
            return None

Two operational details matter: the 900-second timeout for genuine 1M-token work (a premature timeout is the most common false negative in this space), and streaming the returned usage statistics back to the ledger so the budget cap sees true token counts, not estimates.

The LangGraph Router

# orchestrator/graph.py
import os
from langgraph.graph import StateGraph, END
from schemas import Lane, RoutingDecision, RoutingRequest
from tools import FrontierAdapter, NemoAdapter
from budget import ledger_append

COST_PER_1M = {"nemo": 0.02, "nemo_quant": 0.008, "frontier": 12.0}
LONG_CTX_THRESHOLD = int(os.getenv("LONG_CONTEXT_THRESHOLD", "120000"))

NEMO = NemoAdapter(os.getenv("NEMO_LIGHTNING_ENDPOINT"))
NEMO_Q = NemoAdapter(os.getenv("NEMO_LIGHTNING_QUANT"), quant=True)
FR = FrontierAdapter(os.getenv("FRONTIER_ENDPOINT"), os.getenv("FRONTIER_KEY"))


def extract_keys(state: dict) -> dict:
    req = state["request"]
    state["keys"] = {
        "context": req.context_estimate,
        "hardness": req.reasoning_hardness,
        "est_cost": req.context_estimate / 1_000_000 * COST_PER_1M["frontier"],
    }
    return state


def decide(state: dict) -> dict:
    req, keys = state["request"], state["keys"]
    if req.context_estimate >= LONG_CTX_THRESHOLD and req.reasoning_hardness in ("low", "medium"):
        lane, reason = Lane.NEMO_BF16, "long-context cheap lane"
    elif req.reasoning_hardness in ("high", "frontier"):
        lane, reason = Lane.FRONTIER, "hard reasoning required"
    else:
        lane, reason = Lane.NEMO_QUANT, "quant worker lane"
    state["decision"] = RoutingDecision(lane=lane, reason=reason, keys=keys,
                                        estimated_cost=keys["est_cost"],
                                        guardrun_id=maybe_guardrun())
    return state


def run_lane(state: dict) -> dict:
    d = state["decision"]
    out = run_adapter(d.lane, state["request"], state["messages"])
    state["output"] = out
    if out is not None:
        ledger_append(state["request"], d, out)   # the routing ledger
    return state


def fallback_needed(state: dict) -> bool:
    return state.get("output") is None


def fallback(state: dict) -> dict:
    for lane in (Lane.NEMO_QUANT, Lane.FRONTIER):
        if lane == state["decision"].lane:
            continue
        out = run_adapter(lane, state["request"], state["messages"])
        if out is not None:
            state["output"] = out
            state["decision"].lane = lane
            state["reason"] = f"fell back to {lane.value}"
            break
    return state


def run_adapter(lane, req, messages):
    if lane == Lane.NEMO_BF16:
        return NEMO.generate(messages, req.context_estimate)
    if lane == Lane.NEMO_QUANT:
        return NEMO_Q.generate(messages, req.context_estimate)
    return FR.generate(messages)


g = StateGraph(dict)
g.add_node("extract", extract_keys)
g.add_node("decide", decide)
g.add_node("run", run_lane)
g.add_node("fallback", fallback)
g.set_entry_point("extract")
g.add_edge("extract", "decide")
g.add_edge("decide", "run")
g.add_conditional_edges("run", lambda s: "yes" if fallback_needed(s) else "no",
                        {"yes": "fallback", "no": END})
g.add_edge("fallback", END)

router = g.compile()


def invoke_router(payload: dict) -> dict:
    req = RoutingRequest(**payload["request"])
    return router.invoke({"request": req, "messages": payload["messages"]})

The router is deliberately boring: extract keys, pick a lane from a policy, run it, log the decision. Boring is the point — all the interesting machinery, guardruns, caps, and fallbacks, hangs off that one deterministic decision.

main.py: The FastMCP Gateway

# main.py
import os
import json
from mcp.server.fastmcp import FastMCP
from orchestrator.graph import invoke_router

mcp = FastMCP("model-routing-gateway")


@mcp.tool()
def route_subtask(payload: str) -> str:
    """Route an agent subtask to the cheapest eligible model lane."""
    return json.dumps(invoke_router(json.loads(payload)))


@mcp.tool()
def routing_ledger(query: str) -> str:
    """Ask the routing ledger anything: spend by lane, guardrun deltas."""
    return json.dumps(query_ledger(query))


if __name__ == "__main__":
    mcp.run(transport="stdio")

Serving the gateway as an MCP server means every agent that speaks MCP gets routing through a uniform route_subtask tool — including the whole ecosystem cataloged in our MCP Directory — regardless of which orchestrator it runs.

Budget Caps and the Routing Ledger

# budget.py
import json
import os

LEDGER = os.getenv("ROUTING_LEDGER", "./ledger/routing.jsonl")
CAP = float(os.getenv("DAILY_BUDGET_CAP", "120.0"))


def ledger_append(req, decision, output):
    usage = extract_usage_tokens(output)          # real input+output token counts
    with open(LEDGER, "a") as f:
        f.write(json.dumps({
            "request_id": req.request_id,
            "lane": decision.lane.value,
            "reason": decision.reason,
            "estimated_cost": decision.estimated_cost,
            "actual_tokens": usage,
            "guardrun_id": decision.guardrun_id,
        }) + "
")


def spend_today() -> float:
    day = date_key_today()
    total = 0.0
    with open(LEDGER) as f:
        for line in f:
            rec = json.loads(line)
            if rec["request_id"].startswith(day):
                total += rec.get("actual_cost", rec["estimated_cost"])
    return total


def hard_stop_hit() -> bool:
    return os.getenv("BUDGET_HARD_STOP") == "true" and spend_today() >= CAP

The hard stop is enforced in the gateway process before any adapter call, so a crowd of agents cannot race the cap by issuing calls simultaneously. When hard_stop_hit() returns true, the gateway refuses to route, agents see an explicit budget-exhausted signal, and the money wall is a wall, not a negotiation.

A/B Guardruns

Run the current production policy next to a challenger policy on a shadow sample (5% of traffic via GUARDRUN_SAMPLE_RATE), scoring both on quality (pass rate on a held-out eval set), cost, and latency. Promote the challenger only if it wins on at least two of the three metrics with a minimum five-point quality margin. Policy is code, and the ledger is the evidence — every guardrun_id row lets you re-argue last month's winner with real numbers.

Fallback Chains and Retry Rules

The discipline at the bottom of the table: never retry more than once per production lane, and never return "no decision." An agent that cannot route should fail closed and surface the reason, not improvise a model call on its own.

Security Section

• The gateway is the only route to model endpoints; agents hold no direct endpoint keys.
• FRONTIER_ENDPOINT and FRONTIER_KEY live only in the gateway secrets file, never in agent prompts, plans, or the routing ledger — routing decisions log cost keys, not credentials.
• The ledger is append-only and shipped to the observability store; it contains request IDs and costs and strips message content.
• The budget hard stop runs inside the gateway process before any adapter call, so bursts cannot race the cap.
• Quant lanes run in their own network namespace on your iron, consistent with the egress discipline from our AI Workflows containment guidance, so a compromised worker lane cannot reach the front door of the frontier lane.

Wrap-up

Nemotron 3.5 Lightning is the first open-weight model that makes a 1M-token agentic worker lane an economic no-brainer: 30B total / 3B active MoE, up to 1M context, BF16 or quantized, on a deployment surface from OpenRouter to JumpStart. The workflow in this article gives that hardware its seat at the table — a LangGraph routing gateway served over MCP that spends the cheapest eligible token first, keeps the routing ledger honest, A/B-tests every policy change, and slams the door at the budget cap. Routing is not a recommendation engine; it is an accounting discipline. Wire the keys, run the guardruns, cap the budget, and let Lightning do what Lightning is for: the long, cheap, repetitive work that used to go to a very expensive friend for no good reason.

For more agent-orchestration and cost plays, check the AI Workflows library, hook additional model and gateway tooling from the MCP Directory, and follow latest AI news for NVIDIA releases and OpenRouter pricing changes that will rewrite your thresholds.

Frequently Asked Questions

Is Nemotron 3.5 Lightning actually frontier-grade at 3B active parameters? No — and the gateway design leans on that. Lightning is the capable, cheap, long-context worker: enormous context for broad repository work, not the model you send novel algorithm design or adversarial debugging to. The router exists precisely because the two tiers are different.

How do I pick the long-context threshold? Start at roughly 120k and calibrate against your guardruns. Watch quality deltas in the ledger: tasks below the threshold that keep losing quality in the cheap lane should be reclassified upward; tasks above it that keep passing are being billed by the expensive model for no reason.

Where can I run Nemotron 3.5 Lightning in production? OpenRouter and build.nvidia.com for a managed API, NeMo Switchyard for your own fleet, and AWS SageMaker JumpStart if you are inside AWS. The adapters in this workflow are endpoint-swappable per lane, so none of these choices is binding.

What happens if the budget cap hits mid-task? The gateway fails closed: no subtask routes, no adapter call happens, and the agent receives an explicit budget-exhausted signal. That is the design — a money wall is a hard stop, not an invitation to improvise.

How do guardruns avoid endlessly shuffling policy? Every decision and every comparison is logged with a guardrun_id, so promotion battles are evidence-driven and time-delimited. You promote when a challenger wins two of three metrics with a quality margin; otherwise the production policy stays put and the ledger tells you why.