Quantization vs Tool Calls

FP8, BF16, and INT4 do not break agents evenly. On 100K-token agentic replays with real tool calls, weight precision decides whether structured output survives long context, and KV-cache precision decides it even harder. A community INT8 W8A16 quant beat first-party FP8, NVIDIA NVFP4 flipped half its tokens by 88K context and broke tool calls, and INT4 KV cache never recovered where INT8 did.

• Method: forced-history replay with top-1 flip counting every 32nd prompt token in FP64, weights isolated first, then KV cache isolated.
• Weights: FP8 and INT8 W8A16 preserved tool calls past 90K tokens; both 4-bit options botched closes and a CLI check.
• KV cache: INT8 flipped enough top tokens during tool calls to cause a reproducible error; INT4 failed fully; keep agent KV at BF16.

I pin local-agent stacks to this data after a quantized 27B embarrassed us in front of a customer. Here is the full bakeoff.

Chat Benchmarks Hide Agent Failures

MMLU deltas say FP8 recovers 99 percent on Llama 3 8B and 70B against INT8 at 96 to 97 percent. True and useless for agents. Chat turns last a few thousand tokens. Agent sessions live at 40K, 80K, 100K tokens where small per-token divergences compound every step until the run is somewhere else entirely. The failure lands on exact syntax: malformed JSON, unclosed tool calls, wrong CLI verbs.

The Level1Techs experiment from August 2026 fixed the method. Official BF16 Qwen3.6-27B on an RTX PRO 6000 with pinned vLLM, eager execution, no graphs, no speculative decoding, no prefix caching, one variable at a time. Workload is a captured 100K-token agentic workstream nobody could benchmark-max for. At every 32nd prompt token, full-vocabulary logits are captured in BF16 and compared in FP64. Top-1 flips mark positions where the config would have greedily chosen differently than baseline under identical history. When 2 percent of decisions flip, compounding does the rest. For the cost lens on the same inference choices, see price per task vs price per token at 58x spread.

War Story 1: The Demo Where show arp Became show run

We shipped a network-ops agent on a 4-bit AWQ 27B to fit a single 24GB card. The customer demo replayed a 60K-token incident thread. At the CLI step the agent emitted show run instead of show arp, then failed to close its tool call, then looped re-explanations for six turns. The engineer in the room typed the right command in four seconds. The deal stalled two weeks. Back in the lab, the BF16 baseline on the same history completed cleanly.

The bakeoff later named our bug. Both 4-bit options in the study failed the identical pattern: broken tool closes plus the same class of CLI verb error. Both FP8 and INT8 completed correctly. Our quant was not cursed. Four-bit weights plus long agentic context breaks structured output first. We moved agents to INT8 W8A16 on a larger card the same week. Demos pass since.

• Before: 4-bit 27B, 60K context, wrong CLI verb plus unclosed call, stalled deal.
• After: INT8 W8A16 same model, same history, correct call, 90K-token sessions clean.

Do not grade agent quants on chat evals. Replay your longest session. Here is why three prompts prove nothing.

Weight Bakeoff: Five Configs, One History

KV cache held at BF16 to isolate weights. Baseline is the official BF16 checkpoint. Challengers: official FP8 E4M3 in 128x128 blocks with dynamic FP8 activations, community INT8 W8A16 static symmetric with BF16 activations and no calibration set, NVIDIA NVFP4 mixed FP8 plus 4-bit, and community AWQ INT4 W4A16 group 32 calibrated on STEM plus agentic data.

Two surprises matter. First, the community INT8 beat the first-party FP8, attributed to BF16 activations plus unquantized Gated DeltaNet projections. Provenance does not beat architecture. Second, NVFP4 finished last at half flips by 88K. Four bits buy memory and sell the exact capability agents need. Default agents to W8A16 INT8 or first-party FP8. Keep 4-bit weights for chat, summarization, and batch classification. For the cache system that decides how long sessions cost, see KV cache design for 1M-token agents.

KV Cache: The Long-Context Killer

Weights fixed at BF16, only the cache quantized. INT8 cache flipped enough top tokens during tool calls to produce a fully reproducible tool-calling error; the baseline completed fine and INT8 eventually recovered. INT4 cache did not recover at all. Memory math tempts the other way: FP8 halves weight bytes and cache bytes, doubling effective context or concurrency. For agents the savings land precisely on tool syntax after 40K tokens. A doubled context window that cannot close a call is not capacity.

Rule: keep agent KV at BF16. If memory forces quantization, INT8 is the floor and only with replay validation on your history. Treat INT4 KV as disqualified for tool-using agents. Validate with your capture, not vendor prompts: replay a 50K to 100K session, diff tool-call success against short sessions, and gate releases on zero new structured-output failures.

Step 1: Capture and Replay Your Session

Export one real session: prompts, tool calls, tool results, and work products across 50K to 100K tokens. Strip secrets, freeze the history, and replay it under forced history against each candidate config. Log full-vocabulary top-1 flips every N tokens plus tool-call open and close success, JSON validity, and verb-exactness on CLI steps.

File replay.py loads the frozen JSONL, drives vLLM with a pinned commit plus eager execution, disables graphs, speculative decoding, and prefix caching for the comparison, and writes flip counts plus tool outcomes per config. File quants.yaml pins the five candidates with model IDs, dtype flags, and calibration datasets disclosed. One variable per run. Everything else frozen.

vllm serve Qwen3.6-27B --dtype bf16 --enforce-eager --no-spec-decode
python replay.py --session captures/incident-60k.jsonl --config quants.yaml --every 32

Gate on structured output, not trivia scores. A 50 percent flip rate does not mean half the trivia is wrong. It means the model in long context is measurably not the model downloaded, diverging exactly where agents are brittle.

Step 2: Serve Flags That Preserve Calls

On Hopper and Blackwell with native FP8 paths, FP8 KV plus attention costs 1 to 3 accuracy points on validated paths and buys about 13 percent throughput on Llama-3.1-8B class loads. Check context length against the 7K break-even before bothering. On Ampere with no FP8 path, the real choice is BF16 versus INT8, and INT8 trails FP8 by 2 to 3 points of recovery on the same models. Newer vLLM fixed the long-context FP8 collapse from 91 to 13 percent back to 89 percent on 128K retrieval with two-level accumulation, plus break-even fixes that moved one model from 741K tokens to under 8K. Old conclusions from pre-fix builds have expired. Re-run them.

vllm serve MODEL --kv-cache-dtype fp8 --enforce-eager
vllm serve MODEL --kv-cache-dtype auto --quantization awq  # chat only, never agents

Keep activations BF16 where the recipe allows, leave sensitive projections unquantized, and skip CUDA graphs during validation so kernel selection does not confound the dtype signal. For the bill that results from these serving choices, see prompt caching cuts agent bills 80 percent.

War Story 2: The Kernel That Confessed

Our INT8 candidate looked broken until an engineer noticed the nightly build had enabled a fused attention kernel only for that config. Same weights, different kernel, 9 percent extra flips. We pinned eager execution across all configs and the gap vanished. Kernels are part of the quant. Pin the stack or the bakeoff measures the wrong thing.

When NOT to Use This Rigor

Let us be direct. Full replay bakes cost time.

• Chat and summarization fleets under 8K context: MMLU-style deltas plus spot checks suffice. Ship INT4 where it wins.
• Single-GPU hobby setups with no tool use: pick whatever fits VRAM. Structured-output gates do not bind.
• Vendor-managed APIs: precision is their choice. Spend the effort on prompts and harness instead.
• One-off evals with three prompts: admit they are vibes, not validation. Do not cite them in launch notes.

Bottlenecks and Trade-offs

Capture hygiene caps validity. Secrets, PII, and customer data must be stripped before replay stores grow. Calibration datasets bias AWQ-style quants toward their tuning mix; disclose the mix and re-test on your domain. Driver and vLLM version drift reopens fixed bugs; pin commits and re-run the gate on every upgrade.

Memory pressure still matters. FP8 halves bytes and can drop a 70B from two cards to one, which dwarfs dtype throughput deltas. Where agents must fit, prefer W8A16 INT8 first, FP8 second, and buy the bigger card before reaching for 4-bit weights on tool paths.

Ship Checklist

1. One frozen 50K-plus session with tool calls, replayed per config with flips plus structured-output gates.
2. KV at BF16 for agents, INT8 floor only with replay proof, INT4 disqualified.
3. Pinned vLLM, eager validation, disclosed calibration, re-run on every stack change.

Start with your longest failing session. The quant that survives it is your default.

By Deepak Bagada, Founder and Editor-in-Chief at Daily AI World. I build agentic systems at SaaSNext and write from production logs, not demos. Follow @deeepakbagada and read more at https://deepakbagada.in.