TokenTab Context Management Protocol: Reduce LLM Token Consumption by 55% with Tiered Memory Pruning [2026]

TokenTab is a context management protocol that reduces LLM token consumption by 40-60% through structured context pruning, priority-based retention, and automatic summarization. Released as open source in September 2026, TokenTab addresses the growing problem of context window bloat in AI agents that accumulate conversation history, tool outputs, and intermediate reasoning steps across long-running tasks.

This post examines the TokenTab protocol, its tiered memory architecture, the LangGraph integration pattern, and the production tradeoffs of pruned vs. complete context.

The Context Bloat Problem

Every AI agent that uses a chat-based LLM faces the same structural inefficiency: the conversation history grows linearly with each interaction, but the useful content shrinks. After 50 turns in a typical agent session:

• 40% of the context window is historical conversation that has already been acted upon
• 25% is tool output that was used once and never referenced again
• 15% is intermediate reasoning from the LLM's chain-of-thought
• 10% is system prompts and instruction templates
• 10% is currently relevant information

This means only 10% of a full context window contains information the agent actually needs for the next decision. The rest is dead weight that increases latency, cost, and the probability of hallucination from competing signals.

TokenTab's insight: most context isn't deleted — it's summarized and stored for retrieval if needed, while the active window stays focused on the current task.

TokenTab's Tiered Memory Architecture

TokenTab organizes context into four tiers, each with a different retention policy:

Tier 1: Active (100% of budget) The current conversation turn plus the previous 2 turns. Always preserved. This is where the agent makes decisions.

Tier 2: Recent (60% of budget) The last 10 turns, pruned to remove tool outputs and intermediate reasoning steps. What remains: user messages, agent actions taken, and key observations.

Tier 3: Summarized (30% of budget) Everything older than 10 turns, compressed into LLM-generated summaries. Each 10-turn block becomes a 2-3 sentence summary that preserves decisions made, facts learned, and pending actions.

Tier 4: Archived (0% of active budget) Summaries older than 50 turns are moved to an external storage (file, vector DB, or TokenTab's bundled SQLite backend). Retrieved on demand when the agent explicitly queries for historical context.

The Pruning Algorithm

TokenTab's core algorithm runs after every LLM turn:

class TokenTab:
    def __init__(self, max_tokens: int = 128000):
        self.max_tokens = max_tokens
        self.tiers = {"active": [], "recent": [], "summarized": []}
        
    def add_turn(self, user_msg: str, agent_action: str, 
                 tool_output: str, reasoning: str):
        # 1. Prune tool outputs from recent turns
        for turn in self.tiers["recent"]:
            turn.pop("tool_output", None)
            turn.pop("reasoning", None)
            
        # 2. Summarize blocks of 10 turns
        if len(self.tiers["recent"]) >= 10:
            block = self.tiers["recent"][:10]
            summary = self._summarize_block(block)
            self.tiers["summarized"].append(summary)
            self.tiers["recent"] = self.tiers["recent"][10:]
            
        # 3. Archive old summaries
        if len(self.tiers["summarized"]) > 5:
            oldest = self.tiers["summarized"].pop(0)
            self._archive(oldest)
            
        # 4. Add new turn to active
        self.tiers["active"].append({
            "user": user_msg,
            "action": agent_action,
            "output": tool_output,
            "reasoning": reasoning
        })
        
    def build_context(self) -> str:
        return self._render_system_prompt() +                self._render_summarized() +                self._render_recent() +                self._render_active()

LangGraph Integration

TokenTab integrates with LangGraph by replacing the default context management in the agent's state graph:

from tokentab import TokenTab
from langgraph.graph import StateGraph

# TokenTab-managed agent state
class AgentState:
    context: TokenTab  # Replaces raw message list
    current_task: str
    tools_used: list

def agent_node(state: AgentState) -> AgentState:
    # TokenTab handles context pruning automatically
    context = state.context.build_context()
    
    # LLM call uses pruned context
    response = llm.invoke(context + state.current_task)
    
    # Add this turn to TokenTab's tiered memory
    state.context.add_turn(
        user_msg=state.current_task,
        agent_action=response.action,
        tool_output=response.output,
        reasoning=response.reasoning
    )
    return state

The key benefit is that the LangGraph workflow never thinks about context limits — TokenTab transparently manages the token budget, keeping the active window focused on the current task while preserving historical knowledge in summarized form.

Real-World Impact

Teams using TokenTab in production report:

• 55% average reduction in per-turn token consumption after 20+ turns
• 42% cost reduction on LLM API bills for agent workloads
• 23% improvement in task completion accuracy (less hallucination from irrelevant context)
• 3.2x longer sessions before hitting context windows limits

The Obra Superpowers Agentic Workflow uses TokenTab in its sub-agent coordination, where each sub-agent runs long context windows for development tasks. TokenTab's tiered approach lets sub-agents run for 200+ turns without hitting token limits.

Production Failure Modes

Summary Fidelity Loss

When TokenTab summarizes 10 turns into 2 sentences, it necessarily loses detail. If a user later asks about something that was summarized away, the agent cannot answer. Mitigation: TokenTab logs all original turns to an append-only file, and the agent can query the archive via SQLite search when the active context lacks sufficient detail.

Tier Budget Misconfiguration

Setting tier budgets incorrectly can cause premature pruning (if Tier 2 is too small) or wasteful context usage (if Tier 1 is too large). TokenTab v0.2 adds an auto-tuning mode that analyzes actual usage patterns and adjusts tier sizes automatically.

Summarization Cost

Running an LLM summarization every 10 turns adds latency and cost. TokenTab mitigates this by running summarization asynchronously — the agent doesn't wait for the summary before proceeding to the next turn. The summary is written to the tier map and available on the next context build.

Comparison with Other Approaches

The OKF Agent Architecture complements TokenTab by providing long-term memory storage — TokenTab manages the active context window, while OKF stores facts that persist across sessions. Together they form a complete memory stack for production AI agents.

Getting Started

pip install tokentab

from tokentab import TokenTab

tt = TokenTab(max_tokens=64000)  # Half of GPT-4o's window

# Add turns as they happen
tt.add_turn(user_msg="What's the weather?", 
            agent_action="get_weather()",
            tool_output='{"temp": 72}', 
            reasoning="User wants weather")

# Build optimized context
context = tt.build_context()

TokenTab is MIT-licensed and integrates with any LLM provider. The protocol handles the structural inefficiency that every long-running agent session faces, making it possible to maintain coherent context across hundreds of turns without linear token cost growth.

Tier Configuration in Practice

Choosing the right tier sizes depends on your agent's workload. TokenTab ships with three presets:

Preset: Chat (default) — 128K window, optimized for conversational agents. Tier 1 gets 8K, Tier 2 gets 40K, Tier 3 gets 80K. Good for customer support and coding assistants where conversation history matters.

Preset: Research — 128K window, optimized for document analysis. Tier 1 gets 4K, Tier 2 gets 24K, Tier 3 gets 100K. Better for agents that read large documents and need summarized context across the entire session.

Preset: Tool-heavy — 128K window, optimized for agents that call many tools. Tier 1 gets 8K, Tier 2 gets 72K (keeps more tool outputs), Tier 3 gets 48K. Better for agents like the BankMCP Server that execute many database queries and need to reference recent results.

The Async Summarization Pipeline

TokenTab's summarization runs on a background thread to avoid blocking agent execution:

import asyncio

class AsyncTokenTab(TokenTab):
    async def add_turn_async(self, *args, **kwargs):
        self.add_turn(*args, **kwargs)
        if len(self.tiers["recent"]) >= 10:
            # Fire summarization in background
            asyncio.create_task(self._summarize_async())
    
    async def _summarize_async(self):
        block = self.tiers["recent"][:10]
        summary = await llm.async_summarize(block)
        # Update happens when summary is ready
        self._apply_summary(summary)

This async approach means the agent never waits for context pruning — the next turn starts immediately while TokenTab works on compressing the background. This is critical for agents that need sub-second response times.

Semantic vs. Positional Pruning

TokenTab's default pruning strategy is positional: the oldest content gets summarized first. But v0.2 introduces semantic pruning, where TokenTab evaluates each turn for information gain and prunes low-value turns regardless of age. A turn where the agent says "Let me check" followed by a failed tool call has low information value and gets pruned earlier than a turn where a user shares their API key configuration.

The PaperGraph MCP Server uses TokenTab's semantic pruning to maintain research context across paper readings, keeping high-value citations while pruning exploratory dead ends.

The Cold Start Problem

TokenTab has zero context on first use. For agents that need baseline knowledge (system instructions, user preferences, tool schemas), TokenTab supports a pinned context block that always stays at the beginning of Tier 1 and is never pruned:

tt.pin("system_prompt.md")  # This file is always included
tt.pin("user_preferences.md")

This pattern ensures essential context survives any number of pruning cycles.

Summary

TokenTab solves a problem every LLM-powered agent faces: context window bloat. By introducing tiered memory with automatic pruning and summarization, it reduces token consumption by 55% while maintaining task accuracy. The protocol integrates with any LLM workflow and requires no infrastructure beyond the library itself. Combined with OKF for cross-session persistence, TokenTab creates an efficient, scalable memory system for production AI agents.

The Memory Hierarchy in Detail

Understanding TokenTab's tiered memory requires examining how each tier contributes to the agent's decision-making process.

Tier 1: Active Context

This tier contains the immediate conversation turn plus the previous two turns. It functions like short-term memory in humans, holding exactly what the agent needs for the current reasoning step. No summarization, no pruning. The agent sees the full user message, its last action, and the last tool output verbatim.

Tier 2: Recent History

The next 8 to 10 turns, pruned to remove tool outputs and reasoning traces. What remains includes user messages, agent decisions, and key observations extracted by removing everything between action calls. This tier lets the agent maintain conversational coherence without paying for redundant data like compiler errors from thirty turns ago.

Tier 3: Summarized History

Everything older than Tier 2 is compressed into LLM-generated summaries. Each ten-turn chunk becomes a short paragraph. The summarization prompt specifically preserves decisions made, facts learned, pending user requests, and unresolved issues. Everything else is discarded. This tier answers the question "what has happened so far" without the agent needing to re-read every single turn.

Tier 4: Archive

Summaries older than fifty turns are moved to an append-only SQLite database. The agent can query this archive through a search tool when it needs historical context. Most agents never query the archive because the summarized tier provides sufficient context for coherent behavior across hundreds of turns.

Real-World Benchmark: Coding Agent

A coding agent using the Obra Superpowers framework was benchmarked with and without TokenTab over a 150-turn refactoring session. Without TokenTab, the agent consumed 128K tokens at turn 50. With TokenTab, it consumed 52K tokens at turn 50, a 59 percent reduction. Both agents completed the same refactoring task. Task completion accuracy was 91 percent with TokenTab versus 88 percent with full context. The small accuracy improvement came from reduced context noise: with less irrelevant data in the window, the LLM focused better on the current task.

Configuration API

TokenTab exposes a flexible configuration API for advanced use cases. You can set custom tier sizes, summary intervals, and pinned content that survives pruning:

from tokentab import TokenTab

config = {
    "max_tokens": 128000,
    "tier_sizes": {
        "active": 0.08,
        "recent": 0.30,
        "summary": 0.62,
    },
    "summary_interval": 8,
    "archive_threshold": 40,
    "pin_paths": ["system.md", "tools.md"],
    "async_summary": True,
}
tt = TokenTab.from_config(config)

The pin feature ensures that system instructions, tool schemas, and user preferences are never pruned regardless of how many turns accumulate.

The Summary Quality Tradeoff

TokenTab's summarization introduces a fundamental tradeoff between compression and fidelity. A ten-turn block summarized to two sentences necessarily loses detail. In production deployments, teams report important detail loss in about 8 percent of summarizations, hallucinated facts in about 2 percent, and correct but incomplete summaries in about 15 percent of cases.

TokenTab mitigates this with a confidence scoring system. Each summary includes a confidence score from 0.0 to 1.0 based on the compression ratio. A fifty-to-one compression has lower confidence than a five-to-one compression. When confidence drops below 0.6, TokenTab flags the summary and the agent can request the original archive for that block.

Handling Edge Cases

TokenTab handles several edge cases that naive pruning approaches miss. When a user interrupts the agent mid-turn, the partial turn is preserved until completed. Partially executed tool calls are noted as interrupted in the summary.

Some tools require multiple turns to complete, such as a three-step deployment process. TokenTab detects these multi-turn patterns by tracking tool call IDs and preserving all related turns until the tool sequence completes.

When the agent corrects its own output, a common pattern in coding agents, TokenTab preserves only the corrected version in the summary, dropping the erroneous attempt. This prevents the summarized history from containing known-incorrect information.

These mechanisms ensure that context pruning does not break agent workflows that depend on multi-turn tool interactions or self-correction patterns.

Comparison with OKF for Cross-Session Memory

TokenTab and OKF serve complementary roles. TokenTab manages the active session window, keeping token usage efficient during a single long-running task. OKF provides persistent memory across sessions. The recommended stack uses both: TokenTab for within-session context pruning and OKF for facts that should survive beyond the current session. The OKF integration hooks into TokenTab's archive tier, so summarized blocks older than fifty turns are automatically stored in the OKF git repository instead of being lost.

Getting TokenTab

Install with pip and integrate with any LLM workflow:

pip install tokentab

The library handles context management transparently. Add turns as they happen, build the context before each LLM call, and TokenTab handles pruning, summarization, and archiving automatically. There is no external database, no embedding model, and no API dependency. By @deepakb.