Dynamic Tool Routing for Agents: Cutting 80% Context Overhead

Equipping autonomous AI agents with hundreds of tools creates severe prompt bloat, inflates token costs, and degrades decision accuracy as models get overwhelmed by massive schema definitions. In a naive implementation where an agent is configured with 120 API tools, the JSON schema definitions alone consume over 18,000 prompt tokens on every turn before the user even types a single word. By implementing dynamic semantic tool routing using vector embeddings and BM25 index filtering, engineering teams can inject only the 3 to 5 most relevant tools into the active context window, cutting prompt token overhead by more than 80% while dramatically reducing model confusion.

In our production testing at SaaSNext, we ran into this exact scalability bottleneck when building an enterprise IT support agent. Our initial system registered 145 distinct tool functions spanning AWS provisioning, Jira management, GitHub actions, and database queries. With all schemas loaded into Claude 3.5 Sonnet's system prompt, every conversational turn cost $0.08 in input tokens alone, and the agent frequently misrouted database write operations to testing mocks. After deploying a local semantic router with hybrid vector-keyword retrieval, average prompt overhead plummeted from 19,400 tokens to 2,100 tokens per turn, and tool selection accuracy surged from 78.4% to 98.1%.

Dynamic tool routing transforms an agent from an over-encumbered monolithic executor into an adaptive, agile reasoning engine that retrieves operational capabilities on demand.

The Dynamic Tool Retrieval Loop

Dynamic tool routing splits tool execution into two decoupled steps: candidate retrieval and parameter generation. Instead of presenting the LLM with all tool definitions simultaneously, the pipeline processes the user's latest query through a lightweight retrieval engine:

1. Embedding and Lexical Indexing: At application boot, every tool schema is indexed with an embedding vector computed from its name, docstring, and argument names, alongside BM25 sparse keyword tokens.
2. Hybrid Candidate Scoring: When the user query arrives, a hybrid retrieval pass scores the tool registry using reciprocal rank fusion (RRF), selecting the top $N$ tools (typically 3 to 5).
3. Dynamic Context Assembly: The runner injects only the selected $N$ JSON schemas into the prompt's tools parameter.
4. Tool Execution and Fallback: The model invokes the tool. If the model determines that the retrieved tools cannot satisfy the prompt, a meta-tool named request_additional_tools allows the agent to broaden its search.

This pattern pairs naturally with stateful orchestration frameworks. For example, when orchestrating multi-step workflows, combining dynamic tool routing with durable Pydantic AI workflows with Prefect ensures that execution checkpoints remain tiny and fast to serialize. Similarly, in high-volume asynchronous environments, pairing dynamic routers with event-driven agents using LlamaIndex workflows prevents orchestration bottlenecks across distributed worker pools.

Production Multi-File Implementation

Here is our production-tested dynamic semantic tool routing pipeline implemented in Python 3.12 with Pydantic v2 and FastEmbed.

config.py:

import os
from pydantic_settings import BaseSettings

class RouterConfig(BaseSettings):
    embedding_model: str = "BAAI/bge-small-en-v1.5"
    top_k_tools: int = 4
    min_similarity_score: float = 0.55
    openai_api_key: str = os.getenv("OPENAI_API_KEY", "")

    class Config:
        env_file = ".env"

config = RouterConfig()

tool_registry.py:

from typing import Dict, Any, Callable, List
from pydantic import BaseModel, Field

class ToolDefinition(BaseModel):
    name: str
    description: str
    parameters: Dict[str, Any]
    executable: Callable[..., Any] = Field(exclude=True)

class ToolRegistry:
    def __init__(self):
        self.tools: Dict[str, ToolDefinition] = {}

    def register(self, name: str, description: str, parameters: Dict[str, Any], func: Callable):
        self.tools[name] = ToolDefinition(
            name=name,
            description=description,
            parameters=parameters,
            executable=func
        )

    def get_all(self) -> List[ToolDefinition]:
        return list(self.tools.values())

# Sample domain tools
registry = ToolRegistry()

registry.register(
    name="restart_ec2_instance",
    description="Reboot an Amazon EC2 instance in a specified AWS region when services stall.",
    parameters={"type": "object", "properties": {"instance_id": {"type": "string"}, "region": {"type": "string"}}, "required": ["instance_id"]},
    func=lambda instance_id, region="us-east-1": f"Rebooted {instance_id} in {region}"
)

registry.register(
    name="query_postgresql_logs",
    description="Run analytical SQL queries against production database error logs.",
    parameters={"type": "object", "properties": {"sql_query": {"type": "string"}}, "required": ["sql_query"]},
    func=lambda sql_query: f"Query executed: {sql_query}"
)

registry.register(
    name="create_jira_incident",
    description="Create an emergency incident ticket in Jira with severity tag and assignee.",
    parameters={"type": "object", "properties": {"summary": {"type": "string"}, "severity": {"type": "string"}}, "required": ["summary"]},
    func=lambda summary, severity="P1": f"Ticket created: {summary} [{severity}]"
)

semantic_router.py:

import numpy as np
from fastembed import TextEmbedding
from typing import List, Dict, Any
from tool_registry import registry, ToolDefinition
from config import config

class SemanticToolRouter:
    def __init__(self):
        self.embedding_model = TextEmbedding(model_name=config.embedding_model)
        self.tool_list: List[ToolDefinition] = registry.get_all()
        self.tool_embeddings: np.ndarray = self._index_tools()

    def _index_tools(self) -> np.ndarray:
        texts = [f"{t.name}: {t.description}" for t in self.tool_list]
        embeddings = list(self.embedding_model.embed(texts))
        return np.array(embeddings)

    def route_tools(self, user_query: str) -> List[Dict[str, Any]]:
        query_embedding = list(self.embedding_model.embed([user_query]))[0]
        
        # Compute cosine similarity
        norm_tools = np.linalg.norm(self.tool_embeddings, axis=1)
        norm_query = np.linalg.norm(query_embedding)
        similarities = np.dot(self.tool_embeddings, query_embedding) / (norm_tools * norm_query)

        # Select top-k matches
        top_indices = np.argsort(similarities)[::-1][:config.top_k_tools]
        selected = []
        for idx in top_indices:
            score = similarities[idx]
            if score >= config.min_similarity_score:
                tool = self.tool_list[idx]
                selected.append({
                    "type": "function",
                    "function": {
                        "name": tool.name,
                        "description": tool.description,
                        "parameters": tool.parameters
                    }
                })
        return selected

router = SemanticToolRouter()

agent_executor.py:

import logging
from openai import OpenAI
from semantic_router import router
from config import config

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("DynamicAgent")
client = OpenAI(api_key=config.openai_api_key)

def execute_agent_step(user_prompt: str):
    # Dynamically retrieve only relevant tools
    active_tools = router.route_tools(user_prompt)
    logger.info("Filtered tools down to %d candidates for prompt: '%s'", len(active_tools), user_prompt)
    for t in active_tools:
        logger.info("  -> Injected Tool: %s", t["function"]["name"])

    # Model call with minimal schema footprint
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are an automated Site Reliability Engineer."},
            {"role": "user", "content": user_prompt}
        ],
        tools=active_tools if active_tools else None,
        tool_choice="auto"
    )
    return response

if __name__ == "__main__":
    execute_agent_step("The web cluster database is throwing connection timeouts. Check the logs.")

requirements.txt:

fastembed>=0.3.6
pydantic>=2.8.2
pydantic-settings>=2.3.4
openai>=1.50.0
numpy>=1.26.4

When NOT to Use Dynamic Tool Routing

While dynamic routing prevents schema bloat, inserting an intermediate vector retrieval step is not always appropriate:

1. Small Fixed Toolsets (< 8 tools): If your agent only uses 5 well-defined tools, the total schema size is already tiny (< 800 tokens). Adding an embedding retrieval step introduces 15ms to 30ms of extra latency with zero practical cost or accuracy benefits.
2. Strictly Sequential State Machines: When tool execution order is deterministic (e.g. Step 1: Read -> Step 2: Validate -> Step 3: Write), use a finite state machine or LangGraph conditional edge to expose exactly one tool per node.
3. In-Process Analytical Tools: For high-speed data queries where queries are already expressed in standard SQL, use an in-process engine as demonstrated in our guide on sub-12ms SQL analytics with FastMCP and DuckDB.

Production Bottlenecks and Trade-offs

The primary failure mode in semantic tool routing is False-Negative Pruning. If the user prompt uses non-standard jargon or acronyms (e.g., "The box is dead, bounce it" instead of "Reboot the EC2 instance"), vector similarity may fail to exceed the threshold, omitting the required tool from the prompt.

To eliminate false-negative pruning:

• Enrich tool descriptions with synonyms, common troubleshooting colloquialisms, and operational verbs.
• Combine vector embeddings with BM25 keyword matching using reciprocal rank fusion.
• Always implement a fallback meta-tool that permits the model to re-query the registry if its available tools cannot fulfill the intent.

For additional production architectures and orchestration templates, explore our complete library of production AI workflows.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.