By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

agentgateway MCP Enterprise Security Proxy Pipeline

Every enterprise now ships AI agents that can read a database, post to Slack, trigger CI/CD deploys, and query internal wikis. The Model Context Protocol (MCP) makes that possible in minutes — which is exactly why it has become the new attack surface. The tool your agent calls is not your agent. The difference between a weekend demo and a production system is a security proxy sitting between the agent and every tool it touches.

agentgateway is a next-generation agentic proxy purpose-built for enterprise MCP security. It sits in front of your MCP servers and gives platform teams a single choke point for routing, authentication, rate limiting, cost control, and audit. This guide walks through a complete 2026 deployment: a LangGraph agent that talks to three MCP servers through agentgateway, with per-tenant policies, deterministic retry logic, and a tamper-evident audit trail. If you are weighing Otari or Portkey instead, the comparison table later in this article will help you decide. For more production patterns, browse the Daily AI World workflows library and the MCP directory.

What agentgateway actually does

agentgateway is not another LLM router. Where a router picks a model, agentgateway governs the tools an agent is allowed to call. Every MCP request crosses the proxy, where four engines act in sequence:

1. Identity and auth — mTLS, OIDC tokens, or service-to-service keys are validated before a request reaches any tool.
2. Policy engine — RBAC and per-tool allow/block lists are resolved against the caller's tenant and agent role.
3. Rate limiting and budgets — per-tenant RPM, per-tool cost ceilings, and burst windows are enforced at the edge.
4. Audit and replay — every decision, payload hash, and latency is written to an append-only sink.

The proxy implements the MCP spec on the server side, so your agents keep using the standard MCP client SDK. You insert agentgateway by changing the endpoint URL — not by rewriting your agent.

Architecture diagram

flowchart LR
    U[User / Chat] --> A[LangGraph Agent]
    A --> G[agentgateway Proxy]
    G --> P[Policy Engine]
    P --> R[Rate Limiter]
    P --> B[Budget / Cost Guard]
    R -->|forward| S1[MCP: HR System]
    R -->|forward| S2[MCP: CI-CD]
    R -->|forward| S3[MCP: Analytics]
    G --> Q[(Audit Sink / S3)]
    B --> G

Deployment pipeline

The reference repo below deploys a LangGraph agent that answers employee questions by calling HR, CI-CD, and analytics MCP servers, all through agentgateway. It is split into five files so each concern is easy to review in isolation.

.env

# agentgateway pipeline environment
AGENTGATEWAY_ENDPOINT=https://gateway.corp.example.com/v1
AGENTGATEWAY_KEY=agw_live_xxxxxxxxxxxx
LLM_PROVIDER=anthropic
ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxxxx
MCP_HR_URL=https://mcp.hr.corp.example.com
MCP_CICD_URL=https://mcp.cicd.corp.example.com
MCP_ANALYTICS_URL=https://mcp.analytics.corp.example.com
RATE_LIMIT_RPM=600
RATE_LIMIT_PER_USER_RPM=60
COST_BUDGET_CENTS_PER_DAY=500
AUDIT_SINK=s3://corp-audit/agentgateway/

schemas.py

from uuid import uuid4
from pydantic import BaseModel, Field
from typing import Literal, Optional

class ToolCall(BaseModel):
    server: str
    tool: str
    args: dict
    user_id: str
    tenant_id: str

class ProxyDecision(BaseModel):
    decision: Literal["allow", "block", "rate_limit", "review"]
    reason: str
    trace_id: str = Field(default_factory=lambda: uuid4().hex)

class RetryPolicy(BaseModel):
    max_attempts: int = 4
    base_delay_s: float = 0.5
    backoff_factor: float = 2.0
    jitter: float = 0.15
    retry_on: tuple = (429, 500, 502, 503, 504)

tools.py

import time
import httpx
from .schemas import ToolCall, ProxyDecision, RetryPolicy

def call_gateway(env, tc: ToolCall) -> ProxyDecision:
    resp = httpx.post(
        f"{env.AGENTGATEWAY_ENDPOINT}/mcp/execute",
        headers={"Authorization": f"Bearer {env.AGENTGATEWAY_KEY}"},
        json=tc.model_dump(),
    )
    return ProxyDecision.model_validate(resp.json())

def execute_with_retry(env, tc: ToolCall, rp: RetryPolicy) -> ProxyDecision:
    for attempt in range(rp.max_attempts):
        decision = call_gateway(env, tc)
        if decision.decision != "rate_limit":
            return decision
        delay = rp.base_delay_s * (rp.backoff_factor ** attempt)
        time.sleep(delay * (1 + rp.jitter))
    return ProxyDecision(decision="block", reason="max_retries_exhausted")

graph.py

from typing import TypedDict
from langgraph.graph import StateGraph, START, END
from .schemas import ProxyDecision, RetryPolicy

class AgentState(TypedDict):
    tool_call: dict
    attempts: int
    decision: ProxyDecision

def route(state: AgentState) -> str:
    d = state["decision"]
    if d.decision == "allow":
        return "execute_tool"
    if d.decision == "rate_limit":
        return "backoff"
    return "reject"

def backoff(state: AgentState, retry: RetryPolicy) -> AgentState:
    n = state["attempts"]
    delay = retry.base_delay_s * (retry.backoff_factor ** n)
    time.sleep(delay)
    return {**state, "attempts": n + 1}

builder = StateGraph(AgentState)
builder.add_node("call_gateway", call_gateway_node)
builder.add_node("execute_tool", execute_tool_node)
builder.add_node("backoff", backoff)
builder.add_edge(START, "call_gateway")
builder.add_conditional_edges(
    "call_gateway", route,
    {"execute_tool": "execute_tool", "backoff": "backoff", "reject": END},
)
graph = builder.compile()

main.py

import uvicorn
from fastapi import FastAPI
from .graph import graph

app = FastAPI(title="agentgateway reference agent")

@app.post("/chat")
async def chat(payload: dict):
    return await graph.ainvoke(payload)

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Routing and policy configuration

Routes are declared per tenant, not per developer. A platform team defines which MCP servers a tenant may reach and which tools within each server are callable. A finance tenant gets read-only access to the analytics MCP, while a DevRel tenant can call the CI/CD server but only the trigger_staging_deploy tool. Per-tool allow/block lists are the cheapest security you can buy: most breaches in agentic systems start with an over-permissioned tool call, not an exotic exploit.

Policy configuration should be declarative and versioned in Git. When a policy changes, the gateway reloads it without dropping in-flight requests, and the change itself becomes an audit event. Keep the policy model small — tenants, roles, tools, budgets — and resist the urge to build a rules engine. YAML wins for the first hundred rules, and if you outgrow it, you will outgrow your agent architecture too.

Authentication, rate limiting, and audit

Three guarantees matter more than anything else in an agentic gateway:

1. Authentication is per-call, not per-session. Agents persist and replay conversations; a token minted for one tool call must not be reusable for another. agentgateway issues short-lived, single-purpose tokens scoped to one server and tool.
2. Rate limits are enforced against the tenant, the user, and the tool simultaneously. A single viral chatbot hitting a shared MCP server should throttle itself, not the whole company. Budgets work in cents-per-day so a runaway loop burns your policy cap, not your cloud bill.
3. Audit logs are immutable and replayable. Every allow and every block is recorded with the trace ID, the caller, the tool, the payload hash, and latency. When a security team asks "what did that agent do at 2:14 AM?", the answer should be a one-line query against the audit sink, not a scramble through model logs.

agentgateway vs Otari vs Portkey

If your problem is "pick the best model and fail over," Portkey or Otari is the better fit. If your problem is "which tools may this agent call, at what rate, within what budget, with what audit trail," agentgateway is the right shape. The strongest production setups run both: an LLM router for model selection and an MCP proxy for tool governance.

Retry Rules

Retries are where agentic pipelines either become resilient or amplify outages. These rules are the ones I enforce in every production deployment:

• Retry only on idempotent tool calls. Re-running send_slack_message is a duplicate; re-running get_employee_record is safe. Mark non-idempotent tools as no_retry in the policy.
• Use exponential backoff with jitter: base delay of 500 ms, factor of 2, capped at 30 seconds, plus 15% jitter to avoid thundering herds.
• Retry on 429, 500, 502, 503, 504 — never on 400 or 401. A 401 means your auth is broken, and retrying it just extends the outage.
• Give up after 4 attempts and surface a structured error with the trace ID instead of an empty string. A failed tool call is a debugging affordance.
• Apply circuit-breaking at the server level: if an MCP server returns 5xx for 10 consecutive requests, open the circuit and fail fast for 60 seconds.
• Retry with the same tenant context and the same policy revision so the audit trail stays coherent across attempts.

Honest limitations

agentgateway is not a silver bullet, and you should hear the limitations before you budget for the migration. First, it only protects what goes through it. Any agent that bypasses the proxy — a hardcoded API key, a side-channel database connection — is invisible to the audit trail. Second, prompt injection that tricks the model into calling a permitted tool is not stopped by a policy engine; you still need output validation and tool-call confirmation for high-risk actions. Third, the control plane is one more system to operate: certificate rotation, policy reviews, and the audit sink all need owners. Finally, MCP is moving fast; pin the spec version your agents and the gateway agree on, or you will chase breaking changes every quarter.

Where to go next

Start with a single high-risk MCP server and one tenant, then expand. Wire the audit sink into your SIEM before you announce success, and simulate a runaway agent loop in staging to prove the budgets actually cut it off. The pattern — route, authenticate, rate-limit, audit — is the same regardless of which proxy you pick, and it will age well. Keep an eye on the latest AI news for spec changes, and revisit the workflows library for the next pipeline you will build.

FAQ

Q: Does agentgateway replace our existing API gateway?

A: No. Your API gateway protects your HTTP surface; agentgateway protects your MCP tool surface. They compose: requests arrive at the API gateway, get authenticated, and are forwarded to the agent, which then calls tools through agentgateway.

Q: Can I keep using my existing MCP client SDK?

A: Yes. agentgateway implements the MCP protocol on the server side, so standard MCP clients work by pointing at the gateway endpoint. You change a URL, not your agent code.

Q: What is the minimum deployment footprint?

A: A single container behind a load balancer can run the control plane for a few thousand agents. The proxy is stateless; the audit sink and policy store are the stateful pieces you must operate.

Q: How do I handle a tool that is not idempotent?

A: Mark it no_retry in the policy and require a confirmation step for it. The gateway will reject retries for that tool and log the reason, so a double charge or duplicate message never happens.

Q: Does it work with non-LangGraph agents?

A: Yes. The gateway is language- and framework-agnostic. Any agent that speaks the MCP protocol over HTTP or stdio can use it, whether it is built on LangGraph, Claude Code, or a custom runtime.