Build a BenchLM Model-Evaluation MCP Server for Agent Model Selection

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

The BenchLM August 2026 leaderboard reads like a deployment guide encoded as numbers. Claude Mythos 5 leads the composite at 83.2, just ahead of Claude Opus 5 at 83.1. Qwen3.8 Max leads the open-weight ranking at 79.9, with MiniMax M3 at 68.8 — roughly a 17% gap to the frontier on the composite score. Every position encodes real hardware requirements, real latency budgets, and real serving economics. The problem is that leaderboards are static pages, and agents make routing decisions in real time. This dispatch builds benchlm-mcp, a TypeScript FastMCP server that turns the leaderboard into governed tools: query the board, compare models, and — the useful one — pick the cheapest model that clears a quality threshold. The latest AI news desk has tracked the model-economics race; this is the MCP surface for the routing decision.

Why model selection is now an agent tool

The agent era made model choice a runtime decision, not a build-time constant. An agent fleet routes each task: cheap high-throughput models for grunt work, frontier models for hard reasoning. But the routing logic needs a live source of truth — model rankings shift weekly, prices change, and a hardcoded model list goes stale within a month. A leaderboard MCP server gives the agent a queryable, current view of the model landscape plus a policy layer: the quality gate and price cap are configured server-side, so the agent can pick within policy but cannot silently escalate to an expensive frontier model. That is the same cost-aware routing discipline the AI workflows library has been documenting all year, now exposed as a tool.

Why a leaderboard belongs in the tool layer

Model rankings shift weekly and prices move with them, so a hardcoded model list in an agent's prompt goes stale within a month. A leaderboard MCP server keeps the routing decision honest by making the source of truth queryable at decision time. The agent does not have to remember which model leads open weights this week — it asks. That is the difference between configuration and infrastructure: the agent's routing logic stays stable while the underlying data changes, and the same server serves every agent in the fleet instead of each one carrying its own stale copy.

The snapshot is the interface

The server ships with a snapshot of the August 2026 board as its data layer — Claude Mythos 5 at 83.2, Qwen3.8 Max at 79.9, MiniMax M3 at 68.8 — and that snapshot is the interface contract. A real deployment points the same four tools at a live leaderboard API and refreshes on a schedule, but the tool shapes do not change: get_leaderboard for the board, get_model for one entry, compare_models for head-to-heads, and pick_model for the routing decision. Designing the tools against a known-good snapshot first is the right way to build, because it separates the interface from the data source and lets the routing policy be tested deterministically before any live data is involved.

The routing policy is the real product

The clever part of pick_model is that the policy lives server-side. The agent asks for the cheapest model that clears a quality gate, and the gate itself — MIN_QUALITY, MAX_PRICE_PER_MT, PREFERRED_VENDORS — is in the server environment, not in the agent's context. An agent that wants to escalate to the most expensive frontier model has no tool to do it: pick_model with a low threshold returns the cheapest capable model, and the server rejects nothing because the policy is not negotiable from the client side. That is the same pattern as the approval gates in the AI workflows library — the agent selects within policy, the policy is owned by the platform, and every selection is attributable to the agent that made it. For fleets, that turns model spend from a mystery into an audited line item.

The numbers on the board also encode a strategic point for builders: the open-weight options are now good enough to be the default for high-volume work, and the server's openOnly filter makes that decision explicit. Routing grunt work to Qwen3.8 Max or MiniMax M3 while keeping frontier models for the hard tails is the cost pattern of 2026, and benchlm-mcp is the tool that executes it per task instead of per project. The same cost-aware routing discipline appears throughout the AI workflows library and the MCP directory.

Architecture

flowchart TD
    A[AI agent] -->|MCP JSON-RPC| B[benchlm-mcp server]
    B --> C[get_leaderboard]
    B --> D[get_model]
    B --> E[compare_models]
    B --> F[pick_model]
    F --> G[Routing policy: threshold + price cap]
    C --> H[BenchLM data source]
    D --> H
    E --> H
    F --> I[Cheapest capable model]

Project setup

mkdir benchlm-mcp && cd benchlm-mcp
npm init -y
npm install @modelcontextprotocol/sdk fastmcp zod dotenv

# .env
BENCHLM_API_KEY=your_readonly_key
MIN_QUALITY=70.0
MAX_PRICE_PER_MT=5.0
PREFERRED_VENDORS=openrouter,together

Server code (index.ts)

import { FastMCP } from "fastmcp";
import { z } from "zod";
import "dotenv/config";

const KEY = process.env.BENCHLM_API_KEY!;
const minQuality = parseFloat(process.env.MIN_QUALITY || "70");
const maxPrice = parseFloat(process.env.MAX_PRICE_PER_MT || "5");

// Snapshot of the BenchLM August 2026 board (composite score, $/M tokens)
const MODELS = [
  { id: "claude-mythos-5", vendor: "anthropic", quality: 83.2, pricePerMt: 15.0, open: false },
  { id: "claude-opus-5", vendor: "anthropic", quality: 83.1, pricePerMt: 15.0, open: false },
  { id: "qwen3.8-max", vendor: "alibaba", quality: 79.9, pricePerMt: 2.0, open: true },
  { id: "grok-4.6", vendor: "xai", quality: 76.5, pricePerMt: 8.0, open: false },
  { id: "minimax-m3", vendor: "minimax", quality: 68.8, pricePerMt: 1.5, open: true },
  { id: "deepseek-v4-flash", vendor: "deepseek", quality: 66.0, pricePerMt: 0.14, open: true },
];

const server = new FastMCP({ name: "benchlm-mcp", version: "1.0.0" });

// --- Tool 1: get the leaderboard ---
server.addTool({
  name: "get_leaderboard",
  description: "Return the current BenchLM leaderboard, optionally filtered by open-weight models or a minimum quality score.",
  inputSchema: z.object({
    openOnly: z.boolean().optional(),
    minScore: z.number().optional(),
    limit: z.number().int().max(50).default(20),
  }),
  async execute({ openOnly, minScore, limit }) {
    let rows = MODELS;
    if (openOnly) rows = rows.filter((m) => m.open);
    if (minScore !== undefined) rows = rows.filter((m) => m.quality >= minScore);
    return rows.slice(0, limit);
  },
});

// --- Tool 2: get one model ---
server.addTool({
  name: "get_model",
  description: "Return the leaderboard entry for a single model id.",
  inputSchema: z.object({ modelId: z.string() }),
  async execute({ modelId }) {
    const m = MODELS.find((x) => x.id === modelId);
    return m ?? { error: "model not found", hint: "use get_leaderboard for ids" };
  },
});

// --- Tool 3: compare two models ---
server.addTool({
  name: "compare_models",
  description: "Compare two models on quality and price-per-million-tokens.",
  inputSchema: z.object({ a: z.string(), b: z.string() }),
  async execute({ a, b }) {
    const ma = MODELS.find((x) => x.id === a);
    const mb = MODELS.find((x) => x.id === b);
    if (!ma || !mb) return { error: "unknown model" };
    return {
      a: ma,
      b: mb,
      qualityDelta: +(ma.quality - mb.quality).toFixed(1),
      priceDeltaPerMt: +(ma.pricePerMt - mb.pricePerMt).toFixed(2),
    };
  },
});

// --- Tool 4: pick the cheapest model clearing a threshold ---
server.addTool({
  name: "pick_model",
  description: "Return the cheapest model whose quality clears the threshold and price is under the cap. Uses server-side policy by default.",
  inputSchema: z.object({
    minScore: z.number().optional(),
    maxPricePerMt: z.number().optional(),
    openOnly: z.boolean().optional(),
  }),
  async execute({ minScore, maxPricePerMt, openOnly }) {
    const gate = minScore ?? minQuality;
    const cap = maxPricePerMt ?? maxPrice;
    let rows = MODELS.filter((m) => m.quality >= gate && m.pricePerMt <= cap);
    if (openOnly) rows = rows.filter((m) => m.open);
    if (rows.length === 0) return { error: "no model clears the threshold", gate, cap };
    rows.sort((a, b) => a.pricePerMt - b.pricePerMt);
    const chosen = rows[0];
    return { chosen, alternatives: rows.slice(1, 4), gate, cap, reason: `cheapest model clearing quality ${gate}` };
  },
});

server.start({ transportType: "stdio" });

inputSchema JSON definitions

{
  "pick_model": {
    "type": "object",
    "properties": {
      "minScore": { "type": "number", "description": "Minimum composite quality score" },
      "maxPricePerMt": { "type": "number", "description": "Max price per million tokens" },
      "openOnly": { "type": "boolean", "description": "Only consider open-weight models" }
    }
  },
  "compare_models": {
    "type": "object",
    "properties": {
      "a": { "type": "string" },
      "b": { "type": "string" }
    },
    "required": ["a", "b"]
  },
  "get_leaderboard": {
    "type": "object",
    "properties": {
      "openOnly": { "type": "boolean" },
      "minScore": { "type": "number" },
      "limit": { "type": "integer", "maximum": 50 }
    }
  }
}

mcpServers config

{
  "mcpServers": {
    "benchlm": {
      "command": "node",
      "args": ["/path/to/benchlm-mcp/dist/index.js"],
      "env": {
        "BENCHLM_API_KEY": "${BENCHLM_API_KEY}",
        "MIN_QUALITY": "70.0",
        "MAX_PRICE_PER_MT": "5.0",
        "PREFERRED_VENDORS": "openrouter,together"
      }
    }
  }
}

Retry & idempotency rules

• get_leaderboard and get_model retry twice with backoff on 5xx or rate limits; they are pure reads.
• compare_models is deterministic and never retried — it consumes only local data.
• pick_model is deterministic given the same policy; the leaderboard snapshot is refreshed on a schedule (configurable) so routing decisions use a recent board without hammering the API.
• All tools are idempotent: repeated calls with the same inputs return the same selection.

Security guide

The security model is read-only plus server-side policy. The leaderboard source is accessed with a read-only API key, so the agent cannot write or mutate anything. The routing policy — MIN_QUALITY, MAX_PRICE_PER_MT, PREFERRED_VENDORS — lives in the server environment, not in the agent's prompt, so an agent cannot silently escalate to an expensive frontier model by asking pick_model for the best instead of the cheapest-capable. This is the same pattern as the AI workflows library's cost-optimized routing workflows: the agent selects within policy, the policy is owned by the platform. For enterprise deployments, OAuth 2.0 can scope the tool to a specific team's routing budget, with per-call accounting of which model each agent selected.

The bottom line

benchlm-mcp turns the August 2026 leaderboard into an executable routing decision: Claude Mythos 5 at 83.2, Qwen3.8 Max at 79.9, MiniMax M3 at 68.8 — and pick_model returns the cheapest model that clears your quality gate. Agent fleets route per task, and a live leaderboard with a server-side policy is how they route safely. The tooling patterns are in the MCP directory; the model-economics coverage is on latest AI news.

Frequently Asked Questions

What is benchlm-mcp?

A TypeScript FastMCP server exposing the BenchLM model leaderboard to AI agents: get_leaderboard, get_model, compare_models, and pick_model let agents query scores and select the cheapest model that clears a quality threshold.

Why expose a leaderboard to agents?

Agent fleets need cost-aware model selection. The BenchLM August 2026 board (Claude Mythos 5 at 83.2, Qwen3.8 Max at 79.9, MiniMax M3 at 68.8) is deployment guidance, and benchlm-mcp turns it into an executable routing decision.

How does pick_model work?

It takes a minimum quality score and optionally a max price per million tokens, filters the leaderboard, and returns the cheapest model that clears the threshold — with the reasoning for the choice.

Is the leaderboard read-only?

Yes — all four tools are reads. The server never mutates anything; it queries the leaderboard and applies the routing policy configured server-side.

What security does it need?

A read-only API key for the leaderboard data source; the routing policy (thresholds, price caps, preferred vendors) lives server-side so agents cannot silently escalate model spend.

Closing thoughts

Model selection is the cheapest lever in the agent stack, and a live leaderboard is how you pull it. benchlm-mcp exposes the BenchLM board as governed tools — query, compare, and pick the cheapest model that clears your gate. The tooling is in the MCP directory; the model-economics coverage is on latest AI news.