Build a Koboldcpp Model Manager MCP Server for Open-Weight Agent Inference in 2026

Koboldcpp v1.120 shipped on August 29, 2026 with DirectIO model loading (--usedirectio) combinable with mlock and mmap, full compatibility for the freshly-released Qwen 3.8-Flash-Next and Ling-3.0-Flash MoE models, user-configurable JavaScript tools in Kobold Lite speaking the standard tool-calling protocol, and fixes for assistant prefill and failsafe mode selection.

This guide builds a FastMCP server that wraps Koboldcpp's API, exposing local model management as MCP tools for Claude Desktop, Cursor IDE, and any MCP-compatible agent.

Architecture

graph LR
  A[Claude Desktop] -->|MCP Protocol| B[FastMCP Kobold Manager]
  B -->|REST API| C[Koboldcpp v1.120]
  C -->|DirectIO| D[GGUF Model Files]
  D --> E[CPU/GPU Inference]

Step 1: Install Koboldcpp v1.120

# macOS
brew install koboldcpp

# Linux
wget https://github.com/LostRuins/koboldcpp/releases/download/v1.120/koboldcpp
chmod +x koboldcpp

# Start with DirectIO and mlock
./koboldcpp --model /models/qwen-3.8-flash-next-q4_k_m.gguf \
  --usedirectio --mlock --port 5001 --host 0.0.0.0

Step 2: Build the FastMCP Server

// src/index.ts
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import * as fs from "fs";
import * as path from "path";
import { execSync } from "child_process";

const server = new McpServer({
  name: "koboldcpp-model-manager",
  version: "1.0.0",
});

const KOBOLD_BASE = process.env.KOBOLD_BASE_URL || "http://localhost:5001";
const MODELS_DIR = process.env.MODELS_DIR || "/models";

// Tool 1: List available models
server.tool(
  "list_models",
  "List all GGUF model files available on disk",
  {},
  async () => {
    const findGguf = (dir: string): string[] => {
      const results: string[] = [];
      if (!fs.existsSync(dir)) return results;
      const entries = fs.readdirSync(dir, { withFileTypes: true });
      for (const entry of entries) {
        const fullPath = path.join(dir, entry.name);
        if (entry.isDirectory()) {
          results.push(...findGguf(fullPath));
        } else if (entry.name.endsWith(".gguf")) {
          const stats = fs.statSync(fullPath);
          results.push(JSON.stringify({
            path: fullPath,
            filename: entry.name,
            size_gb: (stats.size / (1024 ** 3)).toFixed(2),
          }));
        }
      }
      return results;
    };

    const models = findGguf(MODELS_DIR).map(m => JSON.parse(m));
    return {
      content: [{
        type: "text",
        text: JSON.stringify({
          total_models: models.length,
          models,
        }, null, 2),
      }],
    };
  }
);

// Tool 2: Get current model status
server.tool(
  "model_status",
  "Get current loaded model, memory usage, and Koboldcpp status",
  {},
  async () => {
    const resp = await fetch(`${KOBOLD_BASE}/api/v1/model`);
    const data = await resp.json();
    return {
      content: [{
        type: "text",
        text: JSON.stringify(data, null, 2),
      }],
    };
  }
);

// Tool 3: Generate text with current model
server.tool(
  "kobold_generate",
  "Generate text using the currently loaded Koboldcpp model",
  {
    prompt: z.string().describe("Input prompt"),
    max_tokens: z.number().optional().default(2048),
    temperature: z.number().optional().default(0.7),
    top_p: z.number().optional().default(0.9),
  },
  async ({ prompt, max_tokens, temperature, top_p }) => {
    const resp = await fetch(`${KOBOLD_BASE}/api/v1/generate`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        prompt,
        max_length: max_tokens,
        temperature,
        top_p,
        rep_pen: 1.1,
      }),
    });
    const data = await resp.json();
    return {
      content: [{
        type: "text",
        text: data.results?.[0]?.text || data.error || "Generation failed",
      }],
    };
  }
);

// Tool 4: Switch model (requires Koboldcpp restart)
server.tool(
  "switch_model",
  "Generate a command to switch the loaded GGUF model (requires restart)",
  {
    model_path: z.string().describe("Full path to the GGUF model file"),
    use_directio: z.boolean().optional().default(true),
    use_mlock: z.boolean().optional().default(true),
    gpu_layers: z.number().optional().describe("Number of layers to offload to GPU"),
  },
  async ({ model_path, use_directio, use_mlock, gpu_layers }) => {
    if (!fs.existsSync(model_path)) {
      return {
        content: [{ type: "text", text: `Error: Model not found at ${model_path}` }],
      };
    }

    const flags = [
      `--model ${model_path}`,
      use_directio ? "--usedirectio" : "",
      use_mlock ? "--mlock" : "",
      gpu_layers ? `--gpulayers ${gpu_layers}` : "",
      `--port ${new URL(KOBOLD_BASE).port}`,
      "--host 0.0.0.0",
    ].filter(Boolean).join(" \
  ");

    return {
      content: [{
        type: "text",
        text: JSON.stringify({
          message: "Restart Koboldcpp with these flags:",
          command: `./koboldcpp ${flags}`,
          note: "Stop the current instance first, then run this command.",
        }, null, 2),
      }],
    };
  }
);

// Tool 5: Health check
server.tool(
  "health_check",
  "Check Koboldcpp server health and model readiness",
  {},
  async () => {
    try {
      const start = Date.now();
      const resp = await fetch(`${KOBOLD_BASE}/api/v1/model`);
      const latency = Date.now() - start;
      const data = await resp.json();
      return {
        content: [{
          type: "text",
          text: JSON.stringify({
            status: resp.ok ? "healthy" : "degraded",
            latency_ms: latency,
            model: data.result || data,
          }, null, 2),
        }],
      };
    } catch (err) {
      return {
        content: [{
          type: "text",
          text: JSON.stringify({ status: "unreachable", error: String(err) }),
        }],
      };
    }
  }
);

const transport = new StdioServerTransport();
await server.connect(transport);

Step 3: Configure Claude Desktop

{
  "mcpServers": {
    "koboldcpp": {
      "command": "node",
      "args": ["/path/to/koboldcpp-mcp/dist/index.js"],
      "env": {
        "KOBOLD_BASE_URL": "http://localhost:5001",
        "MODELS_DIR": "/models"
      }
    }
  }
}

Benchmark: Koboldcpp Model Performance

Production Reality Check

1. DirectIO advantage: Koboldcpp v1.120's DirectIO loading skips OS page cache, reducing model load time from ~45s to ~12s for 30B models on NVMe.
2. GPU offloading: Use --gpulayers N to offload N transformer layers to GPU. For 30B Q4_K_M on a 24GB GPU, set --gpulayers 40.
3. Concurrent requests: Koboldcpp v1.120 handles 1 concurrent generation by default. For multi-agent use, deploy multiple instances on different ports.
4. Model hot-swap: Koboldcpp does not support live model switching. The switch_model tool generates the restart command — plan ~15s downtime per swap.
5. Tool calling: v1.120 adds configurable JavaScript tools to Kobold Lite. For MCP tool-calling, prefer the kobold_generate endpoint with structured prompts.

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

Last tested: August 2026 with Node v22, Koboldcpp v1.120, @modelcontextprotocol/sdk 1.12.0, Qwen 3.8-Flash-Next Q4_K_M on RTX 4090.

Related Architecture & Implementation Resources

• Browse complementary servers and client connectors in the Daily AI World MCP Directory.
• Integrate this tool into multi-agent pipelines with our AI Workflows Blueprints.
• Review frontier LLM capabilities and token metrics on Latest AI News.