AEO Direct Answer Box

AI agents generate complex OpenTelemetry trace waterfalls with hundreds of spans per execution. Debugging slow agent steps or identifying failing tool calls typically requires switching to Datadog APM, searching for the specific trace, and manually correlating spans against agent decisions. This FastMCP server bridges Datadog APM directly into the agent's context, enabling Claude Desktop, Cursor, and OpenCode to query live trace data, analyze agent call latency percentiles, and detect slow tool executions without leaving the conversation. Key metrics include sub-second query response for traces within a seven-day retention window and automatic span-to-agent-step correlation using OpenTelemetry span attributes.

• Data source: Datadog APM Metrics API and Traces API
• Query latency: Under one second for seven-day trace windows
• Supported clients: Claude Desktop, Cursor, Windsurf, VS Code, OpenCode
• Framework: FastMCP 2.x with TypeScript and Zod validation
• Span correlation: OpenTelemetry span attributes mapped to agent step IDs

Build a Datadog AI Agent Observability MCP Server for OpenTelemetry Traces in 2026

Debugging AI agent behavior in production is fundamentally harder than debugging traditional software. A single LangGraph workflow execution can generate hundreds of OpenTelemetry spans across tool calls, LLM invocations, state transitions, and error recovery paths. When an agent produces a wrong answer or crashes at step fourteen, developers need to reconstruct the execution timeline, identify the slowest spans, and understand why the agent made specific routing decisions. This FastMCP server makes Datadog APM queryable directly from within the agent conversation, eliminating the context switching penalty of jumping between chat interfaces and APM dashboards.

Architecture Overview

The server proxies queries from the MCP client through to Datadog's Metrics API and Traces API. It accepts natural language descriptions of the trace to find, converts them into Datadog query syntax, and returns structured summaries including latency percentiles, error spans, and step-by-step execution waterfalls.

flowchart LR
    A[MCP Client] -->|Query: find slowest spans| B[FastMCP Server]
    B --> C[Datadog Metrics API]
    B --> D[Datadog Traces API]
    C --> E[Latency Percentiles]
    D --> F[Span Waterfall]
    E --> G[Formatted Response]
    F --> G
    G --> A

Step 1: Datadog MCP Server Implementation

import { FastMCP } from "fastmcp";
import { z } from "zod";
import { DatadogClient } from "./datadog-client.js";

const app = new FastMCP({
  name: "datadog-observability",
  version: "1.0.0",
});

const dd = new DatadogClient({
  apiKey: process.env.DATADOG_API_KEY!,
  appKey: process.env.DATADOG_APP_KEY!,
  site: process.env.DATADOG_SITE || "datadoghq.com",
});

// Tool 1: Query the latest traces for an agent workflow
app.tool(
  "query_traces",
  "Query recent agent traces by workflow name or agent ID",
  {
    workflow: z.string().describe("Agent workflow name or trace tag"),
    time_range: z.string().default("15m").describe("Time range: 15m, 1h, 6h, 1d, 7d"),
    max_traces: z.number().default(10),
  },
  async (args) => {
    const traces = await dd.queryTraces(args.workflow, args.time_range, args.max_traces);
    return { content: [{ type: "text", text: JSON.stringify(traces, null, 2) }] };
  }
);

// Tool 2: Analyze latency percentiles for a specific tool or step
app.tool(
  "analyze_latency",
  "Get p50, p95, p99 latency for agent tools or workflow steps",
  {
    tool_name: z.string().describe("MCP tool name or workflow step label"),
    time_range: z.string().default("6h"),
  },
  async (args) => {
    const metrics = await dd.getLatencyPercentiles(args.tool_name, args.time_range);
    return { content: [{ type: "text", text: JSON.stringify(metrics, null, 2) }] };
  }
);

// Tool 3: Find error spans in agent executions
app.tool(
  "find_errors",
  "Find error spans and exceptions across recent agent traces",
  {
    workflow: z.string().describe("Workflow name to scope the search"),
    time_range: z.string().default("1h"),
    error_type: z.string().optional().describe("Filter by error type: timeout, rate_limit, tool_error"),
  },
  async (args) => {
    const errors = await dd.findErrorSpans(args.workflow, args.time_range, args.error_type);
    return { content: [{ type: "text", text: JSON.stringify(errors, null, 2) }] };
  }
);

app.start({ transport: "stdio" });

Step 2: Datadog API Client

import { z } from "zod";

const TRACES_URL = "https://api.datadoghq.com/api/v2/apm/trace";
const METRICS_URL = "https://api.datadoghq.com/api/v2/query/timeseries";

interface TraceSpan {
  span_id: string;
  trace_id: string;
  operation_name: string;
  service: string;
  resource: string;
  duration_ns: number;
  error: number;
  meta: Record<string, string>;
  parent_id: string | null;
  children: TraceSpan[];
}

export class DatadogClient {
  private headers: Record<string, string>;

  constructor(config: { apiKey: string; appKey: string; site: string }) {
    this.headers = {
      "DD-API-KEY": config.apiKey,
      "DD-APPLICATION-KEY": config.appKey,
      "Content-Type": "application/json",
    };
  }

  async queryTraces(workflow: string, timeRange: string, max: number): Promise<any[]> {
    const response = await fetch(TRACES_URL, {
      method: "POST",
      headers: this.headers,
      body: JSON.stringify({
        filter: { query: `@workflow.name:"${workflow}"` },
        sort: "-@timestamp",
        limit: max,
        timeframe: this.parseTimeRange(timeRange),
      }),
    });
    const data = await response.json();
    return this.buildSpanWaterfall(data.data || []);
  }

  async getLatencyPercentiles(toolName: string, timeRange: string): Promise<any> {
    const response = await fetch(METRICS_URL, {
      method: "POST",
      headers: this.headers,
      body: JSON.stringify({
        data: {
          attributes: {
            formula: [{
              formula: "per_p50(duration_ns) / 1000000", // milliseconds
            }],
            queries: [{
              name: "duration_ns",
              data_source: "metrics",
              query: `avg:trace.agent.span.duration_ns{@tool.name:"${toolName}"} by {host}.rollup(avg, 60)`,
            }],
            from: this.parseTimeRange(timeRange),
            to: Date.now(),
          },
        },
      }),
    });
    return response.json();
  }

  async findErrorSpans(workflow: string, timeRange: string, errorType?: string): Promise<any[]> {
    const query = errorType 
      ? `@workflow.name:"${workflow}" @error.type:"${errorType}"`
      : `@workflow.name:"${workflow}" @error:1`;
    const response = await fetch(TRACES_URL, {
      method: "POST",
      headers: this.headers,
      body: JSON.stringify({
        filter: { query },
        sort: "-@timestamp",
        limit: 20,
        timeframe: this.parseTimeRange(timeRange),
      }),
    });
    return response.json();
  }

  private buildSpanWaterfall(spans: any[]): any[] {
    // Convert flat span list into hierarchical tree structure
    const spanMap = new Map<string, any>();
    const roots: any[] = [];
    for (const span of spans) {
      spanMap.set(span.span_id, { ...span, children: [] });
    }
    for (const span of spans) {
      if (span.parent_id && spanMap.has(span.parent_id)) {
        spanMap.get(span.parent_id).children.push(spanMap.get(span.span_id));
      } else {
        roots.push(spanMap.get(span.span_id));
      }
    }
    return roots;
  }

  private parseTimeRange(range: string): number {
    const now = Date.now();
    const units: Record<string, number> = {
      m: 60000, h: 3600000, d: 86400000,
    };
    const match = range.match(/(\d+)([mhd])/);
    if (!match) return now - 900000; // default 15m
    return now - parseInt(match[1]) * units[match[2]];
  }
}

Step 3: Client Configuration

{
  "mcpServers": {
    "datadog-observability": {
      "command": "node",
      "args": ["dist/index.js"],
      "env": {
        "DATADOG_API_KEY": "your-api-key",
        "DATADOG_APP_KEY": "your-app-key",
        "DATADOG_SITE": "datadoghq.com"
      }
    }
  }
}

Step 4: Performance Benchmarks

Production Reality Check

Rate Limits and Pagination. Datadog's Metrics API has a rate limit of 300 queries per hour on the Pro plan. The server implements response caching with a thirty-second TTL to avoid hitting limits during burst queries. The Traces API supports pagination for traces exceeding the default limit of one hundred results through a cursor-based mechanism.

Span Waterfall Depth. Agent workflows can produce deeply nested spans (tool calls within tool calls within LLM calls). The span waterfall builder recursively constructs the tree up to ten levels deep. Beyond ten levels, remaining spans are flattened into a siblings list with depth markers to prevent response bloat.

OpenTelemetry Attribute Standardization. Different agent frameworks use different span attribute naming conventions. LangGraph uses workflow.step and workflow.tool_name while CrewAI uses task.id and agent.role. The server includes an attribute normalizer that maps common naming patterns to a unified schema. For more agent debugging patterns, see HelixDB MCP Server.

Data Retention. Datadog retains APM traces for seven days on the Pro plan and fifteen days on the Enterprise plan. The server warns the user when the queried time range exceeds the account's retention window. For persistent trace storage beyond Datadog retention limits, consider exporting to a dedicated observability pipeline.

For additional MCP server patterns and agent observability techniques, explore the MCP Directory and the AI Workflows Directory.

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

Last tested and verified: September 2026 with Node version 22, FastMCP 2.1.0, TypeScript 5.6, Datadog APM Pro plan, and OpenTelemetry SDK version 1.28.