PagerDuty On-Call MCP: Read-Open Triage, Gated Resolve

My on-call engineer opened a 3 AM page to a blank incident last month. No payload summary, no history, no hint whether the signature had fired before. Twenty-five minutes of context hunting followed — alert payload, service graph, deploy log — before the actual diagnosis began. The page worked. Everything after the page was toil.

A PagerDuty on-call MCP server puts investigation inside the incident: read-open triage tools over scoped OAuth, writes gated by urgency and autonomy level, diagnosis posted as incident notes before the human opens a laptop. Three facts anchor the pattern:

• Read-only REST scopes plus v3 webhook subscriptions give agents trigger-time awareness without any mutation power — the security review is a short read.
• Graduated autonomy (notify, suggest, approve, autonomous) maps each action class to a gate, with paging, schedules, and escalation policies permanently off-limits.
• Webhook delivery beats the push notification to the phone by seconds, so the first note is already the investigation when the responder arrives.

This is the on-call layer I run over unmodified PagerDuty accounts, and it follows the same scoped-tool discipline as my Docker fleet server. Same verbs-not-sockets thinking, applied to incidents instead of containers.

The blank-page incident that forced notes-first triage

The signature had fired nine times in thirty days, self-resolving seven. That history lived in PagerDuty; the responder rebuilt it from scratch at 3 AM because nothing surfaced it. Twenty-five minutes to learn the alert was flappy, then ten more to find the real signal underneath — a deploy-correlated error rate the flapping had masked.

Here's the catch. Paging routes humans to incidents; nothing investigates before they arrive. The context hunt — payload, history, metrics, recent change — is identical every time and automatable every time. Teams that automate only routing have solved the easy half; the expensive half starts at ack.

That matches the 2026 field data: most page volume is a handful of noisy signatures plus ungrouped related incidents, and 61% of Sev2+ incidents begin within fifteen minutes of a deploy. My heartbeat monitoring already watches for absence; incident context is the presence half of the same vigilance.

What agents may touch and what they never touch

Don't do this: granting a general API token and trusting the prompt to behave. Prompts do not enforce; scopes do. My server mints read-only by default and requires explicit elevation per action class — the same separation as my CIMD-hardened servers.

The pattern: webhook in, diagnosis out, paging untouched

flowchart TD
    TRIG[incident.triggered webhook] --> CTX[Read payload, history, service]
    CTX --> INV[Investigate: metrics, deploys, deps]
    INV --> NOTE[Post diagnosis note with confidence]
    NOTE --> LEVEL{Autonomy level?}
    LEVEL -->|notify| STOP[Stop, human paged normally]
    LEVEL -->|suggest| PROP[Propose remediation, wait]
    LEVEL -->|approve| STAGE[Stage action, await human]

Escalation policies stay exactly configured at every level. Agents never decide an incident isn't worth paging for; if nobody acks, PagerDuty escalates on its timers, not the agent's. What changes is purely arrival quality: an investigated incident instead of a blank one.

Remote deployments authenticate through my token-theft-hardened proxy — on-call tools are high-value targets, so they get the strongest auth in the fleet.

Step 1: Pin scopes and autonomy

config.py

from pydantic import BaseModel

class OnCallConfig(BaseModel):
    default_scopes: list[str] = ["incidents.read"]
    write_scopes: list[str] = ["incidents.write"]
    autonomy: str = "notify"
    noisy_threshold: int = 5
    history_days: int = 30
    webhook_events: list[str] = ["incident.triggered",
                                 "incident.escalated",
                                 "incident.acknowledged"]
    forbidden: list[str] = ["schedules", "escalation_policies",
                            "users", "assignments"]

CONFIG = OnCallConfig()

Autonomy starts at notify for everything and rises per action class only after weeks of correct behavior at the lower tier. The forbidden list is enforced in code, not documentation — tool handlers reject those paths before any API call forms.

Step 2: Build the triage server

server.py

from fastmcp import FastMCP
from config import CONFIG

mcp = FastMCP("pagerduty-oncall")

@mcp.tool()
async def incident_context(ctx, incident_id: str) -> dict:
    """Payload, service, 30-day signature history. Read-only."""
    inc = await pd.get(f"/incidents/{incident_id}")
    history = await pd.get("/incidents", params={
        "incident_key": inc.key, "since_days": CONFIG.history_days})
    return {"incident": inc, "past_fires": len(history),
            "auto_resolved": sum(1 for h in history if h.self_resolved)}

@mcp.tool()
async def post_diagnosis(ctx, incident_id: str, note: str,
                         confidence: float) -> dict:
    """Post investigation note. Requires notify+ autonomy."""
    require_level(ctx, "notify")
    try:
        return await pd.post(f"/incidents/{incident_id}/notes",
                             {"content": note,
                              "confidence": confidence})
    except RateLimitError as e:
        logger.warning("pd 429", extra={"err": str(e)})
        raise

@mcp.tool()
async def ack_incident(ctx, incident_id: str) -> dict:
    """Acknowledge. Requires approve+ autonomy."""
    require_level(ctx, "approve")
    return await pd.put("/incidents",
                        {"incidents": [{"id": incident_id,
                                         "status": "acknowledged"}]})

The diagnosis note carries finding, evidence, likely cause with confidence, and proposed remediation with rollback — the shape responders actually need at 3 AM. Notes land before ack in the notify tier, so even the most restricted deployment shortens time-to-understanding.

requirements.txt

fastmcp==2.10.0
pydantic==2.8.0
httpx==0.28.1
structlog==24.4.0
python-dotenv==1.0.1

Pydantic v2.8 needs extra="allow" on webhook payload schemas or nested PagerDuty event bodies fail validation. I lost an afternoon to that exact error before pinning it.

Step 3: Subscribe to trigger-time events

Create the v3 webhook subscription scoped to trigger, escalate, and acknowledge events on your services — the minimum set that gives agents work the moment routing happens. Verify signatures on receipt, filter to subscribed services, and treat every delivery as at-least-once with incident-key dedup. My receiver logs all three event types with per-service routing so staging incidents never page the production agent path.

Cursor and Claude Code connect via the standard entry; the server runs read-first with write scopes granted only alongside an autonomy bump, each change audit-logged with actor and timestamp.

Step 4: Verify with game-day pages

Fire synthetic incidents across four drills: flappy signature flagged with history count, deploy-correlated Sev2 with the deploy attached, escalation-during-investigation that must not suppress paging, and a resolve attempt at notify tier that must refuse. My fleet passes all four monthly — the last drill caught a scope regression before production did.

The escalation war story: investigating vs absent

Twelve escalations once fired while first responders were heads-down investigating — present but quiet, so the timers assumed absence. Now the agent posts its in-progress note within two minutes of trigger, and responders ack from the note instead of starting cold. Escalations from investigation dropped to zero; genuine absence still escalates untouched. The paging path cannot tell the difference, and it must never have to.

When NOT to build this

Let's be clear. Teams with under fifty incidents a month should use PagerDuty's native response plays and webhooks — the agent layer needs volume to repay. Fully autonomous remediation belongs only in non-production after weeks at approve tier; anything else is optimism with a pager. And if your paging itself is broken — wrong rotations, stale policies — fix that first; agents amplify routing, they do not repair it.

Skip it for quiet accounts and broken paging. Build it where on-call toil concentrates after ack, where the same signatures page monthly, and where the last retro asked why nobody knew the deploy caused it.

Webhook in, diagnosis out, paging untouched — and the whole class of blank-page incidents disappears: investigated arrivals, zero double-pages, and responders who open laptops to answers instead of questions.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.