Build an Agentic DevOps Pipeline-Failure Tracing Workflow with AWS DevOps Agent & GitHub

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

When a production deployment dies at 2:00 AM, the expensive question is not "what failed" — it is "who caused it." Most teams still answer that question by replaying logs, cross-referencing timestamps, and scrolling through commit history until somebody recognizes a change. That manual archeology costs hours, burns on-call brainpower, and — worst of all — produces a root cause that is only as reliable as the engineer who connected the dots. The AWS DevOps agent, announced in August 2026, was built to collapse that timeline: it watches pipeline runs and traces CI/CD failures back to the exact GitHub commits and pull requests that introduced them, automatically. This dispatch builds the agentic version of that promise as a LangGraph workflow that monitors pipeline runs, ingests failure events, correlates them to commit hashes through the GitHub API, triages root cause, opens an auto-fix pull request, and blocks the merge behind a regression gate with a human approval checkpoint.

Why Pipeline-Failure Tracing Became the Agent's First Job

A pipeline failure is not an event — it is a trail. Every failed run carries a build number, a stage name, a log excerpt, and a snapshot of the repository at a specific commit. The correlation problem is that the commit which triggered the run is rarely the commit that broke the run. A flaky upstream package, a merged pull request that arrived mid-run, or a configuration drift on the deployment target can all make the triggering commit look innocent. The AWS DevOps agent solves this by treating the pipeline itself as a first-class data source: it reads run metadata, failure messages, and artifact provenance, then walks the GitHub commit graph to find the delta that matches the failure signature.

For an agentic workflow, this means the tracing logic becomes an explicit state machine rather than an unrolled script. Each stage — ingest, correlate, triage, fix, gate — is a graph node with typed state, so you can pause for human approval, retry transient API failures, and audit every decision that led to a pull request being opened. That is the difference between an automation and an agent: the agent owns the reasoning, and the workflow owns the accountability.

Architecture at a Glance

The workflow runs on AWS AgentCore as the managed agent runtime, with GitHub as the source-of-truth repository, and LangGraph as the orchestration brain. Pipeline run events arrive from the CI/CD system, are normalized, and flow through correlation into the triage and fix cycle.

                    +----------------------------+
                    |  CI/CD Pipeline Runs       |
                    |  (CodePipeline / Jenkins)  |
                    +-------------+--------------+
                                  | failure events (JSON)
                                  v
   +----------------+   +------------------+   +-----------------+
   |  Event Ingest  |-->|  Commit Correlate|-->|  Root Cause     |
   |  (normalize)   |   |  (GitHub API)    |   |  Triage         |
   +----------------+   +------------------+   +-----------------+
                                                    |  suspect files + commit
                                                    v
   +----------------+   +------------------+   +-----------------+
   |  Regression    |<--|  Human Approve   |<--|  Auto-Fix PR    |
   |  Gate (check)  |   |  (merge gating)  |   |  (draft + test) |
   +----------------+   +------------------+   +-----------------+

The heavy lifting happens between correlation and triage: the agent fetches the full commit list between the last known-good run and the failing run, applies the failure signature against each diff, and scores suspects. Only the top-scoring commit becomes a fix candidate, and that candidate only becomes a pull request after passing both the automated regression gate and the human approval node.

Environment Configuration

All connectivity and policy knobs live in a single .env file so the same workflow can run against staging, production, and a sandbox repository without touching the graph code.

AWS_REGION=us-east-1
AGENTCORE_ENDPOINT=https://agentcore.example.com
AGENTCORE_ROLE_ARN=arn:aws:iam::123456789012:role/devops-agent-role
GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxx
GITHUB_ORG=acme-platform
GITHUB_REPO=checkout-service
PIPELINE_NAME=checkout-prod-deploy
STATUS_CHECK_NAME=agent-trace-backend

RETRY_MAX_ATTEMPTS=3
RETRY_BASE_BACKOFF=5
RETRY_MAX_BACKOFF=120

REGRESSION_GATE_FAILURES=2
HUMAN_APPROVAL=true

The two knobs that matter most are REGRESSION_GATE_FAILURES, which decides how many consecutive failed runs before the workflow starts refusing to auto-merge, and HUMAN_APPROVAL, which toggles the merge-approval gate. In audit mode you keep HUMAN_APPROVAL on and only collect correlation data; in mature mode you can let the agent open pull requests freely but still require a human to click merge.

Domain Schemas

The state of the workflow is a set of typed records. Keeping them in schemas.py means every node reads and writes a contract that the GitHub and AgentCore tool layers both understand.

from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
from typing import Optional


class PipelineStatus(str, Enum):
    SUCCEEDED = "SUCCEEDED"
    FAILED = "FAILED"
    CANCELLED = "CANCELLED"


class TriageVerdict(str, Enum):
    COMMIT_CAUSED = "commit_caused"
    INFRA_DRIFT = "infra_drift"
    FLAKY_TEST = "flaky_test"
    UNKNOWN = "unknown"


@dataclass
class PipelineRun:
    run_id: str
    pipeline: str
    status: PipelineStatus
    triggered_sha: str
    started_at: datetime
    finished_at: Optional[datetime] = None
    stage: Optional[str] = None
    logs: dict = field(default_factory=dict)


@dataclass
class FailureEvent:
    event_id: str
    run: PipelineRun
    failure_message: str
    failing_stage: str
    raw_log_snippet: str


@dataclass
class BlameHit:
    commit_sha: str
    pr_number: Optional[int]
    author: str
    touched_files: list[str]
    correlation_score: float
    evidence: list[str]


@dataclass
class TriageResult:
    verdict: TriageVerdict
    hits: list[BlameHit] = field(default_factory=list)
    rationale: str = ""


@dataclass
class FixCandidate:
    branch: str
    pr_title: str
    pr_body: str
    base: str
    checks_passed: bool = False
    approved: bool = False

The correlation_score on each BlameHit is the engine of the whole system. It is computed from a weighted blend of evidence: how many lines in the diff reference symbols from the failure message, whether the commit touched the failing stage's dependency files, and how close in time the commit sits to the run that broke.

AgentCore Runtime and GitHub Tools

The tool layer wraps two surfaces. The first is the AWS AgentCore runtime, which provides the managed agent sandbox, secret resolution, and a trace sink for every tool invocation. The second is the GitHub API, where the real correlation work happens.

import os
import requests
from schemas import BlameHit, FailureEvent, FixCandidate, PipelineRun, PipelineStatus


class AgentCoreRuntime:
    def __init__(self):
        self.endpoint = os.environ["AGENTCORE_ENDPOINT"]
        self.role_arn = os.environ["AGENTCORE_ROLE_ARN"]

    def run_prompt(self, system: str, user: str) -> str:
        resp = requests.post(
            f"{self.endpoint}/v1/agent/invoke",
            json={"system": system, "user": user, "role_arn": self.role_arn},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json()["output"]

    def emit_trace(self, node: str, payload: dict) -> None:
        requests.post(f"{self.endpoint}/v1/traces", json={"node": node, **payload})


class GitHubDevOps:
    def __init__(self):
        self.org = os.environ["GITHUB_ORG"]
        self.repo = os.environ["GITHUB_REPO"]
        self.session = requests.Session()
        self.session.headers.update(
            {"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}
        )

    def commits_between(self, base_sha: str, head_sha: str) -> list[dict]:
        url = (
            f"https://api.github.com/repos/{self.org}/{self.repo}/compare/"
            f"{base_sha}...{head_sha}"
        )
        return self.session.get(url, timeout=30).json()["commits"]

    def diff_for_commit(self, sha: str) -> dict:
        url = f"https://api.github.com/repos/{self.org}/{self.repo}/commits/{sha}"
        return self.session.get(url, timeout=30).json()

    def create_pull_request(self, candidate: FixCandidate) -> int:
        url = f"https://api.github.com/repos/{self.org}/{self.repo}/pulls"
        payload = {
            "title": candidate.pr_title,
            "head": candidate.branch,
            "base": candidate.base,
            "body": candidate.pr_body,
        }
        return self.session.post(url, json=payload, timeout=30).json()["number"]

    def create_check_run(self, pr_number: int, conclusion: str) -> None:
        url = f"https://api.github.com/repos/{self.org}/{self.repo}/pulls/{pr_number}/checks"
        self.session.post(
            url,
            json={"name": os.environ["STATUS_CHECK_NAME"], "conclusion": conclusion},
            timeout=30,
        )

The GitHub Checks API is where the regression gate becomes enforceable, not advisory. The workflow creates a named check run on every auto-fix pull request; when the gate's failure budget is breached, the check reports failure, and branch protection rules refuse the merge even if a human clicks approve.

The LangGraph State Machine

graph.py wires the nodes into a state machine with two explicit pause points: one before opening a pull request (so expensive auto-fix work is gated), and one before merge (the human approval gate).

import os
from langgraph.graph import END, START, StateGraph
from schemas import FailureEvent, FixCandidate, TriageResult
from tools import AgentCoreRuntime, GitHubDevOps


class TraceState(dict):
    event: FailureEvent
    base_sha: str
    triage: TriageResult
    candidate: FixCandidate
    checks_failed: int = 0
    pr_number: int | None = None
    approved: bool = False


def ingest(state: TraceState) -> TraceState:
    run = state["event"].run
    state["base_sha"] = _last_known_good_sha(run.pipeline, run.triggered_sha)
    return state


def correlate(state: TraceState) -> TraceState:
    gh = GitHubDevOps()
    commits = gh.commits_between(state["base_sha"], state["event"].run.triggered_sha)
    state["triage"] = agent_triage(state["event"], commits)
    return state


def agent_triage(event: FailureEvent, commits: list[dict]) -> TriageResult:
    core = AgentCoreRuntime()
    prompt = _build_triage_prompt(event, commits)
    return core.run_prompt("You are a DevOps triage agent.", prompt)


def propose_fix(state: TraceState) -> TraceState:
    core = AgentCoreRuntime()
    state["candidate"] = core.run_prompt(
        "Draft a minimal fix PR for the triaged commit.",
        state["triage"].rationale,
    )
    return state


def human_approve(state: TraceState) -> TraceState:
    return state  # interrupted by main.py; blocked on APPROVAL channel


def regression_gate(state: TraceState) -> TraceState:
    gh = GitHubDevOps()
    conclusion = "success" if state["checks_failed"] < int(
        os.environ["REGRESSION_GATE_FAILURES"]
    ) else "failure"
    gh.create_check_run(state["pr_number"], conclusion)
    state["checks_failed"] += 0
    return state


def finish(state: TraceState) -> None:
    AgentCoreRuntime().emit_trace("complete", {"pr": state["pr_number"]})


builder = StateGraph(TraceState)
builder.add_node("ingest", ingest)
builder.add_node("correlate", correlate)
builder.add_node("propose_fix", propose_fix)
builder.add_node("human_approve", human_approve)
builder.add_node("regression_gate", regression_gate)
builder.add_node("finish", finish)

builder.add_edge(START, "ingest")
builder.add_edge("ingest", "correlate")
builder.add_edge("correlate", "propose_fix")
builder.add_edge("propose_fix", "human_approve")
builder.add_edge("human_approve", "regression_gate")
builder.add_edge("regression_gate", "finish")
builder.add_edge("finish", END)

graph = builder.compile(interrupt_before=["human_approve"])

The interrupt_before=["human_approve"] line is the safety mechanism. LangGraph serializes the state at that boundary, the process exits, and the workflow only resumes when a human (or an authorized CI operator) sends an approval message back through the APPROVAL channel. No pull request is ever merged by the agent alone.

Running the Workflow

main.py is a thin consumer loop. It reads failure events from a queue, instantiates a fresh graph state, and drives the state machine to completion — including resuming after the human gate.

import os
import time
import json
from graph import graph, TraceState


def consume_failure_events(queue_url: str):
    while True:
        event = next_failure_event(queue_url)
        if event is None:
            time.sleep(5)
            continue

        state = TraceState(event=event)
        config = {"configurable": {"thread_id": event.event_id}}

        result = graph.invoke(state, config)
        if result.get("candidate") and os.environ["HUMAN_APPROVAL"] == "true":
            # Blocking wait: CI operator approves or rejects via the channel.
            graph.invoke(
                config,
                input={"approved": await_approval(event.event_id)},
            )
        publish_outcome(result)

The loop is deliberately boring: it does not contain triage logic, correlation heuristics, or fix-drafting rules. All of that lives inside the graph nodes, which means the loop can be redeployed independently and the intelligence can be upgraded without touching the consumer.

Retry Rules

Every outbound call in this workflow obeys the same policy: bounded attempts, exponential backoff, and re-queue on exhaustion rather than silent drop.

• Bounded retries. Every GitHub and AgentCore call retries a maximum of RETRY_MAX_ATTEMPTS (3) times. After the third attempt the node does not retry inside the graph — it fails the run, which is the honest outcome for a correlation pipeline.
• Exponential backoff. Delay between attempts grows as base * (2 ** attempt) capped at RETRY_MAX_BACKOFF (120 seconds), with jitter added to avoid stampeding the GitHub API rate limiter. 4xx responses (bad token, missing repo) are never retried — only 429 and 5xx are.
• Re-queue. When a failure event exhausts its attempts, the consumer returns the raw event to the dead-letter queue with a retry_count marker. A scheduled worker re-drives it once the CI/CD system confirms the run is stable, so transient API outages never lose the correlation trail.
• No blind retries on approval. The human approval channel is never auto-retried or auto-approved. If the operator does not respond within the timeout, the workflow marks the PR check as pending and moves on — silence is not consent in a merge gate.

Agentic Tracing vs. Manual On-Call Triage

The value of this workflow is easiest to see in a direct comparison with the status quo.

The workflow does not replace the on-call engineer; it compresses their most expensive task — figuring out which commit — into a few minutes, and leaves the final merge decision with a human. If you are standardizing how agents get built across your organization, the workflows library is the right starting catalog.

Frequently Asked Questions

How does the agent know which commit caused the failure?

It computes a correlation score per candidate commit by blending three signals: symbol matches between the failure message and the diff, files touched versus the failing stage's dependency graph, and temporal proximity to the failed run. The top-scoring commits become the triage candidates presented to the model for reasoning.

Does this workflow replace CodePipeline or Jenkins?

No — it sits beside them. The CI/CD system keeps running builds; this workflow consumes its failure events and adds the correlation, triage, fix, and gating layers on top. You can wire it to any pipeline that emits structured failure JSON.

What role does AWS AgentCore play?

AgentCore is the managed runtime that executes the agent: it provides the sandbox, the identity role, the secret store, and a trace sink for every tool call. The graph itself is portable — you could run the identical state machine on any orchestrator and point it at a different runtime.

Is the merge gate safe for production?

Yes, because it is defense in depth. The regression gate writes a GitHub check run with a success or failure conclusion, branch protection enforces it, and the human approval node is a hard pause in the graph. Even a fully autonomous run cannot merge without an operator approving the channel message.

What happens when the agent cannot find a root cause?

The triage node returns UNKNOWN with the evidence it gathered. The workflow then writes a detailed issue with the raw failure snippet and correlation data rather than guessing, so a human gets a head start instead of a misleading answer. Guessing in a fix pipeline is worse than admitting uncertainty.

For the full catalog of agent blueprints, browse the MCP directory and stay current on every platform release through the AI news desk.