By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

Build a Coding-Agent Benchmarking Workflow with SWE-Bench Pro Regression Gates

Every AI platform team in 2026 has the same problem: your coding agent aced the demo, so you promoted it to production — and three weeks later it regressed on a class of tasks you never tested, and your merge rate silently collapsed. The reason is structural. Agent quality is a moving target: a model update, a system-prompt tweak, a new tool-version, or a changed evaluation harness can shift pass rates 10 points overnight. If you are not benchmarking continuously, you are deploying on vibes.

This guide builds a Coding-Agent Benchmarking Workflow in LangGraph that answers one question before any agent — or any new model version of an existing agent — gets promoted: does it still pass the bar on the tasks that matter? It constructs a golden task suite, runs agents across that suite in parallel (Map-Reduce style), scores each run, computes cost-per-task, diffs the results against the pinned baseline (regression gates), generates a regression report, and ends in a hard go/no-go promotion gate. No promotion, no merge, no rollout, no gate bypass.

The pattern is the agent-era equivalent of SWE-bench CI. If you are newer to these orchestration shapes, our AI workflows hub covers the full catalog, and the MCP directory lists the task-runner tool servers you'll wire into the harness. To track the model-version churn that makes this necessary, latest AI news is updated daily.

The Case for an Agent Benchmark Loop

SWE-bench gave the field a fixed set of real GitHub issues to grade agents against. "SWE-bench Pro" — the operational discipline — takes the idea and makes it continuous and yours:

1. Your tasks, not the public set. Public benchmarks optimize for leaderboards. Your golden suite is built from your repo's resolved issues, your own regressions, and your own security patterns.
2. Regression gates, not just scores. The question isn't "is the new version better than the old?" — it's "did it get worse on anything that shipped to customers?" You must catch a drop on critical task classes even when the aggregate score goes up.
3. Cost per task as a first-class metric. A 3-point pass-rate gain that costs 4x per task is a business decision, not an engineering decision. The workflow reports it either way.
4. Model-version pinning. Every run records the exact agent version, model ID, harness commit, and prompt hash. Unpinned benchmarks are un-auditable, and un-auditable gates get bypassed.

The Workflow at a Glance

flowchart TD
    A[START] --> B[task_builder]
    B --> C[parallel_runner]
    C --"fan-out"--> W1[worker_task_0]
    C --"fan-out"--> W2[worker_task_1]
    C --"fan-out"--> W3[worker_task_N]
    W1 --> R[(Results Ledger)]
    W2 --> R
    W3 --> R
    R --> D[scorer]
    D --> E{regression gate}
    E --"no critical regression + pass rate >= threshold"--> F[regression_report_generator]
    E --"critical regression"--> F
    F --> G[promotion_gate]
    G --"GO"--> H[promote / rollout]
    G --"NO-GO"--> I[hold + flag to team]
    H --> J[END]
    I --> J

The map is the parallel_runner (fan-out of worker nodes, one per task), the reduce is the scorer (aggregating into pass-rate, cost, and per-class breakdowns). Everything else — gates and reports — hangs off that aggregation.

Node by Node

task_builder

Builds the golden suite from three sources: a curated golden_tasks.jsonl (hand-marked, high value), a sample of recently fixed issues pulled from your tracker, and regression-prone classes flagged in past reports. Each task has a criticality class (P0-critical, P1, P2), an acceptance command (the test that must pass), and a max runtime. The builder pins the suite to a commit hash so comparisons across runs are apples-to-apples.

parallel_runner (Map-Reduce)

Fans out one worker node per task — each worker boots a clean fixture environment, invokes the agent under test, applies the produced patch, and runs the acceptance command. Because workers are independent, this is trivially parallel; a 200-task suite that took 40 hours serial drops to ~2 hours at 20-way concurrency. This is the single biggest operational win in the workflow, and LangGraph's fan-out/join (Send/Reduce) makes it a few lines instead of a job-scheduler project.

scorer

The reduce step. Aggregates raw worker results into: overall pass rate, pass rate by criticality class, cost per task and per class, mean iterations-to-solve, and the 90th-percentile latency. It also computes the delta vs the pinned baseline (the last approved agent/model version) — the input to the regression gate.

regression_report_generator

Produces the artifact humans actually read: a table of per-class pass rates (new vs baseline), a ranked list of regressed tasks with the failing test output, cost deltas, and the exact versions under test. This report is what gets attached to a rollout ticket — it makes the promotion decision auditable, which is what separates a gate from a checkbox.

promotion_gate

The go/no-go decision. Default logic: NO-GO if any P0-critical task regressed, if overall pass rate is below the floor threshold, or if cost-per-task exploded beyond the ceiling. Otherwise GO. The gate is configurable per team, but the shape — hard stop on critical regression, automatic hold, artifact attached — never varies.

The Multi-File Implementation

# .env
BENCHMARK_REPO=/opt/agent-bench
FIXTURE_REPO_URL=git@github.com:org/core-service.git
GOLDEN_TASKS=config/golden_tasks.jsonl
MAX_PARALLEL_WORKERS=20
WORKER_TIMEOUT_S=1800
PASS_RATE_FLOOR=0.70
COST_PER_TASK_CEILING_USD=1.50
PINNED_MODEL_ID=frontier-code-2026.08.04
PINNED_HARNESS_COMMIT=$(git -C . rev-parse HEAD)
REPORT_OUTPUT=reports/
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00/B00/xxxx

# ============================================================
# schemas.py
# ============================================================
from __future__ import annotations
import datetime as dt
from enum import Enum
from typing import Optional
from pydantic import BaseModel, Field


class Criticality(str, Enum):
    P0_CRITICAL = "P0-critical"
    P1 = "P1"
    P2 = "P2"


class GoldenTask(BaseModel):
    id: str
    title: str
    criticality: Criticality
    setup_cmd: str = ""
    patch: str = ""                    # agent's output for this task
    acceptance_cmd: str = Field(..., description="test that must pass")
    max_runtime_s: int = 1800
    fixture_ref: str = ""


class WorkerResult(BaseModel):
    task_id: str
    passed: bool
    iterations: int
    latency_s: float
    cost_usd: float
    log_tail: str = ""
    error: Optional[str] = None


class SuiteResult(BaseModel):
    run_id: str
    started_at: dt.datetime
    agent_version: str
    model_id: str
    harness_commit: str
    total_tasks: int
    passed: int
    pass_rate: float
    cost_per_task_usd: float
    by_criticality: dict[str, dict] = Field(default_factory=dict)
    deltas_vs_baseline: dict[str, float] = Field(default_factory=dict)
    regressed_tasks: list[str] = Field(default_factory=list)


class PromotionDecision(BaseModel):
    decision: str = "NO-GO"            # GO | NO-GO
    reasons: list[str] = Field(default_factory=list)
    report_path: str = ""


class BenchState(BaseModel):
    tasks: list[GoldenTask] = Field(default_factory=list)
    results: dict[str, WorkerResult] = Field(default_factory=dict)   # reduce field
    suite: Optional[SuiteResult] = None
    decision: Optional[PromotionDecision] = None
    baseline: Optional[SuiteResult] = None
    attempts: int = 0
    last_error: Optional[str] = None

# ============================================================
# tools.py — environment, agent invocation, cost tracking
# ============================================================
from __future__ import annotations
import asyncio, hashlib, json, os, subprocess, time
from schemas import GoldenTask, WorkerResult

WORKER_TIMEOUT_S = int(os.getenv("WORKER_TIMEOUT_S", "1800"))


def backoff(attempt: int) -> float:
    import random
    return min(2 ** attempt, 30.0) * (0.5 + random.random() / 2)


def provision_fixture(task: GoldenTask) -> str:
    """Clean checkout of the pinned repo at the fixture ref. Idempotent."""
    ref = task.fixture_ref or os.getenv("PINNED_MODEL_ID", "HEAD")
    workdir = f"/tmp/bench-{hashlib.md5(task.id.encode()).hexdigest()[:8]}"
    if not os.path.isdir(workdir):
        subprocess.run(["git", "clone", "-q", os.environ["FIXTURE_REPO_URL"], workdir],
                       check=True, timeout=300)
    subprocess.run(["git", "-C", workdir, "checkout", "-q", ref], check=True, timeout=60)
    if task.setup_cmd:
        subprocess.run(["sh", "-c", task.setup_cmd], cwd=workdir, check=True, timeout=300)
    return workdir


async def run_worker(task: GoldenTask, agent_cmd: str,
                     budget_usd: float) -> WorkerResult:
    """One golden task, one environment, one acceptance run. Retried 2x on infra errors."""
    for attempt in range(2):
        try:
            wd = provision_fixture(task)
            started = time.monotonic()
            proc = await asyncio.create_subprocess_shell(
                f"{agent_cmd} --task {task.id} --patch-out /tmp/{task.id}.patch",
                cwd=wd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
            )
            try:
                out, _ = await asyncio.wait_for(proc.communicate(), timeout=WORKER_TIMEOUT_S)
            except asyncio.TimeoutError:
                proc.kill()
                return WorkerResult(task_id=task.id, passed=False, iterations=0,
                                    latency_s=WORKER_TIMEOUT_S, cost_usd=budget_usd,
                                    log_tail="worker timeout", error="timeout")

            patch = f"/tmp/{task.id}.patch"
            accepted = subprocess.run(["sh", "-c", task.acceptance_cmd],
                                      cwd=wd, capture_output=True, text=True,
                                      timeout=int(task.max_runtime_s))
            return WorkerResult(
                task_id=task.id,
                passed=accepted.returncode == 0,
                iterations=int(proc.returncode) if proc.returncode else 1,
                latency_s=time.monotonic() - started,
                cost_usd=budget_usd * (0.5 if accepted.returncode == 0 else 1.0),
                log_tail=out.decode()[-2000:],
            )
        except (subprocess.CalledProcessError, FileNotFoundError):
            if attempt == 1:
                return WorkerResult(task_id=task.id, passed=False, iterations=0,
                                    latency_s=0.0, cost_usd=budget_usd,
                                    log_tail="fixture provisioning failed", error="infra")
            await asyncio.sleep(backoff(attempt))


def load_golden_tasks() -> list[GoldenTask]:
    path = os.environ["GOLDEN_TASKS"]
    tasks = []
    with open(path) as fh:
        for line in fh:
            if line.strip():
                tasks.append(GoldenTask(**json.loads(line)))
    return tasks

# ============================================================
# graph.py
# ============================================================
from __future__ import annotations
import json, os
from langgraph.graph import StateGraph, START, END
from langgraph.types import Send
from schemas import (BenchState, GoldenTask, WorkerResult, SuiteResult,
                     PromotionDecision, Criticality)
import tools

FLOOR = float(os.getenv("PASS_RATE_FLOOR", "0.70"))
COST_CEILING = float(os.getenv("COST_PER_TASK_CEILING_USD", "1.50"))


async def task_builder(state: BenchState) -> dict:
    tasks = tools.load_golden_tasks()
    return {"tasks": tasks}


def fan_out(state: BenchState) -> list[Send]:
    """Map step: one worker per golden task, bounded by the parallel cap."""
    cap = int(os.getenv("MAX_PARALLEL_WORKERS", "20"))
    agent_cmd = os.getenv("AGENT_CMD", "agent-run")
    budget = COST_CEILING
    return [Send("worker_task", {"task": t, "agent_cmd": agent_cmd,
                                 "budget_usd": budget})
            for t in state.tasks[:cap]]


def reduce_results(left: dict, right: dict) -> dict:
    """Reduce step for the shared results ledger."""
    left.update(right)
    return left


async def worker_task(payload: dict) -> dict:
    task: GoldenTask = payload["task"]
    result = await tools.run_worker(task, payload["agent_cmd"], payload["budget_usd"])
    return {"results": {task.id: result}}


async def scorer(state: BenchState) -> dict:
    results = state.results
    passed = sum(1 for r in results.values() if r.passed)
    total = max(len(results), 1)
    by_crit: dict[str, dict] = {}
    for c in Criticality:
        subset = [r for t, r in results.items()
                  if next(x for x in state.tasks if x.id == t).criticality == c]
        if subset:
            by_crit[c.value] = {
                "passed": sum(1 for r in subset if r.passed),
                "total": len(subset),
                "rate": sum(1 for r in subset if r.passed) / len(subset),
            }
    suite = SuiteResult(
        run_id=f"run-{state.attempts}",
        started_at=__import__("datetime").datetime.utcnow(),
        agent_version=os.getenv("AGENT_VERSION", "dev"),
        model_id=os.getenv("PINNED_MODEL_ID", "unknown"),
        harness_commit=os.getenv("PINNED_HARNESS_COMMIT", "unknown"),
        total_tasks=total,
        passed=passed,
        pass_rate=passed / total,
        cost_per_task_usd=sum(r.cost_usd for r in results.values()) / total,
        by_criticality=by_crit,
        regressed_tasks=[t for t in results if not results[t].passed],
    )
    return {"suite": suite}


def regression_check(state: BenchState) -> str:
    suite = state.suite
    reasons = []
    for c in (Criticality.P0_CRITICAL,):
        rate = suite.by_criticality.get(c.value, {}).get("rate", 1.0)
        if rate < 1.0:
            reasons.append(f"{c.value} pass rate {rate:.0%} < 100%")
    if suite.pass_rate < FLOOR:
        reasons.append(f"overall pass rate {suite.pass_rate:.0%} < floor {FLOOR:.0%}")
    if suite.cost_per_task_usd > COST_CEILING:
        reasons.append(f"cost/task ${suite.cost_per_task_usd:.2f} > ceiling ${COST_CEILING:.2f}")
    # -> 'report' either way; the gate uses the reasons list.
    state.suite.regressed_tasks = suite.regressed_tasks
    state.suite.deltas_vs_baseline = _deltas(suite, state.baseline)
    return "report"


async def regression_report_generator(state: BenchState) -> dict:
    suite = state.suite
    lines = [
        f"# Bench Report {suite.run_id}",
        f"model: {suite.model_id} | harness: {suite.harness_commit[:8]}",
        f"pass rate: {suite.pass_rate:.0%} ({suite.passed}/{suite.total_tasks})",
        f"cost/task: ${suite.cost_per_task_usd:.2f}",
        "## by criticality",
    ]
    for c, v in suite.by_criticality.items():
        lines.append(f"- {c}: {v['rate']:.0%} ({v['passed']}/{v['total']})")
    lines.append("## regressed tasks")
    lines += [f"- {t}" for t in suite.regressed_tasks]
    os.makedirs(os.environ["REPORT_OUTPUT"], exist_ok=True)
    path = f"{os.environ['REPORT_OUTPUT']}/{suite.run_id}.md"
    with open(path, "w") as fh:
        fh.write("
".join(lines))
    return {"decision": PromotionDecision(decision="NO-GO", reasons=["pending"],
                                         report_path=path)}


async def promotion_gate(state: BenchState) -> dict:
    suite, decision = state.suite, state.decision
    reasons = []
    p0 = suite.by_criticality.get(Criticality.P0_CRITICAL.value, {}).get("rate", 1.0)
    if p0 < 1.0:
        reasons.append("P0-critical regression detected — promotion blocked")
    if suite.pass_rate < FLOOR:
        reasons.append(f"pass rate {suite.pass_rate:.0%} below floor {FLOOR:.0%}")
    if suite.cost_per_task_usd > COST_CEILING:
        reasons.append(f"cost/task ${suite.cost_per_task_usd:.2f} above ceiling")
    go = not reasons
    decision = PromotionDecision(
        decision="GO" if go else "NO-GO",
        reasons=reasons or ["all thresholds satisfied"],
        report_path=decision.report_path,
    )
    print(f"[gate] {decision.decision}: {decision.reasons}")
    return {"decision": decision}


builder = StateGraph(BenchState)
builder.add_node("task_builder", task_builder)
builder.add_node("worker_task", worker_task)
builder.add_node("scorer", scorer)
builder.add_node("regression_report_generator", regression_report_generator)
builder.add_node("promotion_gate", promotion_gate)

builder.add_edge(START, "task_builder")
builder.add_conditional_edges("task_builder", fan_out, ["worker_task"])
builder.add_edges(["worker_task"], "scorer", reducer=reduce_results)
builder.add_edge("scorer", "regression_report_generator")
builder.add_edge("regression_report_generator", "promotion_gate")
builder.add_edge("promotion_gate", END)

benchmark = builder.compile()

# ============================================================
# main.py
# ============================================================
from __future__ import annotations
import asyncio, os
from schemas import BenchState
from graph import benchmark


async def main() -> None:
    # Baseline pin: the last GO-approved suite is loaded from disk.
    baseline = None
    if os.path.exists("reports/last_approved.json"):
        baseline = __import__("json").load(open("reports/last_approved.json"))

    result = await benchmark.ainvoke(BenchState(baseline=baseline))
    decision = result["decision"]
    suite = result["suite"]
    print(f"run_id={suite.run_id} pass_rate={suite.pass_rate:.0%} "
          f"cost/task=${suite.cost_per_task_usd:.2f}")
    print(f"GATE: {decision.decision}")
    for reason in decision.reasons:
        print(" -", reason)
    if decision.decision == "GO":
        print("Promote agent to production shadow rollout.")
    else:
        print("Hold promotion. Attach report to follow-up ticket.")


if __name__ == "__main__":
    asyncio.run(main())

Retry Rules & Error Handling

Benchmarking must never let infrastructure noise masquerade as a score. Every infra-level failure is retried; every agent-level failure is recorded, not retried silently:

The golden rule: a NO-GO default on any orchestration failure. If the harness cannot prove the agent passed, the agent does not promote. Timeouts count as failures (never auto-excluded), and only infra-tagged results are dropped from scoring — and always shown in the report so nobody hides a broken harness behind "pass rates."

Thresholds & Go/No-Go Matrix

Run this on every candidate — new agent, new model version, new system prompt, new tool release — and keep the last-approved suite as your pinned baseline. That is the whole loop: build tasks → parallel-run → score → diff against baseline → gate → report. Do it every release and promotion becomes a number on a report instead of a gut call, which is exactly how you ship agents you can actually sleep through.