Gate Agent Deploys on Evals: Block 65% Regressions Before Users

Traditional CI asks whether code compiles and tests pass. Agents add a harder question: does it still behave. Harness ran a support agent through a blocking eval gate in September 2026 and the first run scored 65% against a 70% bar. Nothing was broken in the classical sense. The build compiled, the endpoint lived. The behavior was not good enough, so the deploy stopped.

I run Daily AI World and ship agents at SaaSNext. Direct answer:

• Gate on delta vs production baseline, not absolute thresholds — regressions are relative to what you ship today
• Three layers: 10 to 20 input smoke, stratified 50 per PR, full 500 nightly plus on main
• Shadow then canary: mirror traffic invisibly, compare with judge models, roll 1% to 10% to 50% to 100%

Here is the pipeline that catches confident wrong answers before customers do.

Why unit tests cannot hold agents

A prompt edit like "be more concise" can collapse error-checking. The agent still returns valid JSON, still answers fast, still looks healthy. Harness data shows the dangerous row precisely: relevancy 1.0 with task completion 0.3. The reply reads naturally and the fact is wrong. No linter catches that. Only behavioral evals do.

Three components make a baseline that actually regresses-catches. First, a golden dataset from real production traffic, 50 careful examples beating 500 synthetic ones, heavy on edge cases. Second, a rubric with dimensions and score levels, signed off by whoever defines done. Third, a comparison baseline: fail the build when the candidate scores statistically worse than production, not when it misses an absolute 0.8 carved in spring.

AWS Bedrock AgentCore plus GitHub Actions wires this end to end since September 8: deploy to AgentCore runtime, invoke with the eval dataset, score traces with GoalSuccessRate, Correctness, ToolSelectionAccuracy, and ToolParameterAccuracy, block the PR under threshold. My three-framework benchmark with 97 wins is the dataset mindset behind it: strict assertions on real tasks, not vibes.

Production war story 1: the concise prompt that killed faithfulness

In our support agent a well-meaning edit added "keep replies under 40 words." Unit tests passed. Latency improved 12%. The stratified eval told another story: faithfulness fell from 0.93 to 0.81, a 12-point drop concentrated in refund explanations where the agent now omitted conditions. Two days later a customer escalated a half-explained policy. Rollback took 9 minutes because artifacts were immutable. Damage took a week of trust to repair.

The eval gate would have blocked the merge. Regression exceeded our 5% delta rule on the exact dimension the prompt touched. Since then prompt diffs trigger targeted test generation: an agent step reads the diff, predicts affected behaviors, and emits focused inputs into the same pipeline. The background-tool discipline from my voice agent guide helped too: narration changes got their own eval dimension after terse replies confused callers.

Production war story 2: the judge variance false alarm

When we first gated on LLM-as-judge scores, a 0.79 against a 0.80 bar blocked a genuinely good deploy. Rerun scored 0.84. Same code, same data. Judge noise near the threshold burned half a day and taught the team to distrust the gate. Two engineers started lobbying to remove it.

Fix had three parts. Thresholds now sit with margin below target reliability to absorb judge variance. Improvements count only after holding across repeated runs. New metrics run in observation mode for weeks before becoming hard gates. Per-merge cost holds $0.15 to $0.30 with tiered judges and sampling, comparable to a mid-size integration suite. The gate survived because we calibrated it instead of worshipping it. Soft blocks first, hard blocks after evidence.

Runnable production code: eval gate with delta rule

Golden dataset, dual-run comparison, delta gate, immutable artifact id.

File 1: config.py

from pydantic_settings import BaseSettings
from pydantic import Field

class Settings(BaseSettings):
    eval_model: str = Field(default="opus-5", alias="EVAL_MODEL")
    judge_model: str = Field(default="gemini-3.8-flash", alias="JUDGE_MODEL")
    pass_threshold: float = 0.70
    max_delta: float = 0.05
    smoke_n: int = 15
    pr_n: int = 50
    per_run_budget_usd: float = 0.30

    class Config:
        extra = "allow"

settings = Settings()

File 2: gate.py

import hashlib, json, logging
from config import settings

log = logging.getLogger("eval-gate")

def artifact_id(code: str, prompt: str, model: str) -> str:
    raw = code + "||" + prompt + "||" + model
    return "art-" + hashlib.sha256(raw.encode()).hexdigest()[:12]

def delta_gate(prod_score: float, cand_score: float) -> dict:
    delta = cand_score - prod_score
    if settings.pass_threshold > cand_score:
        return {"ship": False, "reason": f"below bar {cand_score:.2f}"
                f" under {settings.pass_threshold:.2f}"}
    if -settings.max_delta > delta:
        return {"ship": False, "reason": f"regression {delta:.2f}"
                f" beyond {settings.max_delta:.2f}"}
    return {"ship": True, "reason": f"delta {delta:+.2f} within tolerance"}

def budget_guard(spent_usd: float) -> None:
    if spent_usd > settings.per_run_budget_usd:
        raise RuntimeError(f"eval budget exceeded: ${spent_usd:.2f}")

if __name__ == "__main__":
    print(artifact_id("v42", "be concise", "opus-5"))
    print(delta_gate(0.93, 0.81))
    print(delta_gate(0.75, 0.78))

File 3: workflow.yml plus requirements

name: agent-eval-gate
on:
  pull_request:
    paths: ["agents/**", "prompts/**", "evals/**"]
jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: uv pip install -r requirements.txt
      - run: python gate.py --suite smoke --n 15
  stratified:
    needs: smoke
    runs-on: ubuntu-latest
    steps:
      - run: python gate.py --suite stratified --n 50 --compare main

anthropic==0.68.0
google-genai==1.12.0
pydantic==2.8.0
pydantic-settings==2.5.0
deepeval==2.4.0

Run it:

uv pip install -r requirements.txt
python gate.py

Step 1: curate 50 golden cases from production with edge weight. Step 2: run smoke per commit, stratified per PR with delta vs main. Step 3: graduate metrics from observation to hard gate after weeks of calibration. TrueFoundry numbers frame the rest: gateway overhead near 3 to 4ms at 350 RPS, shadow compare before canary, immutable bundles of code plus prompt plus model. The approval-gated money pattern from my Stripe MCP build is the same instinct: propose, verify against policy, then execute.

When NOT to gate hard

Do not hard-gate brand-new metrics. Observation mode for weeks first, or the first false positive kills the program politically.

Do not gate abstract qualities like satisfaction. Gate measurable signals: format compliance, refusal rates on adversarial sets, accuracy on known ground truth, latency distributions. Human review owns the rest.

Do not mock every dependency. Sandbox realistic infrastructure so agents meet real schemas and real errors. Mocked perfection is a green build lying to you. The migration validation loop from the DeepSeek routing switch shows the habit: re-validate against reality after every move.

Verdict for September 2026 agent teams

Prompts are hyperparameters of behavior. Version them immutably, eval them statistically, release them progressively. The teams that ship fastest are the ones whose gates catch their own regressions first.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World. I run eval-gated agent pipelines at SaaSNext and block on deltas, not vibes. More at https://deepakbagada.in.