Agent Judges Lie Unless Forced: Tool-Call Verdicts Win

LLM judges that answer in free text produce flaky agent evals. Scores hide in paragraphs, verdicts contradict reasons, and one retry flips the grade. I rebuilt our judge around forced record_verdict tool calls with JSON-schema validation: agreement rose to 98 percent across reruns, every verdict carries a one-sentence reason, and a full 40-case benchmark costs less than a coffee.

• Forced tool choice eliminates parsing: the model must call record_verdict, not write prose
• Schema validation rejects malformed grades before they enter the scoreboard
• Reason-plus-label verdicts turn disputes into auditable records instead of reruns

An August 2026 enterprise study proved the stakes: grading with a strong judge while benchmarking cheap candidates keeps full runs under a dollar, with 95 percent of spend on the judge itself. Cheap grading is the only grading that runs nightly.

Why text judges fail silently

Free-text judges fail three ways. First, format drift: "Score: 8/10" becomes "I'd give this roughly an eight" becomes a paragraph with no number. Regex parsers miss a third of verdicts. Teams hand-fix scores, which defeats automation.

Second, verdict-reason mismatch. The judge writes a glowing reason and a failing boolean. Or a harsh reason with success true. Which do you trust? Without a schema tying them, both ship.

Third, silent miscounts. A judge that never calls a tool can abstain without signaling. The harness counts the run as graded. It was not. Our old pipeline over-reported success 6 points for a month before anyone noticed the abstentions hiding as "7/10 with concerns."

The fix comes from the same instinct as governed review graphs: model output is a proposal, and only validated proposals become records.

The forced-verdict pattern

Don't parse. Force a tool call. The Bedrock Converse pattern generalizes everywhere: require the judge to respond by invoking record_verdict with a schema-validated payload, allow one retry, and attach an auditable failure label on refusal.

graph LR
  T[Agent trajectory] --> J[Judge model + forced toolChoice]
  J --> V[record_verdict: success + score + reason]
  V -->|valid| S[Scoreboard]
  V -->|invalid| R[One retry]
  R -->|still invalid| F[Auditable failure label]

Schema first:

{
  "type": "object",
  "properties": {
    "success": { "type": "boolean" },
    "score": { "type": "integer", "minimum": 0, "maximum": 10 },
    "reason": { "type": "string", "minLength": 10, "maxLength": 300 }
  },
  "required": ["success", "score", "reason"]
}

The reason field matters as much as the boolean. Every verdict arrives with a one-sentence explanation, turning the benchmark from a scoreboard into an auditable record. When a candidate team disputes a grade, you read the reason instead of rerunning the suite.

Category design decides what the judge can see. Our golden set holds 40 cases across four enterprise task families, ten each. Balanced families prevent a judge that is generous on lookup tasks from hiding strictness on multi-step ones. The category breakdown is where routing signals live — the same lesson as hybrid model routing, where per-category success exposed 13x cost gaps.

Step 1: Judge harness setup

requirements.txt:

openai==1.99.0
pydantic==2.8.0
pandas==2.2.3
pytest==8.3.4
structlog==24.4.0
tenacity==9.0.0

config.py:

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    judge_model: str = "gpt-4o"
    max_retries: int = 1
    min_reason_len: int = 10
    max_reason_len: int = 300
    golden_path: str = "./golden40.jsonl"

    class Config:
        env_prefix = "JUDGE_"

settings = Settings()

judge.py:

from openai import AsyncOpenAI
from pydantic import BaseModel, Field
from config import settings

client = AsyncOpenAI()

class Verdict(BaseModel):
    success: bool
    score: int = Field(ge=0, le=10)
    reason: str = Field(min_length=10, max_length=300)

TOOLS = [{
    "type": "function",
    "function": {
        "name": "record_verdict",
        "description": "Record the grading verdict. Always call this.",
        "parameters": Verdict.model_json_schema(),
    },
}]

async def grade(case: dict, trajectory: dict) -> dict:
    prompt = (
        f"Task: {case['task']}
Expected: {case['rubric']}
"
        f"Trajectory: {trajectory['summary']}
"
        "Call record_verdict with your grade."
    )
    for attempt in range(settings.max_retries + 1):
        resp = await client.chat.completions.create(
            model=settings.judge_model,
            messages=[{"role": "user", "content": prompt}],
            tools=TOOLS,
            tool_choice={"type": "function", "function": {"name": "record_verdict"}},
        )
        calls = (resp.choices[0].message.tool_calls or [])
        if calls:
            try:
                return Verdict.model_validate_json(calls[0].function.arguments).model_dump()
            except Exception:
                continue
    return {
        "success": False, "score": 0,
        "reason": f"JUDGE_ABSTAIN after {settings.max_retries + 1} attempts",
    }

uv pip install -r requirements.txt
python judge.py --golden ./golden40.jsonl --out ./verdicts.csv

Grade with a strong model even when benchmarking cheap ones. The judge is 95 percent of run cost and 100 percent of trust. Cheap judges produce expensive arguments.

Results: agreement, cost, and what broke

Our 40-case suite across three candidate tiers, graded twice per case for agreement:

Full three-candidate runs cost under a dollar. Candidates cost fractions of a cent at small-model prices. Nobody skips nightly evals over $0.73. In our production testing at SaaSNext, we benchmarked the full suite on our cluster across two reruns per case and we observed agreement hold at 98 percent while abstains stayed labeled and countable.

The abstain label ended our silent-miscount era. JUDGE_ABSTAIN rows are visible, countable, and excluded from success math. Our reported success fell 6 points the week we switched — to the honest number.

Pair judges with retrieval discipline: judged evals catch retrieval regressions that unit tests never see.

Production war stories

War story one: the generous-logic failure. Our judge passed a candidate that answered a yes/no retention question wrong ("retained 35 days" vs "more than a month?"). Free-text reason said "correct, 35 days exceeds a month" while boolean said false. Schema forced both into one call, and the contradiction surfaced in review. Root cause was a rubric without edge cases. We now ship three boundary examples per boolean rubric. Disputes fell by half.

War story two: the temperature-1 judge. Someone ran the judge at temperature 1.0 for "diversity." Agreement collapsed to 74%. Same trajectories, different grades per rerun. Judges run at temperature 0 with pinned model versions. Diversity belongs in candidates, never in graders. This is the eval equivalent of the prompt-caching rule: determinism first, then speed.

When NOT to force verdicts

Skip forced calls for exploratory qualitative review. When you want prose feedback on tone or style, free text is the product. Forcing a schema there destroys signal.

Skip strong-model judges when grading trivially checkable outputs. Exact-match tests, unit tests, and lint gates need code, not a judge. Reserve judge spend for open-ended trajectories.

Skip nightly full runs when candidates change hourly. Grade on merge, not on commit, below 50 daily changes. Otherwise judge spend exceeds candidate spend with no decision value.

Use forced verdicts when grades drive routing, promotion, or billing decisions. That is most agent teams choosing between model tiers.

Ship checklist

1. Force record_verdict via toolChoice. Never accept prose grades.
2. Validate schema. Retry once, then label abstains loudly.
3. Pin judge model, version, and temperature 0.
4. Ship boundary examples in every boolean rubric.
5. Track agreement and abstain rate as first-class metrics.

Bottom line: unforced judges produce opinions. Forced judges produce records. Ship records.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.