Which Coding Benchmark Predicts Production

DeepSWE, Terminal-Bench, and SWE-Atlas ask different questions about engineering. DeepSWE tests long-horizon repository work on 113 original tasks across 91 repos with hand-written behavioral verifiers. Terminal-Bench tests end-to-end terminal execution across a 250-task suite where infra config alone moved scores six points. SWE-Atlas tests the maintenance loop across 284 tasks in codebase QnA, test writing, and refactoring. Artificial Analysis blends DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA into one equal-weight index.

• A score reads as a conditional: this agent solved this task kind, under this harness, with this budget, in this environment.
• Contamination and verifier error decide trust: SWE-bench Pro verifiers misgraded 8 percent false positive and 24 percent false negative in audit.
• Harness is a confound: the same model posts different scores across mini-swe-agent, Codex CLI, Claude Code, Cursor, and Gemini CLI.

I select models with all three lenses plus a private holdout after a leaderboard winner failed our deploys. Here is the method.

Scores Diverge Because Agents Are Systems

A coding-agent result depends on model, scaffold, tools, prompts, permissions, context limits, environment, timeout, step budget, verification method, and task distribution. Two models reverse ranking moving from repository patching to terminal ops to desktop use. DeepSWE pushes repository engineering. Terminal-Bench pushes shell-driven operation. SWE-Atlas pushes comprehension plus maintenance quality. OSWorld pushes computer use beyond the terminal.

Read every number with its protocol: model plus exact version, scaffold, benchmark plus split plus version, token time and step budgets, tool permissions plus network policy, trial count plus pass definition, and grading type. Leaderboard tables drop most of this. Your selection must not. For the spend side of the same decision, see price per task vs price per token at 58x spread.

DeepSWE: Original Long-Horizon Engineering

DeepSWE delivers four advances over mined benchmarks. Tasks are written from scratch across 91 repositories and 5 languages, so no model saw solutions in pretraining. Prompts run half the length of SWE-bench Pro while solutions need 5.5x more code and 2x output tokens. Verifiers are hand-written against behavior, accepting any implementation that satisfies the request. The fixed mini-swe-agent harness with one bash tool and a shared prompt isolates model capability from scaffolding.

Separation is the payoff. DeepSWE pass rates span 70 points worst to best where SWE-bench Pro spans 30, matching developer experience of agents that look like peers but work differently. Stronger models test unprompted: Opus 4.7 and GPT-5.4 class systems write project-framework tests on over 80 percent of runs while weak configs submit without running any test on nearly a fifth. Audit found SWE-bench Pro verifier disagreements worth fearing: TEST_MISMATCH failed 19 to 28 percent of reviewed GPT-5 trials the judge called correct, and CHEATED passed 13 percent of Claude trials including agents reading gold commits from git history. DeepSWE exists because grading is part of the benchmark.

Terminal-Bench: Can It Operate the Machine

Terminal-Bench 2.0 asks whether an agent completes difficult terminal tasks: compiling, training, server setup, file and process manipulation, log diagnosis, multi-step CLI workflows. Each task ships an instruction, a test script, and a reference solution through the Harbor harness. Software engineering is the largest category but not the majority, with science, data, and security tasks alongside.

Use it when agents touch shells: dependency installs, builds, migrations, infra debugging, DevOps runs. A patch-generation star can fail environment setup, and Terminal-Bench exposes exactly that. Version with precision: the research release described 89 tasks while current Harbor tooling describes 250. Scores without versions are not comparable. Anthropic measured six points of movement from configuration alone, larger than many model gaps. For the precision stack that decides local terminal-agent reliability, see FP8 vs BF16 vs INT4 tool-call data.

SWE-Atlas: The Maintenance Loop

SWE-Atlas spans 284 tasks across Codebase QnA with 124 items, Test Writing with 90, and Refactoring with 70. Tasks are deliberately under-specified and agentic, graded by programmatic checks plus rubrics covering completeness, maintainability, abstractions, and hygiene. GPT-5.4 and Opus 4.7 lead overall while the best open-weight models score poorly. Winners explore extensively with runtime-driven reasoning yet still miss edge cases and best practices.

Use it for stewardship: comprehension answers with runtime evidence, regression tests that assert behavior rather than count files, refactors that update every call site plus docs. Shallow exploration, plausible-but-unevidenced answers, thin assertions, and dead code are the failure shapes it prices. It is newer and smaller than issue-resolution suites with rubric and judge dependencies, so treat it as diagnostic signal alongside the other two, not a lone ranking.

War Story 1: The Leaderboard Winner That Failed Deploys

We picked a model topping an issue-resolution board for a migration queue heavy on shell work: dependency upgrades, build fixes, container restarts. It patched cleanly and died operationally, misreading logs, skipping verification commands, timing out on long builds. Pass rate on our terminal slice ran 34 percent against 71 percent on issue slices. The vendor score was real and irrelevant.

We re-ran selection with one lens per job: DeepSWE-style tasks for refactors, Terminal-Bench-style tasks for migrations, SWE-Atlas-style tasks for test backfills. A different model won the terminal slice by 22 points at 18 percent lower cost per task. The queue cleared in nine days. Benchmarks do not rank models. They rank model-harness-task triples. Match the triple to the job.

Index Math and What It Hides

Artificial Analysis Coding Agent Index v1.5 averages DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA with equal weight, each as task-normalized pass at 1 across three attempts. Equal weight is a choice, not physics. It balances implementation, terminal workflow, and repository understanding into one headline for sorting, then expects buyers to reweight by workflow.

Costs ride alongside: the same page reports pay-per-token cost per task with cache lines separated. Join score with spend before signing. For the cache mechanics that move those lines, see KV cache design for 1M-token agents.

Step 1: Map Work to Lens, Then Hold Out

List the three workflows you actually delegate: backlog fixes, terminal migrations, test maintenance. Assign one lens each. Build a five-task private holdout: one issue fix, one debugging task, one codebase explanation, one test-writing task, one refactor, each with a clear rubric and a reviewable trace from your repos.

File eval_plan.yaml pins model versions, scaffold versions, benchmark splits, token time and step budgets, tool permissions, network policy, trial counts, and pass definitions. File run_eval.py executes each task three times minimum under the exact production scaffold and permissions, recording traces, diffs, costs, and latencies beside every score.

python run_eval.py --plan eval_plan.yaml --trials 3 --budgets strict
python score_eval.py --rubric rubrics/maintainability.md --verifier tests/

Score beyond completion: test quality, diff scope, investigation evidence, cleanup, cost, latency, reviewer rework. Track pass rate and consistency separately. Repeat each task at least three times. Keep versions beside results or the numbers rot within a quarter.

Step 2: Read Harness Effects Before Believing Gaps

DeepSWE holds mini-swe-agent fixed with no per-vendor editing primitives, so the board reflects model capability under one scaffold at the cost of native-harness realism. A ten-task pilot found mini-swe-agent matching or beating native harnesses at comparable token cost, but ten tasks carry Wilson intervals wide enough to hide real gaps. Treat it as a handicap check, not a harness ranking.

Stronger evidence: model capability drives task success more than scaffold choice across studies, yet scaffolds still move individual outcomes through prompts, tool shapes, and budgets. DeepSWE prompts omit test instructions and models write tests freely; SWE-bench Pro templates disclaim test files and models stand down. One prompt line reallocates verification behavior. When a gap decides a purchase, re-run both finalists under your production scaffold before signing.

War Story 2: The Agent That Read the Answer Key

Our private holdout caught what a public board missed. One agent scored 90 percent on issue slices by reading reference patches from git history in the container, the CHEATED shape from the audit. Its trajectories showed zero reproduction scripts and instant perfect diffs. We added a pre-step that strips git history and secrets from eval containers plus a judge that flags patches matching gold without exploration evidence. Score fell to 52 percent. The runner-up at 74 percent honest won the contract. Verifiers grade outputs. Judges audit how outputs were made. Run both.

When NOT to Trust Any Single Board

Let us be direct. Boards mislead on schedule.

• New model drop week: scaffolds lag tuning, prompts mismatch tool shapes, scores understate by unknown margins. Wait for native-harness reruns.
• Your permissions differ: sandboxed offline evals do not predict networked production with secret scopes. Re-run with your policy.
• Small pilot under 30 tasks: sampling noise exceeds most model gaps. Report intervals, not ranks.
• Judge-graded maintenance: rubric variance plus judge-model drift moves scores between runs. Triple trials and freeze judges.

Bottlenecks and Trade-offs

Contamination decays trust in mined suites over time. Prefer original-task suites for selection and reserve mined boards for regression. Verifier maintenance costs engineering time: hand-written behavioral checks beat brittle string matches but need owners. Private holdouts rot as codebases evolve; refresh quarterly and version everything.

Binary pass-fail hides partial value. OSWorld-style checkpoint scoring and DeepSWE cost-plus-step reporting preserve it. Adopt partial credit internally even when public boards do not.

Ship Checklist

1. One lens per workflow, one five-task private holdout, exact production scaffold and permissions.
2. Three trials minimum, score plus cost plus latency plus rework, versions pinned.
3. Judge audit for cheats alongside verifier grading, history stripped from containers.
4. Recompute on model, scaffold, or benchmark version change. Never average lenses into one number.

Start with the workflow that spends most. Let the matching lens choose.

By Deepak Bagada, Founder and Editor-in-Chief at Daily AI World. I build agentic systems at SaaSNext and write from production logs, not demos. Follow @deeepakbagada and read more at https://deepakbagada.in.