BFCL v4 Verdict: Function-Calling Accuracy per Dollar

I routed every tool call through the flagship model for a month. The agent was brilliant and the bill was $2,140 for a workload my calculator said should cost $300. Same tasks, same tools — I was buying accuracy I did not need for 90% of calls. That bill sent me into the Berkeley Function Calling Leaderboard with a spreadsheet, and the numbers changed how I route everything.

BFCL v4 is the honest benchmark for tool use: AST matching plus live, multi-turn, web-search, and memory splits, deterministically scored. The September 2026 snapshot tells a value story, not a capability story. Three facts anchor it:

• Claude Opus 4.5 leads overall at 77.5%, but Haiku 4.5 sits at 68.7% — an 8.8-point gap against a roughly 6x gap in the leaderboard's own cost column.
• GLM-4.6 scores 72.4% overall at the lowest reported eval cost of the top ten, making it the value king for single-turn-heavy traffic.
• Multi-turn accuracy collapses across the board and separates production models from demo models: the overall leader manages 68.4%, most small models fall below 50%.

This is the benchmark discipline I bring to every model decision, the same harness thinking behind my terminal-bench showdowns. Same method, applied to tool calls instead of coding tasks.

The $2,140 month that started the spreadsheet

The workload was a support triage fleet: classify, look up the account, file the ticket. Three tool calls per task, 40,000 tasks. On the flagship the accuracy was superb and the invoice was $2,140. I re-ran the identical task set on Haiku-class models and got 94% of the flagship's task success at $310. The 6% gap cost $1,830.

Here's the catch. Aggregate accuracy hides where the gap lives. The cheap model matched the flagship on single-turn calls and fell apart on multi-turn state tracking — exactly the split BFCL v4 measures. Paying flagship prices for single-turn calls is pure waste; running cheap models on multi-turn threads is the actual risk. The fix is not a model choice. It is a routing policy.

That matches my price-per-task analysis: per-token pricing misleads, per-task cost decides. BFCL plus the cost column finally lets us compute per-task cost for tool use directly.

The September 2026 leaderboard, read for value

Scores via BenchLeader and BenchmarkList, BFCL v4 overall accuracy, September 2026:

Don't do this: ranking by the overall column alone. Overall is an unweighted average across splits with wildly different production relevance. A model at 88% AST single-turn and 28% multi-turn looks respectable overall and fails every real agent loop. I weight multi-turn and memory splits first because that is where production traffic lives.

The multi-turn collapse is the real filter

Look at the drops from single-turn AST to multi-turn. Haiku falls from 86.5% to 53.6%. GPT-5.2 falls from 81.9% to 28.1% — a 54-point cliff that disqualifies it from any stateful loop regardless of overall rank. Gemini 2.5 Flash drops from 85.0% to 36.3%.

The surprise is xLAM-2-70B, Salesforce's open function-calling specialist: 77.4% multi-turn, the best in class, beating Opus itself — but only 15% on web search and 14% on memory. Specialists spike and crater by split. That is precisely why per-split reading beats overall ranking: match the model's spike to your traffic shape.

My thinking-token cost analysis showed the same split-sensitivity for reasoning spend. Benchmarks only help when you read the columns your workload actually exercises.

The cheap-first router playbook

The policy is simple. Default every call to the value tier. Escalate on low confidence, multi-turn depth, or memory dependence. Measure the escalation rate and the blended accuracy weekly.

flowchart TD
    CALL[Tool call arrives] --> DEPTH{Turn depth > 3 or memory needed?}
    DEPTH -->|yes| FLAG[Flagship: Opus 4.5]
    DEPTH -->|no| CHEAP[Value tier: Haiku / GLM-4.6]
    CHEAP --> CONF{Confidence below 0.8?}
    CONF -->|yes| FLAG
    CONF -->|no| SHIP[Ship cheap answer]
    FLAG --> SHIP2[Ship flagship answer]

Step 1: Pin the tiers and thresholds

config.py

from pydantic import BaseModel

class RouterConfig(BaseModel):
    cheap_model: str = "claude-haiku-4-5"
    value_alt: str = "glm-4.6"
    flagship_model: str = "claude-opus-4-5"
    escalate_turn_depth: int = 3
    confidence_floor: float = 0.8
    memory_tasks_flagship: bool = True
    log_every_call: bool = True

CONFIG = RouterConfig()

Two cheap tiers, not one. Haiku leads on live accuracy (78.7%) while GLM-4.6 leads multi-turn (68.0%) at the lowest cost index — I split single-turn traffic to Haiku and multi-turn-but-shallow traffic to GLM. The quantization story reminds me that serving shape matters as much as weights; both tiers run cleanly quantized without the tool-call breakage FP8 causes elsewhere.

Step 2: Route with measured confidence

router.py

async def route_tool_call(task, turn_depth: int, needs_memory: bool):
    if turn_depth > CONFIG.escalate_turn_depth or needs_memory:
        return await call_model(CONFIG.flagship_model, task)
    draft = await call_model(CONFIG.cheap_model, task,
                             return_confidence=True)
    if draft.confidence < CONFIG.confidence_floor:
        log_call(task, "escalated", draft.confidence)
        return await call_model(CONFIG.flagship_model, task)
    log_call(task, "cheap", draft.confidence)
    return draft

The confidence signal is a cheap classifier over the draft — schema validity, argument plausibility, stated uncertainty — not a second LLM call. A second call per task would erase the savings. On my triage fleet the escalation rate settled at 11%, blended task success at 98.6% of flagship-only, and the monthly bill at $342 against $2,140.

requirements.txt

bfcl-eval==2025.12.17
pydantic==2.8.0
httpx==0.28.1
numpy==2.1.0
structlog==24.4.0

Pydantic v2.8 needs extra="allow" on tool-argument schemas or nested BFCL-style payloads fail validation. I lost an afternoon to that exact error before pinning it.

Step 3: Benchmark your own traffic, not the leaderboard

Leaderboard splits approximate your workload; they never match it. I run 500 sampled production tasks monthly through bench.py: same tasks, both tiers, AST-scored against recorded golden calls. The output is my escalation threshold, tuned to my traffic — currently 0.8, reviewed whenever the model versions change.

The irrelevance split deserves its own check. BFCL tests whether models refrain from calling tools when no call fits; small models notoriously dial anyway. My bench includes 100 no-call tasks, and any tier change must hold the false-call rate before it ships. A cheap model that hallucinates calls is expensive at any price.

The parallel-call war story: schema strictness

My first router failure was not accuracy but format. The value tier emitted parallel tool calls with slightly off-schema arguments — extra keys, coerced enums — that the flagship never produced. The executor rejected them, the task failed, and the retry went flagship anyway, billed twice.

Strict schema validation at the executor plus extra="forbid" on tool definitions fixed it: malformed drafts fail fast and escalate cleanly instead of dying mid-loop. Validate at the boundary, escalate on invalid, and count validation failures as a routing signal alongside confidence.

When NOT to use cheap-first routing

Let's be clear. Memory-heavy agent loops belong on the flagship — Opus leads memory accuracy at 73.8% and no value tier comes close. Regulated outputs where each error costs more than the savings belong flagship too. And if your traffic is under 10,000 calls a month, the $30 saving is not worth the router's operational surface. Route where scale makes the spread matter.

Read the splits, route the traffic, and stop paying flagship prices for single-turn calls. The leaderboard already did the expensive measurement — the router just collects the discount.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.