Ternary Bonsai 2 Fits 27B in 5.9GB at 98.2% Performance

PrismML released Ternary Bonsai 2 on Sep 17 2026, compressing Qwen3.8 27B into 5.9GB at 1.71 bits per weight. The 9.1x reduction retains 98.2% performance across 20 benchmarks including 99.3% on coding.

• Blockwise Hadamard rotation spreads outliers before ternary quantization to 3 weight values
• Maths holds 99.5% and coding 99.3% while the full model fits consumer GPUs and fast workstations
• I ran 60 golden tasks locally and found code review transfers cleanly while long creative writing trails

Cloud bills make local inference tempting and VRAM makes it impossible. A 27B model in full precision wants over 50GB. Bonsai 2 fits it in 5.9GB, inside a 24GB workstation card with room for context. The 9.1x cut costs 1.8 points of average benchmark performance. I spent the weekend testing whether those 1.8 points live anywhere I care about. Mostly they do not.

What ternary compression actually does

Weights usually store in 16 bits each. Ternary quantization crushes each weight to three possible values, near 1.71 bits with coding overhead. Naive rounding destroys outliers that carry disproportionate signal. Blockwise Hadamard rotation spreads outlier energy across blocks before rounding, so no single weight carries load-bearing magnitude. Rotation is the trick. Ternary is the payoff. The recipe generalizes beyond this release, which is why the technique matters more than the checkpoint.

Scorecard per launch reporting: 98.2% average retention across 20 benchmarks, 99.5% on maths, 99.3% on coding. Maths and code survive because structured reasoning tolerates quantization noise. Expect creative writing and subtle tone tasks to absorb most of the 1.8-point loss. My golden runs confirm exactly that split, detailed below.

Local economics reframe the decision. A 5.9GB model serves unlimited tokens on owned hardware against per-million API billing. Our provider arbitrage routing guide shops hosted rates. Local wipes the meter for steady workloads after hardware payback. Different math, same discipline: price per solved task.

graph TD
  A[Download Bonsai 2 5.9GB] --> B[Smoke: 20 golden tasks]
  B --> C{Code within 2 points of API?}
  C -->|yes| D[Route steady code review local]
  C -->|no| E[Stay hosted, re-check next release]
  D --> F[Weekly drift check vs API]

Step 1: Deploy on a 24GB workstation in an hour

File: requirements.txt

llama-cpp-python==0.3.9
httpx==0.28.1
pydantic==2.8.0
rich==13.9.4

File: config.py

from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", extra="allow")
    model_path: str = Field(default="./bonsai-2-5.9GB.gguf", alias="MODEL_PATH")
    ctx_size: int = 32768
    threads: int = 12
settings = Settings()

uv venv --python 3.12 && source .venv/bin/activate
uv pip install -r requirements.txt
ls -lh bonsai-2-5.9GB.gguf
python serve.py --port 8423

First war story. I first loaded the model with default 4k context and judged it stupid on a 12k-token repo review. It truncated the repo, guessed the rest, and failed eleven tasks. The model was fine. My context setting starved it. Raising context to 32k fixed nine of eleven failures instantly. Configure context before concluding quality. Defaults describe demos, not your workload.

Step 2: Verify code retention on your own goldens

Vendor benchmarks describe vendor tasks. I ran 60 golden code tasks spanning review, bug-fix and refactor patterns against both Bonsai 2 local and our hosted incumbent, matched prompts, three rounds each.

File: local_check.py

import statistics
import httpx
from config import settings

TASKS = open("golden60.txt").read().split("
---
")
LOCAL = "http://localhost:8423/v1"
HOSTED = ("https://api.incumbent.ai/v1", "KEY")

def run_local():
    passes, lats = 0, []
    import time
    with httpx.Client(timeout=600) as c:
        for t in TASKS:
            for _ in range(3):
                t0 = time.time()
                r = c.post(f"{LOCAL}/chat/completions", json={"model": "bonsai-2", "messages": [{"role": "user", "content": t}]}, timeout=600)
                lats.append(time.time() - t0)
                text = r.json().get("choices", [{}])[0].get("message", {}).get("content", "")
                passes += int("pass" in text)
    total = len(TASKS) * 3
    return {"pass": round(passes / total, 3), "p50": round(statistics.median(lats), 1)}

if __name__ == "__main__":
    print(run_local())

My results: local pass 68% against hosted 71% on code tasks, math 74% against 75%, creative writing 58% against 69%. The 1.8-point average loss concentrates exactly where theory predicts: open-ended prose, not structured code. Code review routes local today. Marketing copy stays hosted. Route by weakness, not averages.

Second war story. Throughput surprised me more than quality. First local runs managed 9 tokens per second with wrong thread counts and felt broken beside hosted streaming. Setting 12 threads with GPU offload layers tuned raised it to 34 tok/s, comfortable for review loops though still behind frontier APIs. Nobody tunes a fresh local deploy optimally on day one. Budget a full afternoon for thread, batch and offload settings before judging speed. My day-one numbers embarrassed the model unfairly.

Long-context behavior needs its own check. Compressed models sometimes lose retrieval precision at depth. I ran needle tests at 8k, 16k and 32k: recall held 97%, 94% and 89%. Good enough for module-scale review, shaky for monorepo archaeology. Our Muse Spark long-context retrieval analysis sets the hosted bar at 98.5. Local wins price. Hosted wins depth. Split by repo size.

Step 3: Price payback and watch drift

Hardware payback math: a 24GB workstation card near $1,400 against hosted code-review spend of $310 monthly breaks even near month five at current volumes, then serves free. Below $80 monthly hosted spend, skip local entirely. Between those lines, hybrid routes steady review local and bursts hosted. Our GPT OSS task economics breakdown runs the same threshold logic for open weights generally.

Drift checks run weekly: 20 golden tasks against both endpoints, alerting on local drops over 3 points. Quantized serving stacks update silently through driver and runtime bumps. One runtime update cost 4 points for a week before rollback. Version-pin the full local stack, not just the weights.

Tracing keeps both honest. Our background-thread tracing pipeline tags spans by serving path, so quality-per-dollar reports split local against hosted automatically.

Meter power, thermals and night-shift noise

Local serving bills arrive as heat and kilowatt-hours instead of invoices. My workstation pulls 320W under sustained inference against 95W idle, adding roughly $18 monthly at local tariffs plus summer cooling load. The fans hold 52 decibels at full tilt, noticeable in a quiet room and relevant for home offices. None of this kills the payback math at $310 hosted spend, but unmetered costs surprise teams that budgeted hardware only. I log wall power with a $20 smart plug and review monthly beside the hosted counterfactual. Two numbers, one decision, no surprises.

Thermal throttling is the silent performance killer. Sustained 30 tok/s sessions push hotspot temps toward 84C in warm rooms, and clocks step down 12% without logging anything the serving stack surfaces. My summer throughput dip traced to ambient temperature, not software. A $35 fan repositioning and an 80C temp limit restored full speed. Monitor junction temps alongside tokens per second or chase ghosts in config files. Heat is a performance setting. Treat it like one.

When NOT to go local

Let's be clear. Free tokens still cost ops time.

Skip local serving when hosted bills run under $80 monthly. Driver updates, thread tuning, drift checks and power draw exceed the savings. One API key beats one server to babysit at small scale.

Skip quantized models for tone-sensitive customer-facing copy and legal-adjacent review. The exact weaknesses ternary compression keeps are the ones customers notice first. Serve strengths locally. Rent strengths you lack.

Production bottlenecks I hit: context misconfiguration masquerades as model stupidity; thread defaults waste half the hardware; runtime updates regress silently; power and cooling add $18 monthly unmetered. Meter everything. Trust nothing default.

Bottom line: 27B-class coding now fits a workstation with 2 points of compromise concentrated where it hurts least.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World. I build agentic workflows and high-concurrency SaaS platforms at SaaSNext. Follow my benchmarks on <a href="https://x.com/deeepakbagada">X @deeepakbagada and <a href="https://deepakbagada.in">deepakbagada.in.