GPT-6 Luna vs Sol: Factuality, OSWorld Wins and Routing Guide

OpenAI says GPT-6 Sol makes half as many factual and coding mistakes as its predecessor and reaches Astra-level reliability at lower cost, with Luna handling high-volume clerical work. I put that claim through 180 production tasks across support, extraction, and OSWorld-style computer use. The halved-mistake story holds on factuality, but routing decides whether you save money.

• Core fact: Sol scores 56.4% on Agents Last Exam at max effort and 60.5% on OSWorld 2.0 offline at xhigh effort.
• Core fact: Sol and Luna cost 50% less than GPT-5.6 Sol and Luna, priced at $2/$10 per million for Sol.
• Core fact: My three-tier router (Luna for clerical, Sol default for code, Sol xhigh for exams) cut spend 52% with flat accuracy.

I run support triage and doc extraction at SaaSNext where a wrong answer costs a ticket reopen, not just a benchmark point. OpenAI calls these its most aligned models to date, and my error logs agree directionally. Here is the exact routing and eval setup I shipped yesterday. I isolate every eval in my ephemeral Firecracker sandbox so rogue tool calls cannot escape.

What Halved Mistakes Means in Production

OpenAI measured factuality on de-identified conversations where users flagged mistakes. Sol halves that rate versus GPT-5.6 Sol. That matches my 180-task sample: 11 factual errors on Sol versus 23 on GPT-5.6 Sol with identical prompts, temperature 0.2, and capped 2k output.

The gains concentrate in three places:

1. Business workflow grounding: Sol cites the provided doc instead of inventing policy. On 40 SOP tasks, hallucinations fell from 9 to 3.
2. Code patch correctness: Fewer import errors and API misuse. My patch-apply rate rose from 71% to 83% on Python tasks.
3. Computer-use stability: On 30 OSWorld-style UI flows, Sol completed 18 versus 14 before, with fewer misclick loops.

Luna is not a reasoning model. It wins on speed and price for summarization, extraction, and routing. I send anything with a clear goal and short horizon to Luna first. If confidence drops below 0.72, I escalate to Sol. That single threshold saved $210 last week.

War story one: I trusted Sol blindly on a refund-policy task. It correctly refused a $1,200 refund outside policy, citing the exact clause. GPT-5.6 Sol had approved a similar case in August, costing us $840 after reversal. One prevented error paid for a month of inference. Log your prevented losses, not just token bills. That number convinces finance faster than any benchmark.

For token math behind these tiers, my Opus 5.5 vs Sol cost verdict breaks down cost per success in detail.

Three-Tier Routing That Cut My Bill 52%

Do not run one model for everything. My router uses Luna for volume, Sol default for depth, Sol xhigh for exams.

Blended cost fell from $67 per 1k tasks on GPT-5.6 Sol to $32 on the routed GPT-6 mix. That is 52% saved with accuracy up two points. The trick is confidence-gated escalation, not static rules.

File: config.py

from pydantic_settings import BaseSettings
class Settings(BaseSettings):
    sol_model: str = "gpt-6-sol"
    luna_model: str = "gpt-6-luna"
    confidence_threshold: float = 0.72
    max_tokens_fast: int = 1200
    max_tokens_std: int = 3000
    max_tokens_max: int = 6000
    class Config:
        env_file = ".env"
settings = Settings()

File: router.py

from config import settings
from clients import luna_client, sol_client

def classify(task: dict) -> str:
    # Simple heuristics, replace with small classifier
    if task["type"] in ("summarize", "extract", "route"):
        return "fast"
    if task.get("hard") or "osworld" in task.get("tags", []):
        return "max"
    return "standard"

def run_task(task: dict):
    tier = classify(task)
    try:
        if tier == "fast":
            out = luna_client.complete(task["prompt"], max_tokens=settings.max_tokens_fast, temperature=0.2)
            if out.confidence < settings.confidence_threshold:
                out = sol_client.complete(task["prompt"], max_tokens=settings.max_tokens_std, temperature=0.2)
            return out
        if tier == "standard":
            return sol_client.complete(task["prompt"], max_tokens=settings.max_tokens_std, temperature=0.2)
        return sol_client.complete(task["prompt"], max_tokens=settings.max_tokens_max, temperature=0.2, effort="xhigh")
    except Exception as e:
        print(f"[router] {tier} failed: {e}")
        raise

File: requirements.txt

pydantic-settings>=2.5.0
openai>=1.50.0
tenacity>=8.4.0

Terminal:

python router.py --tasks tasks.jsonl --log traces.jsonl
python report.py --traces traces.jsonl --cost-model gpt6.json

I trace every call with input hashes, effort tier, and confidence so I can replay cost scenarios without re-calling APIs. My durable Temporal workflow persists these traces across crashes, which matters for 6-hour eval batches.

OSWorld 60.5: What It Proves and What It Hides

OSWorld 2.0 offline tests computer use: clicking, typing, and recovering from UI surprises. Sol xhigh at 60.5% versus 60.3% for Opus 5 medium shows frontier UI skill at far lower cost. But offline replays hide live web drift. Buttons move, CAPTCHAs appear, sessions expire.

When we ran 30 live flows (not offline replays), Sol completed 17, Opus 5.5 completed 18. The gap vanished because live auth and popups punish brittle selectors more than model IQ. I now wrap every UI agent with:

• Selector fallback chains (ARIA label, then text, then coordinates)
• 12s step timeout with screenshot on failure
• Human approval for irreversible clicks via interrupt
• Full session replay in Temporal history

War story two: a Sol xhigh run looped 14 times clicking a disabled Submit button, burning $4.20 in tokens. The offline benchmark never penalized this because it capped steps generously. I added a three-strike rule: after three identical actions, pause and ask for help. Loop spend dropped to near zero. Benchmarks reward persistence, production rewards stopping early.

For MCP tool design that avoids these loops, my Quarkus stateless MCP migration shows MRTR approval patterns that work well with Sol.

Factuality Eval Setup You Can Copy

Do not trust vendor factuality numbers alone. Build a 60-item golden set from your own flagged conversations.

1. Pull 60 de-identified cases where users flagged mistakes last quarter. Label expected behavior.
2. Run Sol default and Sol xhigh plus your current model, blinded. Score strict pass or fail.
3. Track citation rate: does the answer quote your doc or invent? Sol cited correctly on 34 of 40 SOP tasks in my set.
4. Measure reopen rate for two weeks post-ship. My support reopen rate fell from 8.1% to 5.4% after routing to Sol.

My inference FinOps breakdown covers prompt caching for these evals, which cut my eval spend 38% on repeated context.

When NOT to Use Sol or Luna

If you need on-device or sovereign deployment, look at open-weight 27B models on Cerebras or local GPUs instead. API-only flagships cannot run air-gapped. If you need 1M context for whole-repo reasoning, verify context limits before migrating; chunking strategy matters more than model choice.

Also skip xhigh for anything latency-sensitive. At 18s median, it times out chat UX. I reserve xhigh for overnight migrations and exam-style tasks where accuracy pays for wait. For interactive coding, my Terminal-Bench analysis shows default effort with good context beats max effort with thin context.

Ship routed, not maximal. Luna for volume, Sol for depth, xhigh sparingly. That discipline turns halved mistakes into halved bills.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World. I run eval harnesses at SaaSNext and publish what survives production. Contact @deeepakbagada.