Vals AI Raises $40M: Confidential Benchmarks Beat Contamination

Vals AI raised a $40M Series A led by Andreessen Horowitz, reported Sep 19 2026, to build confidential benchmarks across law, finance, coding, cybersecurity and biosecurity. Test materials stay hidden so models cannot train against them.

• Seed backing from 8VC and Bloomberg Beta with 8x revenue growth and headcount from 8 to 25
• Confidential tests target the core tension: benchmarks can be auditable or uncontaminated, rarely both
• I run held-out internal evals with canary strings that catch leakage before scores rot

Benchmarks have a cheating problem and everyone knows it. Public test sets leak into training data, scores inflate, and procurement teams buy fiction. Vals AI sells the opposite: real professional tasks with hidden materials, confidential by design, across five domains where wrong answers cost money or safety. A $40M Series A says investors believe hidden tests are a business. I agree, and I run my own held-out harness for the same reason. Here is the news plus the method.

What the $40M round signals about evals

Per TechCrunch reporting, the Series A follows a seed from 8VC and Bloomberg Beta, with no valuation disclosed. Co-founder Rayan Krishnan, 25, previously interned at Palantir and worked at Microsoft plus Stanford AI lab. His analogy compares paying for these benchmarks to students paying the College Board for the SAT: his framing, and an honest one. Tests you can preview are tests you can game. Revenue reportedly grew 8x year over year off an undisclosed base, so read momentum, not size. Headcount sits at 25, up from 8 in January. Small team, narrow product, real revenue signal.

The structural point matters more than the money. A benchmark can be auditable or uncontaminated, but hardly both at once. Public sets invite training-data leakage. Private sets demand trust in the vendor. Vals AI picks secrecy with professional-grade tasks in law, finance, coding, cybersecurity and biosecurity. Enterprises in those domains pay for scores they cannot reproduce because reproduction would destroy the score. That tradeoff now has $40M of validation. My take: outsource domain breadth to vendors like this, keep a private held-out core in-house that nobody outside the eval team can see.

This extends our measurement coverage with the integrity layer. Our LLM-as-judge accuracy benchmarks judge outputs. Our reasoning effort tiers study prices passes. Both assume clean tests. Contamination voids both. Audit the test before trusting the score.

graph TD
  A[New model version] --> B[Public sanity suite]
  B --> C[Held-out private suite]
  C --> D{Canary strings present in outputs?}
  D -->|yes| E[Leak: quarantine + investigate]
  D -->|no| F[Vendor confidential suite]
  F --> G[Ship or roll back]

Step 1: Build a held-out core nobody trains on

My private suite holds 120 tasks across our bug-fix, contract-review and runbook patterns. Storage is a separate repo with four readers. No training jobs mount it. No agent logs echo it. Prompts reference task IDs, never content, so stray logs cannot reconstruct items. Rotation runs quarterly: 30 tasks retire into an exhausted archive, 30 fresh tasks enter from recent incidents. Stale secrets teach models history. Fresh incidents teach current work.

First war story. Our public golden set lived in the main repo for convenience. Six months later a fine-tune job ingested the whole repo including eval answers. Scores jumped 9 points overnight and the team celebrated until I traced the data lineage. Retraining without the evals dropped scores back. Nine phantom points bought zero capability. The held-out core now lives behind separate credentials with access logs reviewed monthly. Convenience is the leading cause of contamination. Pay the friction.

Step 2: Canary strings that catch leakage fast

Plant unique nonsense tokens inside held-out items, following the bigbench canary practice. If a model ever emits your canary, its training saw your test. I grep every eval output plus sampled production logs weekly. One hit in fourteen months traced to a contractor pasting eval content into a vendor demo. Caught in days, contained by revoking the items. Canaries cost one line per task. They are the cheapest alarm I own.

File: requirements.txt

httpx==0.28.1
pydantic==2.8.0
rich==13.9.4

File: config.py

from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", extra="allow")
    api_base: str = Field(alias="EVAL_API_BASE")
    api_key: str = Field(alias="EVAL_API_KEY")
    canary: str = "HELDOUT-CANARY-9f3kq7"
settings = Settings()

File: heldout.py

import httpx
from config import settings

TASKS = open("heldout.txt").read().split("
---
")

def run_suite(model):
    passes, leaks = 0, 0
    with httpx.Client(timeout=300) as c:
        for t in TASKS:
            r = c.post(f"{settings.api_base}/chat/completions", json={"model": model, "messages": [{"role": "user", "content": t}]}, headers={"Authorization": f"Bearer {settings.api_key}"}, timeout=300)
            text = r.json().get("choices", [{}])[0].get("message", {}).get("content", "")
            passes += int("pass" in text)
            leaks += int(settings.canary in text)
    return {"model": model, "pass": round(passes / len(TASKS), 3), "canary_hits": leaks}

if __name__ == "__main__":
    print(run_suite("candidate-model"))

uv venv --python 3.12 && source .venv/bin/activate
uv pip install -r requirements.txt
python heldout.py

Second war story. A vendor benchmark report showed our shortlist model leading by 6 points. My held-out suite showed it trailing by 2. Same week, same tasks family. The vendor suite overlapped public data the model had clearly digested. We bought the runner-up and it outperformed in production for six straight months. Vendor scores sell models. Private scores buy outcomes. Keep both, trust yours.

Cost framing keeps eval budgets alive. My held-out runs cost $11 weekly in model calls plus 3 hours of curation monthly. One prevented misbuy saved an estimated $40k in migration waste. Our background-thread tracing pipeline attributes eval spend separately so finance sees insurance, not overhead. Our test-time compute routing guide applies the same spend-where-it-pays logic to eval tokens.

Step 3: Buy third-party breadth without surrendering judgment

Confidential vendor suites complement private cores. Vendors cover five domains with professional tasks no single team replicates. I commission vendor runs quarterly for shortlists, then confirm winners on my held-out core before signing. Disagreements favor my suite by default with one exception: security and biosecurity domains where vendor item quality exceeds anything I can build. Respect expertise gradients. Verify everything else.

When NOT to build secret evals

Let's be clear. Secrecy has overhead.

Skip held-out infrastructure for prototypes finding product-market fit. Public suites plus user feedback steer faster than private benchmarks at that stage. Build secrecy when buying decisions exceed $50k annually or failures reach customers.

Skip vendor confidential suites for narrow single-domain products where your team writes better tests than any generalist vendor. In-house expertise beats purchased breadth inside your own kitchen. Buy breadth, build depth.

Production bottlenecks I hit: eval repos attract curious readers so gate with separate credentials; canary grep across log volumes needs indexed search past 30 days; quarterly rotation meetings slip without a calendar owner; vendor contracts restrict publishing comparisons so negotiate disclosure upfront. Ordinary fixes. Required fixes.

Bottom line: public scores start conversations, private held-outs close decisions, and canaries keep everyone honest.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World. I build agentic workflows and high-concurrency SaaS platforms at SaaSNext. Follow my benchmarks on <a href="https://x.com/deeepakbagada">X @deeepakbagada and <a href="https://deepakbagada.in">deepakbagada.in.