Embedding Showdown for Agents: BGE vs E5 vs Nomic at 12ms

Agent retrieval lives or dies on embedding choice. I benchmarked BGE-M3, E5-large-v2, and Nomic-embed-text-v1.5 on 2,400 real SaaSNext support docs with 300 judged queries. Nomic hit 91% recall@5 at 12.4ms per query on local GPUs. E5-large matched recall but needed 2.5x the RAM. BGE-M3 won multilingual, lost English-only speed.

• Nomic 137M: 91% recall@5, 12.4ms, 274MB RAM, free self-hosted
• E5-large 335M: 92% recall@5, 18.7ms, 1.3GB RAM, free self-hosted
• BGE-M3 568M: 89% recall@5, 30.9ms, 2.2GB RAM, best of 100+ languages

If your agents answer from English docs, stop paying embedding APIs. Here's the full harness.

Why academic leaderboards mislead agent builders

MTEB 2026 ranks Qwen3-Embedding-8B first at 70.6, Gemini Embedding at 68.3, Cohere v4 at 65.2. All API models. All measured on clean academic sets. None of that predicts your support-docs recall.

A July 2026 benchmarking study put numbers to the gap: on FiQA, NFCorpus, SciFact, and TREC-COVID, Google's GE2 averaged 0.638 nDCG while E5-large hit 0.538 and BGE-M3 0.437. But GE2 cost $0.025 per 1M tokens at 231ms median latency with 575ms p95. E5-large answered in 31ms for free. For an agent doing 8 retrieval calls per task, GE2 adds 1.8 seconds and a meter running.

Worse, the same study showed retrieval-optimized models beating similarity-optimized ones on retrieval tasks while losing on STS. Picking by overall MTEB rank is picking the wrong objective. Agents need retrieval recall, not sentence-similarity correlation.

A June 2026 multilingual test confirmed it: Mistral Embed, absent from MTEB leaderboards, beat both BGE-M3 and Qwen3-8B on French and Japanese Wikipedia retrieval. Domain beats leaderboard. Every time.

This is why per-step reliability analysis matters: retrieval misses compound across steps, so one recall point is worth more than any reranker trick.

Test setup: 2,400 docs, 300 judged queries

Corpus: 2,400 SaaSNext help articles, API references, and changelogs, chunked at 512 tokens with 64-token overlap. Queries: 300 real support questions with judged gold passages. Metrics: recall@1, recall@5, MRR, median and p95 latency, RAM, and cost per 1M queries.

Hardware: Mac Studio M2 Ultra for local models, pgvector on Postgres 16 for storage, 768-dim for Nomic, 1024-dim for E5 and BGE. API baseline through OpenRouter for reference.

Chunking mattered more than I expected. Moving from 256 to 512 tokens lifted recall@5 by 6 points across all models. Overlap below 32 tokens dropped answer completeness 11%. I now treat chunk config as a first-class hyperparameter, versioned alongside the model name.

Step 1: Reproducible eval harness

requirements.txt:

sentence-transformers==3.2.1
pgvector==0.3.6
psycopg==3.2.4
numpy==1.26.4
pytest==8.3.4
tqdm==4.66.5

config.py:

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    postgres_dsn: str = "postgresql://agent:secret@localhost:5432/retrieval"
    chunk_tokens: int = 512
    chunk_overlap: int = 64
    top_k: int = 5
    models: list[str] = [
        "nomic-ai/nomic-embed-text-v1.5",
        "intfloat/e5-large-v2",
        "BAAI/bge-m3",
    ]

    class Config:
        env_prefix = "EMBED_BENCH_"

settings = Settings()

eval.py:

import time
import numpy as np
from sentence_transformers import SentenceTransformer
from config import settings

def recall_at_k(ranked: list[str], gold: set[str], k: int) -> float:
    return len(set(ranked[:k]) & gold) / max(1, len(gold))

def benchmark(model_name: str, queries: list[dict], corpus_vecs: np.ndarray):
    model = SentenceTransformer(model_name, trust_remote_code=True)
    lat, r1, r5 = [], [], []
    for q in queries:
        prefix = "query: " if "e5" in model_name.lower() else ""
        t0 = time.perf_counter()
        qv = model.encode(prefix + q["text"], normalize_embeddings=True)
        sims = corpus_vecs @ qv
        top = np.argsort(-sims)[:10].tolist()
        lat.append((time.perf_counter() - t0) * 1000)
        r1.append(recall_at_k(top, set(q["gold"]), 1))
        r5.append(recall_at_k(top, set(q["gold"]), 5))
    return {
        "model": model_name,
        "recall@1": round(float(np.mean(r1)), 3),
        "recall@5": round(float(np.mean(r5)), 3),
        "p50_ms": round(float(np.median(lat)), 1),
        "p95_ms": round(float(np.percentile(lat, 95)), 1),
    }

Note the E5 query prefix. E5 models require query: and passage: prefixes. Forget them and recall drops 9 points silently. No error. Just worse answers. I lost a day to that before diffing the training README. BGE-M3 and Nomic need no prefixes but Nomic wants normalize_embeddings=True for cosine search.

uv pip install -r requirements.txt
python eval.py --corpus ./docs --queries ./judged.jsonl

Results: the table that changed our stack

E5-large wins raw recall by one point. Nomic wins everything else: 34% faster, 5x smaller, identical MRR within noise. At 40k agent queries per day, the API baseline costs $528 monthly plus 231ms per call. Nomic costs one shared GPU we already own.

BGE-M3 earns its keep on multilingual corpora. On our 400 Japanese and French docs, it beat Nomic by 7 recall points. English-only teams should skip it. Global teams should default to it. That matches the published MIRACL results where BGE-M3 led multilingual by wide margins.

We pair embeddings with prompt caching on the generation side: cheap retrieval plus cached reasoning cut per-task cost 68%.

Production war story 1: the silent dimension mismatch

First breakage. We upgraded Nomic Matryoshka truncation from 768 to 512 dims to halve pgvector storage. Recall held. Then half our queries returned garbage. Root cause: old 768-dim rows sat beside new 512-dim rows in the same table. pgvector computed distances across mismatched dims without error. No exception. Just wrong neighbors.

Fix: namespace indexes by model plus dimension (docs_nomic_512, docs_e5_1024), add a startup assertion comparing stored dim to model dim, and backfill on mismatch. Twenty lines. We now treat embedding dim as a schema version, not a setting.

Second breakage: unnormalized E5 vectors. One code path encoded passages without normalization while queries were normalized. Cosine scores compressed into a 0.02 band. Top-k became random. Our nightly eval caught it: recall@5 fell from 0.92 to 0.71 overnight. The fix was one flag. The lesson was the eval. Run it nightly or fly blind.

Our overnight bill taught the same lesson as speculative decoding at batch: measure on your load shape, not the vendor's chart.

When NOT to use local embeddings

Skip self-hosting when you have under 10k queries per day and no GPU. API simplicity beats $0 marginal cost. One engineer maintaining CUDA drivers for 200 queries a day is negative ROI.

Skip 8B embedding giants unless you need their multilingual ceiling. Qwen3-Embedding-8B leads MTEB but ingests at 4.5s per chunk versus 0.55s for BGE-M3. For bulk ingestion of millions of docs, that gap is weeks.

Skip dense-only retrieval for exact-match workloads. Order IDs, error codes, and SKU lookups need BM25 or hybrid. Dense models paraphrase; keyword search does not. We run hybrid with 0.7 dense plus 0.3 BM25 weights, and exact-ID queries route to BM25 only.

Use local Nomic or E5 when agents retrieve over English technical docs at volume. Use BGE-M3 when languages multiply. Use APIs when volume is trivial or multilingual quality must be maximal.

Ship checklist

1. Version model plus dimension plus chunk config as one unit.
2. Normalize all vectors. Assert it at index and query time.
3. Add E5 prefixes. Test without them to prove the gap is real.
4. Run nightly judged evals. Alert on 2-point recall drops.
5. Route exact-match queries to BM25, semantic queries to dense.

Bottom line: Nomic at 12ms does the job for most agent teams. Spend the savings on judged evals, not bigger vectors.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.