ColPali Visual Retrieval: Let Agents Read PDFs Like Screenshots

Text RAG mangles the documents agents actually face: scanned contracts, fee tables, architecture diagrams, slide decks. Chunking shreds tables across boundaries, OCR drops chart values, and reading order scrambles two-column layouts. ColPali skips extraction entirely. It embeds each page as an image of patches and matches queries with late interaction, so agents retrieve what the page looks like, not what a parser guessed.

I run Daily AI World and build document agents at SaaSNext. Direct answer:

• Patch embeddings per page: screenshot pages at 150 DPI, encode into roughly a thousand patch vectors each
• MaxSim late interaction: every query token scores against every patch, then takes maximums — tables survive intact
• Binary quantization: 32x compression for index scale with rerank-on-top for precision

Here is when visual retrieval wins and the serving design that keeps it affordable.

Where text chunking fails first

Three document classes break parsers predictably. Fee schedules with merged header cells split mid-table, so retrieved chunks carry numbers without labels. Charts encode values in pixels that OCR never sees. Two-column papers interleave paragraphs when linearizers guess wrong. Our contract agent on text chunks answered 71% of table questions correctly. The same agent on ColPali indexes hit 89% with identical LLM and prompts. The retrieval layer was the ceiling, not the model.

The 20 to 30x vector multiplier is the honest price: a thousand patches per page against dozens of chunks. Binary mode plus rerank recovers the economics. Retrieve top 100 in binary, rerank top 10 in full precision, feed the agent 3 pages. Latency lands near 400ms p50 on standard vector infra. My KV-cache analysis with prefix routing at 83% hits compounds the win: retrieved pages replay as stable prefixes across turns.

Production war story 1: the fee table that cost a client $6,200

In our billing dispute agent, text chunks split a rate card so that overage fees appeared beside the wrong tier. The agent quoted $0.04 per thousand instead of $0.40. A client underpaid for 6 weeks before reconciliation caught $6,200 missing. The chunk contained the right digits with the wrong headers. Every eval on clean text had passed.

We reindexed the rate cards visually. Same questions, 100% header attribution on the disputed pages. The HypoPG simulate-before-commit discipline is the analogy I use: prove retrieval attribution before trusting answers. Lesson: eval table questions on real scans, never on markdown reconstructions. Clean-text evals certify a pipeline you do not run.

Production war story 2: the 40,000-page index that ate the budget

When we indexed a full document lake in full-precision ColPali, storage hit 2.1TB and monthly vector hosting crossed $2,800. Query latency stayed fine. The bill did not. Finance asked whether each of 40,000 pages earned its keep. Most did not: 8% of pages served 91% of queries.

Fix had three moves. Binary quantization cut storage 32x to 66GB. Tiered indexing kept hot pages full-precision and warm pages binary-only. Cold pages fell back to text chunks. Monthly cost dropped to $340 with answer quality inside 1 point of full-precision on our golden set. Pydantic v2.8 interfered once more, dropping the binary_mode flag from nested index config until extra="allow" restored it, so one shard silently served full-precision for a week. Read effective index settings back after every deploy.

Runnable production code: visual index plus rerank query

Screenshot, embed, binary-index, rerank, answer.

File 1: config.py

from pydantic_settings import BaseSettings
from pydantic import Field

class Settings(BaseSettings):
    dpi: int = Field(default=150, alias="COLPALI_DPI")
    binary_mode: bool = True
    retrieve_k: int = 100
    rerank_k: int = 10
    feed_pages: int = 3

    class Config:
        extra = "allow"

settings = Settings()

File 2: retrieve.py

import logging
from config import settings

log = logging.getLogger("colpali")

def index_pages(pdf_paths: list, indexer) -> dict:
    # Screenshot each page, embed patches, store binary with full-precision hot tier
    stats = {"pages": 0, "hot": 0}
    for path in pdf_paths:
        pages = indexer.screenshot(path, dpi=settings.dpi)
        for page in pages:
            vecs = indexer.embed(page, binary=settings.binary_mode)
            indexer.upsert(page_id=page.id, vecs=vecs)
            stats["pages"] += 1
    log.info("indexed %d pages binary=%s", stats["pages"], settings.binary_mode)
    return stats

def query(question: str, index, llm) -> dict:
    cands = index.maxsim(question, k=settings.retrieve_k)
    top = index.rerank(question, cands, k=settings.rerank_k)
    pages = [c.page for c in top[:settings.feed_pages]]
    answer = llm.answer(question, pages=pages)
    return {"answer": answer, "sources": [p.id for p in pages]}

if __name__ == "__main__":
    print("stages: screenshot, embed, binary index, maxsim, rerank, answer")

File 3: requirements.txt

byaldi==0.4.0
qwen-vl-utils==0.0.8
pydantic==2.8.0
pydantic-settings==2.5.0
pillow==11.0.0

Run it:

uv pip install -r requirements.txt
python retrieve.py

Step 1: screenshot at 150 DPI and embed a 200-page sample. Step 2: measure table-question accuracy against your text baseline. Step 3: flip binary mode on, rerank on top, and confirm quality holds before indexing the lake. The World Labs Atlas photo-to-3D pattern shares the philosophy: pixels first, extraction never.

Resolution tuning: DPI is a cost dial

Screenshot DPI sets the accuracy bill directly. At 72 DPI small table fonts blur and digit confusion rises, with our 6pt footnote accuracy at 64%. At 150 DPI the same set hits 91% while vectors per page roughly double against 72. At 300 DPI gains flatten to 93% while storage and embed compute double again. The knee sits near 150 for Latin business documents. Handwriting and dense CJK sheets push the knee toward 200. I profile each collection once: sample 50 pages at three DPIs, score table questions, and lock the cheapest DPI within 2 points of best. Re-profile when document sources change. Scanner fleets drift over quarters and phone photos vary by device. One quarterly afternoon protects the whole index from silent quality rot.

Tiering strategy that holds at 40,000 pages

Not every page deserves full precision. Query logs consistently show 8% of pages serving over 90% of traffic. Tier by measured heat: hot pages stay full-precision with precomputed embeddings refreshed on source change, warm pages live binary-only with on-the-fly rerank, cold pages fall back to text chunks or re-embed on first hit. Promotion runs weekly from access counters. Demotion runs monthly with a 90-day cold threshold. Our lake settled at 3% hot, 22% warm, 75% cold, holding answer quality within 1 point of all-full-precision while cutting hosting 88%. Pin tier assignments in versioned index manifests so rollbacks restore both vectors and routing. Cold-start latency on demoted pages averages 1.8 seconds for on-demand embed, acceptable for rare queries and invisible in aggregate p50.

When NOT to go visual

Do not index clean markdown-native docs visually. Text chunks win on cost and speed where layout carries no signal. Route by document class, not habit.

Do not skip the rerank stage. Binary retrieval alone bleeds precision on dense tables. Rerank top 10 in full precision always.

Do not feed whole pages blindly. Three sourced pages with citations beat ten dumped pages. The effort-tier economics with 40% savings apply: retrieved context dominates thinking tokens, so precision upstream saves downstream.

Verdict for September 2026 document agents

Parse where layout is noise, screenshot where layout is signal. Measure table accuracy on real scans and let the numbers route each collection.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World. I build document agents at SaaSNext and eval on scans, not markdown. More at https://deepakbagada.in.