Most "documents" aren't text — they're 2D layouts. Slides, contracts, scientific figures, medical charts. OCR throws away 40–60% of the signal. ColPali (Faysse 2024) ended the OCR-RAG era by embedding page images directly with a vision-language encoder and retrieving via ColBERT-style late interaction. Every serious legal, medical, and scientific RAG system flipped to visual retrievers within months. This lesson is the new default for visually rich documents in 2026.
Learning Objectives
After this lesson, you will be able to:
Explain why text-only RAG fails on visually rich documents — and why OCR-then-embed loses 40%+ of the signal in slides, charts, scanned PDFs, and scientific figures
Use ColPali-style late-interaction vision-language retrievers to embed each document patch and retrieve by max-similarity — the 2024-2026 production standard
Pick between OCR-then-retrieve, CLIP-style joint embedding, and ColPali late-interaction based on document complexity and latency budget
Build a multimodal RAG pipeline that retrieves document images and feeds them directly into a multimodal LLM (GPT-4V, Claude 3, Gemini) — the modern visual-RAG architecture
Don't worry if "multimodal retrieval" sounds intimidating — the core idea is simple: stop converting documents to text and back, just embed the document images directly and let a vision-language model do the retrieval and the answering.
Try it: See what OCR throws awayInteractive
Open the /play/embedding-distance playground and compare the embedding distance between a page rendered as image vs the same page after OCR. For tables, charts, and laid-out PDFs, the OCR'd version is consistently further from the query — exactly the 40-60% signal loss ColPali fixes.
CLIP-style retrieval: encode every page as one image embedding, retrieve top-k by query-image cosine similarity. Better than OCR for figure-heavy docs but one embedding per page is too coarse for dense documents (a single page may answer five different queries about five different regions).
ColPali fixes the granularity problem. Encode each page using PaliGemma's vision encoder, but keep all 1024 patch embeddings instead of pooling. At retrieval time, score each (page, query) pair by MaxSim:
score(q,d)=i=1∑∣q∣j=1max∣d∣⟨Eqi,Edj⟩
ColPali results on ViDoRe: 20-40% nDCG@5 improvement over OCR-then-embed pipelines. ColQwen2 (smaller, faster) and ColSmolVLM extend the family.
Before the storage math, let's anchor the late-interaction trick with a quick check.
Quick check
Two retrievers index 10,000 PDF pages. System A: one 768-d CLIP embedding per page. System B: ColPali, 1024 patches × 128-d per page. A query is 'show the row in the revenue table where Q3 2025 exceeded forecast.' Which is more likely to surface the correct page, and why?
Storage cost: ∣pages∣×1024×128 floats per page⇒∼0.5 MB / page
What Do You Think?
You have a 50-page PDF report with mostly text but several embedded charts, tables, and a few diagrams. Which retrieval approach gives the best query-time recall on the chart/table content?
The answer: ColPali wins because late interaction handles both the text content AND the chart/table content in one pipeline. OCR misses the charts entirely. CLIP's single page embedding can't find a specific cell in a table on a busy page. Long-context GPT-4V works but costs ~100× more per query.
PDF / image corpus
↓
ColPali / ColQwen encoder → patch embeddings (1024 per page)
↓
Vespa / Weaviate / Qdrant with late-interaction support
↓
Query → query encoder → MaxSim retrieval → top-k pages (as IMAGES)
↓
Multimodal LLM (Claude 3 / GPT-4V / Gemini) given query + retrieved images
↓
Grounded answer with image citations
The shift from text-RAG: retrieved context is images, not text strings. The multimodal LLM reads the images directly. This preserves layout, figures, tables, equations, signatures — everything that OCR loses.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
Tests · Verify ColQwen2 retrieves pages with relevant content. Compare against CLIP baseline on a query about specific numbers in a table — ColQwen2 should rank the table page much higher.
The whole ColPali trick is one summation: score(q, d) = Σ_i max_j ⟨E_qi, E_dj⟩. The playground below builds patch embeddings, runs MaxSim by hand, and plots storage cost vs corpus size so you can pick the right encoder for your fleet.
OCR-then-text-RAG loses 40-60% of information on visually rich documents. Slides, scanned forms, scientific papers, contracts all carry layout/figure/table signal that OCR throws away
ColPali late interaction is the 2024+ production standard. Encode each page as ~1024 patch embeddings, retrieve by MaxSim over query tokens; beats OCR pipelines by 15-30 NDCG points on ViDoRe
The pipeline shift: retrieved context is images, not text. Modern multimodal LLMs (Claude 3, GPT-4V, Gemini) can take retrieved page images directly, preserving layout end-to-end
Storage cost ~100× higher than single-vector retrieval but quality jump justifies it for visually rich corpora; vector DBs (Vespa, Weaviate, Qdrant) now support late interaction natively
Decision rule: clean text → text RAG; visually rich content → ColPali; image-only → CLIP/SigLIP; mixed → ColPali with text fallback
Why does ColPali store ~1024 embeddings per page instead of one pooled embedding?
Multimodal retrieval flips the RAG playbook — stop converting documents to text and back, just embed the images directly. Next: how to evaluate any RAG system rigorously, with RAGAS and the modern faithfulness/relevancy/recall metrics.