Track 08 · RAG Systems · 12 min
Give your AI a memory.
LLMs hallucinate because they don't know what they don't know. RAG fixes that — give the model a knowledge base it can search, and let it cite sources. Five interactive demos take you through the production-grade pipeline behind Perplexity, Glean, and ChatGPT's enterprise search.
“Retrieval-augmented generation is the cheapest, fastest way to get an LLM to stop making things up.”
#The hook
When you ask ChatGPT "what's the latest version of Postgres?", it might tell you 14, 15, 16, 17 — because its training data ended at some point in the past. When you ask "what does our company's expense policy say about international travel?", it has no idea — your policy was never in its training data.
#Why this matters in 2026 — the receipts
RAG by the numbers
Where AI meets your knowledge
60%
Enterprise LLM deployments using RAG
McKinsey 2025
250M+
Perplexity searches/month
2025 disclosures
1B+
Vector DB queries/day in prod systems
Pinecone/Weaviate
90%
Hallucination reduction with RAG (vs base LLM)
Stanford HELM
90%
Hallucination reduction with RAG vs base LLM
In Stanford HELM and academic benchmarks, RAG-augmented LLMs hallucinate roughly 90% less than the same base model answering from training data alone. The cost: a vector DB and a reranker. The win is enormous — and explains why ~60% of enterprise LLM deployments now use RAG.
Stanford HELM, 2024
Vocabulary
Six RAG terms you'll meet daily
Concept
Chunking
Split docs into manageable pieces — too big dilutes signal, too small misses context.
Like: Cutting a long pizza for sharing.
e.g. Recursive splitter at 500-1000 tokens
Concept
Embedding
Turn each chunk into a vector. Search is then nearest-neighbor.
Like: GPS coordinates for meaning.
e.g. OpenAI text-embedding-3, Voyage 2026
Concept
Vector DB
A database that indexes vectors for fast nearest-neighbor search.
Like: A library cataloged by ideas, not titles.
e.g. Pinecone, Weaviate, pgvector, Qdrant
Concept
Hybrid search
Combine vector similarity + keyword (BM25) for the best of both.
Like: A doctor checking your chart and your symptoms.
e.g. Standard production setup
Concept
Reranking
A cross-encoder re-scores top-K candidates for true relevance.
Like: Second-round interview after the resume scan.
e.g. Cohere Rerank, Voyage Rerank
Concept
Agentic RAG
Agent decides what to retrieve, when, and how — multi-step.
Like: A researcher who chases citations themselves.
e.g. Self-RAG, plan-and-execute
#The five-stage pipeline
The recipe
Five stages from documents to citations
1. Ingest & chunk
Process the docsTake your knowledge base (PDFs, web pages, Slack archive, code) and split into manageable chunks.
- Chunk size matters: too small misses context, too big dilutes signal.
- Semantic chunking (split by topic) beats fixed-size chunking.
- Always store metadata: source URL, document title, last-updated, owner.
2. Embed & index
Vectorize the chunksTurn each chunk into a high-dimensional embedding. Store in a vector database.
- Embedding models: OpenAI text-embedding-3, Voyage, BGE, E5. The choice matters.
- Vector DBs: Pinecone, Weaviate, pgvector, Qdrant, Chroma. Each with tradeoffs.
- Hybrid search (vectors + BM25 keyword) usually beats pure vector search.
3. Retrieve
Find relevant chunksEmbed the user query. Find the K most-similar chunks via vector search.
- Cosine similarity is the default metric. Approximate nearest-neighbor (HNSW, IVF) for speed.
- Query rewriting / expansion can dramatically improve retrieval quality.
- Multi-step retrieval (one query for the question, another for context) is the 2026 default.
4. Re-rank
Filter the noiseUse a cross-encoder to re-score the top-K chunks for true relevance.
- Vector search is fast but coarse. Re-rankers are slow but precise.
- Cohere Rerank, Voyage Rerank, BGE-Reranker — the field leaders.
- Re-rankers add 30%+ accuracy in head-to-head evals.
5. Generate with citations
SynthesizePass the top-N chunks + user question to the LLM. Require it to cite which chunk supports which claim.
- System prompt: 'Answer using only the provided context. Cite sources by number.'
- Track which chunks were used → audit trail for every answer.
- Self-RAG: the LLM decides what to retrieve, when, and what to cite.
#See it work — embeddings and vector search
The core idea: similar meanings end up near each other in a high-dimensional embedding space. Search becomes "find the K nearest neighbors of the query embedding."
#See it work — the full pipeline
#A real RAG-flavored search — runnable
# Tiny RAG end-to-end with NumPy and a fake embedder
import numpy as np
# Step 1: a corpus of "documents"
docs = [
"Python is a high-level programming language created by Guido van Rossum in 1991.",
"Rust is a systems programming language focused on safety and performance.",
"JavaScript runs in browsers and powers most web applications.",
"SQL is the standard language for managing relational databases.",
"Go was designed at Google for concurrent network services.",
]
# Step 2: turn each doc into a (very simplified!) embedding
def fake_embed(text):
"""Toy embedding: count of common keywords. Real systems use neural nets."""
keywords = ["language", "system", "browser", "database", "network", "concurrent", "safe", "fast"]
return np.array([text.lower().count(k) for k in keywords], dtype=float)
doc_embeddings = np.array([fake_embed(d) for d in docs])
# Step 3: a user query
query = "I want a fast language for systems"
query_embedding = fake_embed(query)
# Step 4: cosine similarity — find the closest doc
def cosine(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9)
similarities = [cosine(query_embedding, e) for e in doc_embeddings]
# Step 5: sort and show top 3
ranked = sorted(zip(docs, similarities), key=lambda x: -x[1])
print(f"Query: {query}\n")
print("Top results:")
for i, (doc, sim) in enumerate(ranked[:3]):
print(f" {i+1}. [{sim:.2f}] {doc}")
# Step 6: now an LLM would synthesize an answer from doc #1
print(f"\nAnswer-with-citation: {ranked[0][0]} [source 1]")That's the entire RAG core in 30 lines. Real systems use real neural embeddings and proper vector DBs, but the algorithm is identical.
#What's been built with RAG
RAG in production
What 'AI with citations' has actually shipped
AI search
Perplexity
250M+
Searches / month
Real-time web RAG with citations. Built on top of frontier LLMs + custom retrieval. Replacing Google for many.
Web RAG
Enterprise search
Glean
$1B+
Valuation
RAG over your company's Slack, Google Drive, GitHub, Notion, Salesforce. The first RAG unicorn.
Enterprise RAG
Personal docs RAG
NotebookLM
1M+
Active users
Google's RAG for your own uploaded docs. Generates podcasts, summaries, citations. Free.
Document RAG
App-integrated RAG
ChatGPT Connectors
100+
Connectors available
Connect ChatGPT to Drive, Slack, Notion, Salesforce. RAG productized for everyone.
Connector RAG
Vector DB infrastructure
Vespa / Pinecone
100+
Major customers
Vespa runs Yahoo, Pinecone is the cloud-native default. RAG infra is now its own market.
Vector DB
Reranking-as-a-service
Cohere Rerank
30%+
Accuracy lift over vector-only
Plug-and-play reranker. The cheapest accuracy upgrade for any RAG system.
Reranking
#The 2026 frontier
#Where to go next
- RAG Systems track — 15 lessons: embeddings, vector DBs, chunking, hybrid search, reranking, GraphRAG.
- NLP & Transformers — embeddings come from the same family of models.
- AI Agents — agentic RAG is where the field is going.
- ML Engineering — production RAG monitoring, latency, evaluation.
#Key takeaways
Key Takeaways
- RAG = give the LLM a knowledge base it can search before answering. Eliminates hallucinations and adds citations.
- Five-stage pipeline: ingest+chunk → embed+index → retrieve → re-rank → generate with citations.
- Hybrid search (vector + BM25 keyword) usually beats pure vector search.
- Re-ranking with a cross-encoder adds 30%+ accuracy. The cheapest production upgrade.
- 2026 frontier: agentic RAG (multi-step retrieval), GraphRAG, multi-modal RAG, long-context-vs-RAG tradeoffs.
- RAG vs fine-tuning: RAG for changing knowledge, fine-tuning for style/format. Most teams need both.
#References & further reading
- Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020). The paper that named it.
- Asai et al. — Self-RAG: Learning to Retrieve, Generate, and Critique (2023). Agentic RAG.
- Microsoft Research — GraphRAG (2024). Knowledge-graph-augmented RAG.
- Anthropic — Contextual Retrieval (2024). Best engineering blog on production RAG.
- LlamaIndex / LangChain documentation — practical references.
- Pinecone / Weaviate / Vespa engineering blogs — vector DB internals.