ChatGPT can't read your company's internal docs. It doesn't know your codebase. It hallucinates citations and invents facts with full confidence. RAG fixes all three by giving the model relevant text BEFORE it answers. This is how Perplexity, Notion AI, Glean, and every "AI assistant trained on your data" actually works — and it's the highest-leverage 30 minutes you'll spend in 2026.
Learning Objectives
After this lesson, you will be able to:
Understand why AI models make things up (hallucinate) and how RAG fixes this by letting them look up real facts before answering
Compare three ways to give AI new knowledge -- stuffing info into the prompt, retraining the model, or searching a knowledge base at question time (RAG) -- and know when to use each
Walk through the core RAG loop step by step: take the user's question, search for relevant documents, feed them to the AI, and generate a grounded answer with citations
Identify when RAG is the right choice versus fine-tuning, and understand the trade-offs between retrieval latency, source attribution, and knowledge freshness
You are about to learn one of the most important ideas in modern AI. If anything feels confusing, that is completely normal -- RAG clicked for most people only after seeing it in action. Stick with this lesson and it will make sense.
Large Language Models are remarkably capable. They can write code, summarize documents, translate languages, and reason about complex topics. But they have a fundamental flaw: they make things up. Confidently, fluently, and convincingly -- but entirely wrong. This is the hallucination problem, and it is the single biggest barrier to deploying LLMs in production systems where accuracy matters.
LLMs are trained to predict the next token. They learn statistical patterns from vast corpora of text. When you ask a question, the model does not "look up" the answer -- it generates tokens that are statistically likely to follow your question. Most of the time, this produces correct-sounding output. But the model has no mechanism to distinguish between what it "knows" and what it is fabricating.
What Do You Think?
If you ask an LLM 'What is the population of Springfield, Illinois as of 2024?', what happens?
The answer is that it generates a plausible-sounding number. The model might say "approximately 114,230" with perfect confidence, even if that number is slightly (or wildly) wrong. It has no internal fact-checking mechanism. The generation process is fundamentally about statistical likelihood, not factual accuracy.
Training data is a snapshot: Models are trained on data with a cutoff date. Anything after that date is unknown territory, but the model will still generate answers as if it knows.
Compression, not memorization: The model compresses billions of documents into billions of parameters. Details get lost. The model "remembers" patterns and associations, not exact facts.
No grounding mechanism: There is no connection between the model's internal representations and any external source of truth. It cannot verify its own outputs.
Confidence without calibration: The model produces text with the same fluent confidence whether it is correct or hallucinating. There is no built-in "I'm not sure" signal.
P(hallucination)∝frequency in training data1×specificity of query
When you need an LLM to know something it was not trained on -- your company's internal docs, recent news, domain-specific knowledge -- you have three main options.
Simply paste the relevant information into the prompt:
Given this document: [entire document here]
Answer the following question: ...
Pros: Simple. No infrastructure needed. Works immediately.
Cons: Limited by context window size. Expensive (you pay per token). Does not scale to large knowledge bases. You must know which document to include.
Try it! Open ChatGPT (or any LLM) and paste a short paragraph of text, then ask a question about it. You just did prompt engineering -- the simplest form of giving an AI new knowledge. Notice how the answer is grounded in your text. Now imagine doing this with 10,000 documents. That is where RAG comes in.
Train the model on your specific data so it "memorizes" the knowledge:
Pros: Fast inference (no retrieval step). Can learn domain-specific patterns and tone.
Cons: Expensive to train. Knowledge goes stale -- you must retrain when data changes. Still hallucinates, just with domain-flavored hallucinations. Cannot easily cite sources.
Before any queries, you prepare your knowledge base. Documents are split into chunks, converted to vector embeddings, and stored in a vector database. This is a one-time (or periodic) offline process.
The question is converted to a vector embedding using the same embedding model used during ingestion. This vector captures the meaning of the question.
The query vector is compared against all document chunk vectors in the database. The most similar chunks are retrieved -- typically the top 3-10 most relevant passages.
The retrieved chunks are inserted into the LLM's prompt as context: "Given the following relevant documents: [chunks]. Answer the user's question: [query]."
The LLM generates an answer grounded in the retrieved context. Because the relevant facts are right there in the prompt, the model can produce accurate, sourced answers instead of hallucinating.
The term "Retrieval-Augmented Generation" was coined by Lewis et al. in their 2020 paper. The key insight was combining a parametric memory (the LLM's weights) with a non-parametric memory (a searchable document store).
Tests · Verify that the retrieval returns the two most relevant documents for the refund query. Verify the augmented prompt includes the retrieved text.
RAG grounds LLMs in real facts to prevent hallucination. By retrieving relevant documents before generating answers, RAG gives the model an "open book" rather than relying solely on potentially outdated or incorrect memorized knowledge
The core RAG loop is query-retrieve-augment-generate. Convert the user's question to a vector, find the most relevant documents, inject them into the prompt context, and let the LLM generate a grounded answer
RAG vs fine-tuning vs prompt engineering serve different needs. RAG is best for dynamic, frequently updated knowledge; fine-tuning for teaching new capabilities or styles; prompt engineering for simple customizations
RAG keeps knowledge updatable without retraining. Unlike fine-tuning which bakes knowledge into model weights, RAG lets you add, remove, or update documents in real-time without touching the model
Now you understand why RAG exists and what problem it solves. Next up: Embeddings & Vector Similarity -- the mathematical foundation that makes semantic search possible. How do you convert text into numbers that capture meaning?