Why RAG?

You are about to learn one of the most important ideas in modern AI. If anything feels confusing, that is completely normal — RAG clicked for most people only after seeing it in action. Stick with this lesson and it will make sense.

#The Open-Book Exam

Large Language Models are remarkably capable. They can write code, summarize documents, translate languages, and reason about complex topics. But they have a fundamental flaw: they make things up. Confidently, fluently, and convincingly, but entirely wrong. This is the hallucination problem, and it is the single biggest barrier to deploying LLMs in production systems where accuracy matters.

#The Hallucination Problem

LLMs are trained to predict the next token. They learn statistical patterns from vast corpora of text. When you ask a question, the model does not "look up" the answer — it generates tokens that are statistically likely to follow your question. Most of the time, this produces correct-sounding output. But the model has no mechanism to distinguish between what it "knows" and what it is fabricating.

The answer is that it generates a plausible-sounding number. The model might say "approximately 114,230" with perfect confidence, even if that number is slightly (or wildly) wrong. It has no internal fact-checking mechanism. The generation process is fundamentally about statistical likelihood, not factual accuracy.

#Why Hallucinations Happen

1. Training data is a snapshot: Models are trained on data with a cutoff date. Anything after that date is unknown territory, but the model will still generate answers as if it knows.
2. Compression, not memorization: The model compresses billions of documents into billions of parameters. Details get lost. The model "remembers" patterns and associations, not exact facts.
3. No grounding mechanism: There is no connection between the model's internal representations and any external source of truth. It cannot verify its own outputs.
4. Confidence without calibration: The model produces text with the same fluent confidence whether it is correct or hallucinating. There is no built-in "I'm not sure" signal.

#The Three Strategies for Adding Knowledge

When you need an LLM to know something it was not trained on (your company's internal docs, recent news, domain-specific knowledge) you have three main options.

#Strategy 1: Prompt Engineering (Stuffing Context)

Simply paste the relevant information into the prompt:

Given this document: [entire document here]
Answer the following question: ...

Try it! Open ChatGPT (or any LLM) and paste a short paragraph of text, then ask a question about it. You just did prompt engineering — the simplest form of giving an AI new knowledge. Notice how the answer is grounded in your text. Now imagine doing this with 10,000 documents. That is where RAG comes in.

#Strategy 2: Fine-Tuning

Train the model on your specific data so it "memorizes" the knowledge:

#Strategy 3: RAG (Retrieval-Augmented Generation)

At query time, search a knowledge base for relevant documents, then feed them to the LLM as context:

#The RAG Loop

RAG follows a deceptively simple loop that we will spend the rest of this track unpacking in detail:

#The Original RAG Paper

The term "Retrieval-Augmented Generation" was coined by Lewis et al. in their 2020 paper. The key insight was combining a parametric memory (the LLM's weights) with a non-parametric memory (a searchable document store).

#When RAG Shines (and When It Does Not)

RAG is not a silver bullet. It excels in specific scenarios:

RAG is ideal when

• Knowledge changes frequently (news, docs, inventory)
• You need source citations and traceability
• Your knowledge base is large (thousands to millions of documents)
• You need to keep the LLM's knowledge current without retraining
• Different users need access to different subsets of knowledge

RAG is less ideal when

• You need the model to internalize reasoning patterns, not just facts
• Latency is critical and you cannot afford the retrieval step
• Your knowledge is small enough to fit entirely in the context window
• You need the model to learn a specific tone or style

#Try It Yourself

#Key Takeaways

• RAG grounds LLMs in real facts to prevent hallucination. By retrieving relevant documents before generating answers, RAG gives the model an "open book" rather than relying solely on potentially outdated or incorrect memorized knowledge
• The core RAG loop is query-retrieve-augment-generate. Convert the user's question to a vector, find the most relevant documents, inject them into the prompt context, and let the LLM generate a grounded answer
• RAG vs fine-tuning vs prompt engineering serve different needs. RAG is best for dynamic, frequently updated knowledge; fine-tuning for teaching new capabilities or styles; prompt engineering for simple customizations
• RAG keeps knowledge updatable without retraining. Unlike fine-tuning which bakes knowledge into model weights, RAG lets you add, remove, or update documents in real-time without touching the model

#Quick Check