The RAG Pipeline

Most teams ship a RAG demo in a weekend and a broken RAG product in production. The pipeline looks identical on a whiteboard, but the difference is the seven stages where things silently go wrong. This lesson walks the full pipeline end-to-end — extraction, chunking, embedding, retrieval, reranking, prompt assembly, generation — and shows you exactly where Perplexity, Glean, and Notion AI invested to make theirs actually work.

Learning Objectives

After this lesson, you will be able to:

Follow a user's question through every step of the RAG pipeline — from loading documents, to searching, to generating an answer — and understand what happens at each stage
Spot how a failure at any one stage (bad chunking, poor retrieval, weak prompt) cascades and ruins the final answer
Write prompt templates that tell the AI 'answer ONLY from these documents' so it stays grounded instead of making things up
Debug RAG failures systematically from left to right (extraction first, then chunking, then retrieval, then generation) instead of jumping straight to prompt tuning

Prerequisites

Before You Start

Review

You know why RAG exists. Now let's go deep on how each stage actually works in code.

This is the "putting it all together" lesson. Everything you have learned so far (embeddings, chunking, vector search) connects here into one working system. Take your time and enjoy watching the pieces fit together.

#The Full RAG Pipeline: End to End

Try it: Step through the RAG PipelineInteractive

Loading visualization...

#Step 1: User Question

A user asks a natural language question: "What is our refund policy for enterprise customers?" This is the starting point of every RAG query. The system must find the answer in its knowledge base and return a grounded, cited response.

#Step 2: Embed the Query

The question is passed through an embedding model to produce a dense vector. This vector captures the semantic meaning of the question, not just its keywords, but its intent. The same model that was used to embed documents must be used here.

#Step 3: Vector Search

The query vector is compared against all document vectors in the database using approximate nearest neighbor search (HNSW or IVF). In milliseconds, the system identifies the chunks whose meaning is closest to the question.

#Step 4: Retrieve Top-K Chunks

The most similar chunks are pulled from the database — typically the top 3 to 10. Each chunk includes the original text, similarity score, and metadata (source document, section, page number). These are the raw materials for the answer.

#Step 5: Rerank Results

A cross-encoder re-ranker scores each query-chunk pair more carefully than the initial vector search. This second pass reorders the results, promoting the truly relevant chunks and demoting false positives. It catches nuances that embedding similarity misses.

#Step 6: Assemble the Prompt

The system constructs the final LLM prompt: a system message (instructions on how to answer), the retrieved context (the top-ranked chunks), and the user question. The prompt explicitly tells the model to answer only from the provided context and to cite sources.

#Step 7: LLM Generates the Answer

The assembled prompt is sent to the LLM, which generates an answer grounded in the retrieved context. Because the relevant facts are right there in the prompt, the model produces accurate, specific answers instead of hallucinating from its training data.

#Step 8: Return with Citations

The final answer is returned to the user along with source citations, which documents, sections, and page numbers the answer came from. This makes the response verifiable and builds trust. The user can click through to the original source to verify.

#Phase 1: Offline Ingestion

The ingestion pipeline runs when you add or update documents. It transforms raw documents into a searchable vector index.

#Stage 1: Document Loading

Raw documents come in many formats: PDFs, HTML pages, Markdown files, Word documents, Slack messages, Notion pages, database records. The first stage converts them all into a uniform text representation.

This is harder than it sounds. PDFs have headers, footers, page numbers, and multi-column layouts. HTML has navigation bars, ads, and boilerplate. The quality of your text extraction directly limits the quality of everything downstream.

pythonrunnable cell

# Common document loaders
from langchain.document_loaders import (
    PyPDFLoader,        # PDFs
    UnstructuredHTMLLoader,  # HTML
    NotionDBLoader,     # Notion
    CSVLoader,          # CSV files
)
docs = PyPDFLoader("report.pdf").load()

Try it! Install langchain (pip install langchain) and try loading a PDF you have on your computer. Print the first 500 characters of the extracted text. You will immediately see the messiness (headers, footers, page numbers mixed in) that the next stage must clean up.

#Stage 2: Text Cleaning & Preprocessing

Raw extracted text is messy. Remove:

Headers and footers repeated on every page
Navigation elements from web pages
Excessive whitespace and formatting artifacts
Boilerplate (cookie notices, disclaimers)

Preserve:

Section headers and hierarchy
Tables (convert to structured text)
Code blocks (keep formatting intact)
Meaningful formatting (lists, emphasis)

This stage is often neglected but dramatically affects downstream quality.

#Stage 3: Chunking

Split cleaned documents into appropriately-sized passages using the strategies from the previous lesson. The choice of chunking strategy depends on document type, embedding model, and query patterns.

Key parameters to set:

Chunk size: Typically 256-1024 tokens
Overlap: Typically 10-20% of chunk size
Strategy: Recursive, semantic, or document-type-specific

pythonrunnable cell

from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_documents(docs)

#Stage 4: Embedding

Each chunk is passed through an embedding model to produce a dense vector. This is the most expensive stage computationally (for large corpora) and the most impactful for retrieval quality.

For a corpus of 1 million chunks at 500 tokens each, embedding with OpenAI text-embedding-3-small costs approximately $10 and takes about 30 minutes with batched API calls.

pythonrunnable cell

from openai import OpenAI
client = OpenAI()

def embed_batch(texts, model="text-embedding-3-small"):
    response = client.embeddings.create(input=texts, model=model)
    return [e.embedding for e in response.data]

# Process in batches of 100 for API efficiency
vectors = []
for i in range(0, len(chunks), 100):
    batch = [c.page_content for c in chunks[i:i+100]]
    vectors.extend(embed_batch(batch))

#Stage 5: Indexing & Storage

Store the vectors in a vector database along with the original text and metadata. The metadata is crucial — it enables filtering, source attribution, and deduplication.

pythonrunnable cell

import chromadb
client = chromadb.Client()
collection = client.create_collection("knowledge_base")

collection.add(
    ids=[f"chunk_{i}" for i in range(len(chunks))],
    embeddings=vectors,
    documents=[c.page_content for c in chunks],
    metadatas=[{
        "source": c.metadata["source"],
        "page": c.metadata.get("page", 0),
        "chunk_index": i,
    } for i, c in enumerate(chunks)]
)

At this point, your knowledge base is ready to answer queries.

#Phase 2: Online Query

When a user asks a question, the online pipeline activates. This must be fast — users expect answers in 1-3 seconds.

#Stage 6: Query Processing

The user's raw question may need preprocessing before embedding:

Query expansion: Add related terms. "Python error handling" might expand to "Python exception try except error handling."
Query rewriting: An LLM rephrases the query for better retrieval. "What did the CEO say about Q3?" becomes "CEO statement quarterly earnings Q3 revenue."
Intent classification: Determine if retrieval is needed at all. "Hello, how are you?" does not need a knowledge base search.

pythonrunnable cell

def process_query(raw_query):
    # Optional: use an LLM to rewrite for better retrieval
    rewritten = llm.rewrite(
        f"Rewrite for document retrieval: {raw_query}"
    )
    return rewritten or raw_query

#Stage 7: Query Embedding

Convert the processed query to a vector using the same embedding model used during ingestion. Using a different model is a common bug — the vectors live in different spaces and similarity scores become meaningless.

pythonrunnable cell

query_vec = embed_batch([processed_query])[0]

#Stage 8: Retrieval

Search the vector database for the most similar chunks. Typical parameters:

top_k: Number of chunks to retrieve (3-10)
similarity threshold: Minimum similarity score (optional)
metadata filters: Restrict to specific sources, dates, or access levels

pythonrunnable cell

results = collection.query(
    query_embeddings=[query_vec],
    n_results=5,
    where={"source": {"$in": allowed_sources}}
)

#Stage 9: Post-Retrieval Processing

Before sending chunks to the LLM, optimize the context:

Re-ranking: Use a cross-encoder to re-score and reorder results
Deduplication: Remove near-duplicate chunks
Relevance filtering: Drop chunks below a similarity threshold
Context ordering: Place most relevant chunks first (or use "lost in the middle" aware ordering)

pythonrunnable cell

# Re-rank with a cross-encoder
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, doc) for doc in retrieved_docs]
scores = reranker.predict(pairs)
reranked = sorted(zip(scores, retrieved_docs), reverse=True)
top_docs = [doc for _, doc in reranked[:3]]

#Stage 10: Prompt Construction

Build the final prompt that combines the user's question with retrieved context. This is where prompt engineering meets retrieval.

pythonrunnable cell

prompt = f"""Answer the user's question based ONLY on the
following context. If the context doesn't contain the answer,
say "I don't have enough information to answer that."

Context:
{chr(10).join(top_docs)}

Question: {user_query}

Answer:"""

The system prompt must:

Instruct the model to use only the provided context
Tell the model what to do when context is insufficient
Specify the desired output format
Optionally request source citations

#Stage 11: Generation

Send the constructed prompt to the LLM and stream the response back to the user.

pythonrunnable cell

response = llm.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": augmented_prompt}
    ],
    stream=True,
    temperature=0.1  # Low temp for factual answers
)

Low temperature (0.0-0.3) is standard for RAG because you want the model to faithfully reproduce information from the context, not be creative. Higher temperatures lead to more paraphrasing and potential deviation from the retrieved facts.

#The Cascade Effect

What Do You Think?

If your chunking splits a key paragraph across two chunks, what happens downstream?

Every stage in the pipeline depends on the stages before it. Failures cascade:

\text{Quality}_{\text{answer}} \leq \min(\text{Quality}_{\text{extract}}, \text{Quality}_{\text{chunk}}, \text{Quality}_{\text{embed}}, \text{Quality}_{\text{retrieve}}, \text{Quality}_{\text{generate}})

Watch Out

Common Mistake

The single most common RAG debugging mistake: when the system gives wrong answers, developers immediately start tweaking the LLM prompt. But the problem is almost always upstream:

First check extraction: Is the raw text clean? Are tables, headers, and formatting preserved?
Then check chunking: Are relevant passages in a single chunk? Is the chunk that should answer the question actually coherent?
Then check retrieval: Is the relevant chunk in the top-k results? What is its similarity score? Is it being outranked by irrelevant chunks?
Finally check generation: Only if the correct chunk is retrieved and the LLM still gives a wrong answer should you adjust the prompt.

In our experience, roughly 70% of RAG quality issues are extraction + chunking problems, 20% are retrieval problems, and only 10% are generation problems. Debug from left to right.

Watch Out

Using a different embedding model at query time than at ingestion time

Fix: Every embedding model maps text into its own unique vector space. If you embed your documents with 'text-embedding-ada-002' during ingestion but switch to 'text-embedding-3-small' at query time (perhaps to save cost or improve quality), the query vector and document vectors live in completely different spaces and their cosine similarity is meaningless. The result looks like retrieval failure — no relevant documents are returned — but there are no error messages, just silently wrong results. Always track which embedding model version produced each index, store it as metadata, and enforce that the same model is used for both ingestion and querying. When you upgrade your embedding model, you must re-embed the entire corpus.

#Prompt Engineering for RAG

The prompt template is the bridge between retrieval and generation. Here are battle-tested patterns:

#Basic RAG Prompt

You are a helpful assistant. Answer the user's question using
ONLY the provided context. If the context does not contain
enough information, say "I don't have enough information."

Context:
{retrieved_chunks}

Question: {user_question}

#Citation-Aware Prompt

Answer the question using the provided sources. Cite your
sources using [Source N] notation.

Sources:
[Source 1] {chunk_1}
[Source 2] {chunk_2}
[Source 3] {chunk_3}

Question: {user_question}

Provide your answer with citations:

#Structured Output Prompt

Based on the following context, answer the question.
Respond in JSON format with:
- "answer": your answer
- "confidence": "high", "medium", or "low"
- "sources": list of source document names used
- "reasoning": brief explanation of how you derived the answer

Context: {retrieved_chunks}
Question: {user_question}

\text{Effective context} = \sum_{i=1}^{k} w_i \cdot \text{relevance}(\text{chunk}_i, \text{query})

#The Lost in the Middle Problem

Research has shown that LLMs pay more attention to information at the beginning and end of their context, while information in the middle gets less attention. This has practical implications for how you order retrieved chunks:

Place the most relevant chunk first
Place the second most relevant chunk last
Place less relevant chunks in the middle

Alternatively, keep the number of retrieved chunks small (3-5) to avoid the problem entirely.

#Try It Yourself

pythonplayground.py · Pyodide

import math
import re

# Build a complete mini RAG pipeline in Python

# === KNOWLEDGE BASE ===
documents = [
  {"id": "doc1", "title": "Refund Policy", "text": "Standard refund policy: Full refunds are available within 30 days of purchase. After 30 days, store credit will be issued. Digital products are non-refundable once downloaded."},
  {"id": "doc2", "title": "Enterprise Terms", "text": "Enterprise customers receive extended 90-day refund windows. Dedicated account managers handle all refund requests. Volume discounts of 20% apply for orders over 100 licenses."},
  {"id": "doc3", "title": "Shipping Info", "text": "Free shipping on orders over $50. Express shipping available for $15. International shipping takes 7-14 business days. Tracking numbers are emailed within 24 hours."},
  {"id": "doc4", "title": "Account Security", "text": "Two-factor authentication is required for all accounts. Passwords must be at least 12 characters. Account lockout occurs after 5 failed attempts. Contact security@example.com for compromised accounts."},
]

# === CHUNKING (simplified: each doc is one chunk) ===
chunks = [{"id": d["id"], "text": d["text"], "metadata": {"title": d["title"]}} for d in documents]

# === EMBEDDING (simplified: keyword frequency vector) ===
def simple_embed(text):
  keywords = ["refund", "enterprise", "shipping", "security", "policy", "days", "account", "order"]
  words = re.split(r"\W+", text.lower())
  return [words.count(kw) for kw in keywords]

def cosine_sim(a, b):
  dot = sum(x * y for x, y in zip(a, b))
  na = math.sqrt(sum(x ** 2 for x in a))
  nb = math.sqrt(sum(x ** 2 for x in b))
  return dot / (na * nb) if na and nb else 0

# Embed all chunks
chunk_embeddings = [{**c, "embedding": simple_embed(c["text"])} for c in chunks]

# === RETRIEVAL ===
def retrieve(query, k=2):
  query_vec = simple_embed(query)
  scored = [{**c, "score": cosine_sim(query_vec, c["embedding"])} for c in chunk_embeddings]
  scored.sort(key=lambda x: x["score"], reverse=True)
  return scored[:k]

# === PROMPT CONSTRUCTION ===
def build_rag_prompt(query, retrieved_chunks):
  context = "\n\n".join(
      f"[Source {i+1}: {c['metadata']['title']}] {c['text']}"
      for i, c in enumerate(retrieved_chunks)
  )
  return f"Answer based ONLY on the context below. Cite sources.\n\nContext:\n{context}\n\nQuestion: {query}\nAnswer:"

# === RUN THE PIPELINE ===
query = "What is the refund policy for enterprise customers?"
print("Query:", query)
print()

results = retrieve(query, 2)
print("Retrieved chunks:")
for r in results:
  print(f"  [{r['score']:.3f}] {r['metadata']['title']}: {r['text'][:80]}...")
print()

prompt = build_rag_prompt(query, results)
print("Final prompt for LLM:")
print(prompt)
print()

# CHALLENGE: Add a relevance threshold (e.g., score > 0.3).
# If no chunks pass the threshold, return "I don't have
# enough information" instead of building a RAG prompt.
print("CHALLENGE: Add relevance filtering to handle")
print("queries with no good matches.")

Tests · Verify retrieval returns the refund and enterprise documents for the example query. Verify the prompt includes both retrieved chunks with source citations.

#Key Takeaways

RAG is a multi-stage pipeline where failures cascade. Poor chunking leads to bad embeddings, bad embeddings lead to irrelevant retrieval, and irrelevant retrieval leads to hallucinated answers; quality at each stage matters
Document ingestion is the most underestimated stage. Parsing PDFs, cleaning HTML, handling tables, and extracting metadata correctly determines the ceiling of your entire RAG system's quality
Prompt templates must explicitly ground the LLM in retrieved context. Instruct the model to only use provided context, cite sources, and admit when the context does not contain the answer
The retrieval-generation interface is where most RAG systems fail. Even with perfect retrieval, poorly formatted context injection or LLMs that ignore context can produce wrong answers

#Quick Check

Quick Check1 / 4

In a RAG pipeline, what is the most common root cause of poor answer quality?

You now understand the complete RAG pipeline from end to end. But the basic pipeline has limitations — what about queries that need multiple retrieval steps, or hybrid keyword and semantic search? Next up: Advanced RAG Patterns — the techniques that take RAG from good to great.

The RAG Pipeline

Learning Objectives

After this lesson, you will be able to:

Follow a user's question through every step of the RAG pipeline — from loading documents, to searching, to generating an answer — and understand what happens at each stage
Spot how a failure at any one stage (bad chunking, poor retrieval, weak prompt) cascades and ruins the final answer
Write prompt templates that tell the AI 'answer ONLY from these documents' so it stays grounded instead of making things up
Debug RAG failures systematically from left to right (extraction first, then chunking, then retrieval, then generation) instead of jumping straight to prompt tuning

Prerequisites

Before You Start

Review

You know why RAG exists. Now let's go deep on how each stage actually works in code.

#The Full RAG Pipeline: End to End

Try it: Step through the RAG PipelineInteractive

Loading visualization...

#Step 1: User Question

#Step 2: Embed the Query

#Step 3: Vector Search

#Step 4: Retrieve Top-K Chunks

#Step 5: Rerank Results

#Step 6: Assemble the Prompt

#Step 7: LLM Generates the Answer

#Step 8: Return with Citations

#Phase 1: Offline Ingestion

The ingestion pipeline runs when you add or update documents. It transforms raw documents into a searchable vector index.

#Stage 1: Document Loading

pythonrunnable cell

# Common document loaders
from langchain.document_loaders import (
    PyPDFLoader,        # PDFs
    UnstructuredHTMLLoader,  # HTML
    NotionDBLoader,     # Notion
    CSVLoader,          # CSV files
)
docs = PyPDFLoader("report.pdf").load()

Try it! Install langchain (pip install langchain) and try loading a PDF you have on your computer. Print the first 500 characters of the extracted text. You will immediately see the messiness (headers, footers, page numbers mixed in) that the next stage must clean up.

#Stage 2: Text Cleaning & Preprocessing

Raw extracted text is messy. Remove:

Headers and footers repeated on every page
Navigation elements from web pages
Excessive whitespace and formatting artifacts
Boilerplate (cookie notices, disclaimers)

Preserve:

Section headers and hierarchy
Tables (convert to structured text)
Code blocks (keep formatting intact)
Meaningful formatting (lists, emphasis)

This stage is often neglected but dramatically affects downstream quality.

#Stage 3: Chunking

Split cleaned documents into appropriately-sized passages using the strategies from the previous lesson. The choice of chunking strategy depends on document type, embedding model, and query patterns.

Key parameters to set:

Chunk size: Typically 256-1024 tokens
Overlap: Typically 10-20% of chunk size
Strategy: Recursive, semantic, or document-type-specific

pythonrunnable cell

from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_documents(docs)

#Stage 4: Embedding

Each chunk is passed through an embedding model to produce a dense vector. This is the most expensive stage computationally (for large corpora) and the most impactful for retrieval quality.

For a corpus of 1 million chunks at 500 tokens each, embedding with OpenAI text-embedding-3-small costs approximately $10 and takes about 30 minutes with batched API calls.

pythonrunnable cell

from openai import OpenAI
client = OpenAI()

def embed_batch(texts, model="text-embedding-3-small"):
    response = client.embeddings.create(input=texts, model=model)
    return [e.embedding for e in response.data]

# Process in batches of 100 for API efficiency
vectors = []
for i in range(0, len(chunks), 100):
    batch = [c.page_content for c in chunks[i:i+100]]
    vectors.extend(embed_batch(batch))

#Stage 5: Indexing & Storage

Store the vectors in a vector database along with the original text and metadata. The metadata is crucial — it enables filtering, source attribution, and deduplication.

pythonrunnable cell

import chromadb
client = chromadb.Client()
collection = client.create_collection("knowledge_base")

collection.add(
    ids=[f"chunk_{i}" for i in range(len(chunks))],
    embeddings=vectors,
    documents=[c.page_content for c in chunks],
    metadatas=[{
        "source": c.metadata["source"],
        "page": c.metadata.get("page", 0),
        "chunk_index": i,
    } for i, c in enumerate(chunks)]
)

At this point, your knowledge base is ready to answer queries.

#Phase 2: Online Query

When a user asks a question, the online pipeline activates. This must be fast — users expect answers in 1-3 seconds.

#Stage 6: Query Processing

The user's raw question may need preprocessing before embedding:

Query expansion: Add related terms. "Python error handling" might expand to "Python exception try except error handling."
Query rewriting: An LLM rephrases the query for better retrieval. "What did the CEO say about Q3?" becomes "CEO statement quarterly earnings Q3 revenue."
Intent classification: Determine if retrieval is needed at all. "Hello, how are you?" does not need a knowledge base search.

pythonrunnable cell

def process_query(raw_query):
    # Optional: use an LLM to rewrite for better retrieval
    rewritten = llm.rewrite(
        f"Rewrite for document retrieval: {raw_query}"
    )
    return rewritten or raw_query

#Stage 7: Query Embedding

pythonrunnable cell

query_vec = embed_batch([processed_query])[0]

#Stage 8: Retrieval

Search the vector database for the most similar chunks. Typical parameters:

top_k: Number of chunks to retrieve (3-10)
similarity threshold: Minimum similarity score (optional)
metadata filters: Restrict to specific sources, dates, or access levels

pythonrunnable cell

results = collection.query(
    query_embeddings=[query_vec],
    n_results=5,
    where={"source": {"$in": allowed_sources}}
)

#Stage 9: Post-Retrieval Processing

Before sending chunks to the LLM, optimize the context:

Re-ranking: Use a cross-encoder to re-score and reorder results
Deduplication: Remove near-duplicate chunks
Relevance filtering: Drop chunks below a similarity threshold
Context ordering: Place most relevant chunks first (or use "lost in the middle" aware ordering)

pythonrunnable cell

# Re-rank with a cross-encoder
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, doc) for doc in retrieved_docs]
scores = reranker.predict(pairs)
reranked = sorted(zip(scores, retrieved_docs), reverse=True)
top_docs = [doc for _, doc in reranked[:3]]

#Stage 10: Prompt Construction

Build the final prompt that combines the user's question with retrieved context. This is where prompt engineering meets retrieval.

pythonrunnable cell

prompt = f"""Answer the user's question based ONLY on the
following context. If the context doesn't contain the answer,
say "I don't have enough information to answer that."

Context:
{chr(10).join(top_docs)}

Question: {user_query}

Answer:"""

The system prompt must:

Instruct the model to use only the provided context
Tell the model what to do when context is insufficient
Specify the desired output format
Optionally request source citations

#Stage 11: Generation

Send the constructed prompt to the LLM and stream the response back to the user.

pythonrunnable cell

response = llm.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": augmented_prompt}
    ],
    stream=True,
    temperature=0.1  # Low temp for factual answers
)

#The Cascade Effect

What Do You Think?

If your chunking splits a key paragraph across two chunks, what happens downstream?

Every stage in the pipeline depends on the stages before it. Failures cascade:

\text{Quality}_{\text{answer}} \leq \min(\text{Quality}_{\text{extract}}, \text{Quality}_{\text{chunk}}, \text{Quality}_{\text{embed}}, \text{Quality}_{\text{retrieve}}, \text{Quality}_{\text{generate}})

Watch Out

Common Mistake

The single most common RAG debugging mistake: when the system gives wrong answers, developers immediately start tweaking the LLM prompt. But the problem is almost always upstream:

First check extraction: Is the raw text clean? Are tables, headers, and formatting preserved?
Then check chunking: Are relevant passages in a single chunk? Is the chunk that should answer the question actually coherent?
Then check retrieval: Is the relevant chunk in the top-k results? What is its similarity score? Is it being outranked by irrelevant chunks?
Finally check generation: Only if the correct chunk is retrieved and the LLM still gives a wrong answer should you adjust the prompt.

In our experience, roughly 70% of RAG quality issues are extraction + chunking problems, 20% are retrieval problems, and only 10% are generation problems. Debug from left to right.

Watch Out

Using a different embedding model at query time than at ingestion time

#Prompt Engineering for RAG

The prompt template is the bridge between retrieval and generation. Here are battle-tested patterns:

#Basic RAG Prompt

You are a helpful assistant. Answer the user's question using
ONLY the provided context. If the context does not contain
enough information, say "I don't have enough information."

Context:
{retrieved_chunks}

Question: {user_question}

#Citation-Aware Prompt

Answer the question using the provided sources. Cite your
sources using [Source N] notation.

Sources:
[Source 1] {chunk_1}
[Source 2] {chunk_2}
[Source 3] {chunk_3}

Question: {user_question}

Provide your answer with citations:

#Structured Output Prompt

Based on the following context, answer the question.
Respond in JSON format with:
- "answer": your answer
- "confidence": "high", "medium", or "low"
- "sources": list of source document names used
- "reasoning": brief explanation of how you derived the answer

Context: {retrieved_chunks}
Question: {user_question}

\text{Effective context} = \sum_{i=1}^{k} w_i \cdot \text{relevance}(\text{chunk}_i, \text{query})

#The Lost in the Middle Problem

Place the most relevant chunk first
Place the second most relevant chunk last
Place less relevant chunks in the middle

Alternatively, keep the number of retrieved chunks small (3-5) to avoid the problem entirely.

#Try It Yourself

pythonplayground.py · Pyodide

import math
import re

# Build a complete mini RAG pipeline in Python

# === CHUNKING (simplified: each doc is one chunk) ===
chunks = [{"id": d["id"], "text": d["text"], "metadata": {"title": d["title"]}} for d in documents]

def cosine_sim(a, b):
  dot = sum(x * y for x, y in zip(a, b))
  na = math.sqrt(sum(x ** 2 for x in a))
  nb = math.sqrt(sum(x ** 2 for x in b))
  return dot / (na * nb) if na and nb else 0

# Embed all chunks
chunk_embeddings = [{**c, "embedding": simple_embed(c["text"])} for c in chunks]

# === RUN THE PIPELINE ===
query = "What is the refund policy for enterprise customers?"
print("Query:", query)
print()

results = retrieve(query, 2)
print("Retrieved chunks:")
for r in results:
  print(f"  [{r['score']:.3f}] {r['metadata']['title']}: {r['text'][:80]}...")
print()

prompt = build_rag_prompt(query, results)
print("Final prompt for LLM:")
print(prompt)
print()

Tests · Verify retrieval returns the refund and enterprise documents for the example query. Verify the prompt includes both retrieved chunks with source citations.

#Key Takeaways

RAG is a multi-stage pipeline where failures cascade. Poor chunking leads to bad embeddings, bad embeddings lead to irrelevant retrieval, and irrelevant retrieval leads to hallucinated answers; quality at each stage matters
Document ingestion is the most underestimated stage. Parsing PDFs, cleaning HTML, handling tables, and extracting metadata correctly determines the ceiling of your entire RAG system's quality
Prompt templates must explicitly ground the LLM in retrieved context. Instruct the model to only use provided context, cite sources, and admit when the context does not contain the answer
The retrieval-generation interface is where most RAG systems fail. Even with perfect retrieval, poorly formatted context injection or LLMs that ignore context can produce wrong answers

#Quick Check

Quick Check1 / 4

In a RAG pipeline, what is the most common root cause of poor answer quality?

The RAG Pipeline

#The Full RAG Pipeline: End to End

#Step 1: User Question

#Step 2: Embed the Query

#Step 3: Vector Search

#Step 4: Retrieve Top-K Chunks

#Step 5: Rerank Results

#Step 6: Assemble the Prompt

#Step 7: LLM Generates the Answer

#Step 8: Return with Citations

#Phase 1: Offline Ingestion

#Stage 1: Document Loading

#Stage 2: Text Cleaning & Preprocessing

#Stage 3: Chunking

#Stage 4: Embedding

#Stage 5: Indexing & Storage

#Phase 2: Online Query

#Stage 6: Query Processing

#Stage 7: Query Embedding

#Stage 8: Retrieval

#Stage 9: Post-Retrieval Processing

#Stage 10: Prompt Construction

#Stage 11: Generation

#The Cascade Effect

#Prompt Engineering for RAG

#Basic RAG Prompt

#Citation-Aware Prompt

#Structured Output Prompt

#The Lost in the Middle Problem

#Try It Yourself

#Key Takeaways

#Quick Check

Deep Dive

Related Lessons

Your Reflection

Continue Learning

Discussion

The RAG Pipeline

#The Full RAG Pipeline: End to End

#Step 1: User Question

#Step 2: Embed the Query

#Step 3: Vector Search

#Step 4: Retrieve Top-K Chunks

#Step 5: Rerank Results

#Step 6: Assemble the Prompt

#Step 7: LLM Generates the Answer

#Step 8: Return with Citations

#Phase 1: Offline Ingestion

#Stage 1: Document Loading

#Stage 2: Text Cleaning & Preprocessing

#Stage 3: Chunking

#Stage 4: Embedding

#Stage 5: Indexing & Storage

#Phase 2: Online Query

#Stage 6: Query Processing

#Stage 7: Query Embedding

#Stage 8: Retrieval

#Stage 9: Post-Retrieval Processing

#Stage 10: Prompt Construction

#Stage 11: Generation

#The Cascade Effect

#Prompt Engineering for RAG

#Basic RAG Prompt

#Citation-Aware Prompt

#Structured Output Prompt

#The Lost in the Middle Problem

#Try It Yourself

#Key Takeaways

#Quick Check

Deep Dive

Related Lessons

Your Reflection

Continue Learning

Discussion