The #1 reason production RAG fails isn't the embedding model — it's chunking. Split a contract wrong and you retrieve half a paragraph that contradicts the other half. Split code wrong and you separate imports from the functions that need them. Glean's engineers obsess over chunking; Anthropic's Contextual Retrieval paper proved it cut retrieval errors 49%. Here's how to chunk like the pros.
Learning Objectives
After this lesson, you will be able to:
Understand why you need to cut documents into smaller pieces before the AI can search them, and how the size of those pieces changes how good the search results are
Compare four ways to split documents -- by character count, by sentence, by topic change, and recursively -- and know when each one works best
Apply semantic chunking to detect topic boundaries using embedding similarity, and use parent-child chunking to combine precise retrieval with complete context delivery
Design a chunking strategy that balances finding the right passage (precision), keeping enough context (completeness), and producing good embeddings -- and know how to validate chunk quality empirically
Chunking is one of those behind-the-scenes steps that nobody talks about but everybody gets wrong the first time. Do not worry -- by the end of this lesson, you will know more about it than most working engineers.
This is the most underrated step in the entire RAG pipeline. Teams spend weeks tuning their LLM prompts while ignoring chunking -- but chunking has a bigger impact on retrieval quality than almost any other choice.
Try it: Compare chunking strategies side by sideInteractive
Loading visualization...
Try it! Take any Wikipedia article and try splitting it three ways: every 100 words (fixed-size), by paragraph, and by section heading. Then ask yourself a specific question about the article and see which split gives you the most useful single piece. You will see the quality difference immediately.
Before documents can be embedded and stored in a vector database, they must be split into chunks. The embedding model converts each chunk into a single vector. The vector captures the semantic content of that chunk. If the chunk is poorly constructed -- too broad, too narrow, or splitting a concept mid-sentence -- the embedding will be poor, and retrieval will suffer.
You have a 10-page technical document and want to find the answer to a specific question. Which chunking approach will give the best retrieval results?
The answer depends on the nature of the questions and the structure of the document, but paragraph-level chunking (150-300 tokens) is the most common sweet spot. Here is why:
When chunks are too large (full pages, 1000+ tokens):
The embedding averages too many concepts into one vector -- a page about both "authentication" and "database schemas" gets a blurry embedding that is mediocre at matching either topic
A raw document arrives -- it might be a 50-page PDF, a lengthy web page, or a technical manual. It is far too long to embed as a single vector (the meaning would be hopelessly diluted), so it must be broken into smaller, meaningful pieces.
The system selects a chunking strategy based on the document type. Fixed-size splits at every N tokens regardless of content. Sentence-based splits at sentence boundaries. Semantic splits where the topic changes. Each strategy has different trade-offs between simplicity and quality.
The document is divided into chunks -- typically 200 to 500 tokens each. Each chunk should ideally contain a single coherent idea or topic. A 10,000-token document might yield 20 to 50 chunks depending on chunk size and strategy.
Adjacent chunks share a region of overlapping text (typically 10-20% of the chunk size). This overlap ensures that information sitting right at a chunk boundary is not lost -- it appears in full in at least one chunk, preventing the "split mid-sentence" problem.
Every chunk is passed through the embedding model to produce a dense vector. The vector captures the semantic meaning of that specific chunk. Good chunks produce focused, precise vectors. Bad chunks (mixing multiple topics) produce blurry, unfocused vectors.
Each chunk's vector, original text, and metadata (source document, section header, page number, chunk index) are stored together in the vector database. The knowledge base is now searchable -- any future query can find the most relevant chunks by comparing vector similarity.
The simplest approach: split text into equal-sized chunks with optional overlap.
pythonreference · read-only
1
2
3
4
5
6
7
def fixed_size_chunk(text, chunk_size=500, overlap=50):
tokens = tokenize(text)
chunks = []
for i in range(0, len(tokens), chunk_size - overlap):
chunk = tokens[i:i + chunk_size]
chunks.append(detokenize(chunk))
return chunks
Overlap ensures that information at chunk boundaries is not lost. If a key sentence spans the boundary between two chunks, the overlap ensures it appears (at least partially) in both.
Pros: Simple, predictable, fast.
Cons: Splits mid-sentence, mid-paragraph, even mid-word. No awareness of document structure. A chunk might contain the end of one topic and the beginning of another.
Split on sentence boundaries, then group sentences to reach a target size.
pythonreference · read-only
1
2
3
4
5
6
7
8
9
10
11
12
13
14
def sentence_chunk(text, max_tokens=500):
sentences = split_into_sentences(text) # spaCy, NLTK, or regex
chunks, current = [], []
current_len = 0
for sent in sentences:
sent_len = count_tokens(sent)
if current_len + sent_len > max_tokens and current:
chunks.append(' '.join(current))
current, current_len = [], 0
current.append(sent)
current_len += sent_len
if current:
chunks.append(' '.join(current))
return chunks
Pros: Never splits mid-sentence. Preserves complete thoughts.
Cons: Sentences vary widely in length. Still splits mid-paragraph, potentially separating related ideas. Requires a good sentence detector (harder than it sounds for technical text with abbreviations, code, and tables).
Split by the largest structural unit first, then recursively split if chunks are too large. This is the strategy used by LangChain's RecursiveCharacterTextSplitter.
The hierarchy of separators (from preferred to fallback):
Split by sections (double newlines, headers)
Split by paragraphs (single newlines)
Split by sentences (periods, question marks)
Split by words (spaces)
Split by characters (last resort)
pythonrunnable cell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# LangChain RecursiveCharacterTextSplitter approach
separators = ["\n\n", "\n", ". ", " ", ""]
def recursive_chunk(text, chunk_size, separators):
if len(text) <= chunk_size:
return [text]
for sep in separators:
splits = text.split(sep)
if len(splits) > 1:
chunks = []
current = ""
for s in splits:
if len(current) + len(s) + len(sep) <= chunk_size:
current += (sep + s if current else s)
else:
if current:
chunks.append(current)
current = s
if current:
chunks.append(current)
# Recursively split any chunks that are still too big
result = []
for c in chunks:
result.extend(recursive_chunk(c, chunk_size, separators[1:]))
return result
# Fallback: hard split
return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]
Pros: Respects document structure. Adapts to different content types. Most commonly recommended approach.
Cons: Implementation complexity. Results depend on separator choices. Does not understand semantic boundaries (two paragraphs about different topics still get merged if they fit in one chunk).
The most sophisticated approach: use an embedding model to detect where the topic changes, and split at those boundaries.
\text{split at } i \text{ if } \text{sim}(\text{embed}(s_i), \text{embed}(s_{i+1})) < \theta
Pros: Creates semantically coherent chunks. Each chunk contains a single topic or idea. Best retrieval quality when topics change frequently within documents.
Cons: Requires running the embedding model during chunking (expensive at scale). Sensitive to threshold selection. Can create wildly varying chunk sizes.
This gives the embedding model (and later, the LLM) crucial context about where this chunk comes from. The chunk "Revenue increased 15% year-over-year" becomes much more informative when prefixed with the document and section title.
Tests · Verify fixed chunking produces more chunks than paragraph chunking. Verify section chunking preserves headers with their content.
#Contextual Retrieval: Adding Context to Every Chunk
Instead of embedding raw chunks, Contextual Retrieval generates a 50-100 word context for each chunk using the full document as input:
pythonrunnable cell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
CONTEXTUAL_RETRIEVAL_PROMPT = """
<document>
{full_document}
</document>
Here is a chunk from the document:
<chunk>
{chunk_content}
</chunk>
Write a short context (2-3 sentences) that situates this chunk within the full document.
Focus on what this chunk is about and why it is relevant. Be concise.
"""
def add_context_to_chunk(chunk: str, full_doc: str, llm) -> str:
context = llm.generate(
CONTEXTUAL_RETRIEVAL_PROMPT.format(
full_document=full_doc,
chunk_content=chunk
)
)
return f"{context}\n\n{chunk}"
Result: A bare chunk like "The patient was started on 10mg daily" becomes:
This chunk is from a case report about a 45-year-old patient with Type 2 diabetes. It describes the initial metformin dosing protocol decided at the endocrinology clinic visit.
The patient was started on 10mg daily.
The embedding now captures the full semantic context. Anthropic reported 49% fewer retrieval failures with Contextual Retrieval versus naive chunking.
When to use it: Contextual Retrieval adds LLM inference cost per chunk (paid once at ingestion). The payoff is large for document-heavy RAG systems where chunks lose meaning without their surrounding context — medical records, legal contracts, technical manuals.
Chunk size is the most impactful RAG hyperparameter. Too small and chunks lack context; too large and they contain irrelevant information that dilutes the signal; 200-500 tokens is a common starting range
Overlap between chunks prevents losing information at boundaries. A 10-20% overlap ensures that sentences split across chunk boundaries are still captured in at least one complete chunk
Semantic chunking respects document structure. Instead of fixed-size splits, chunking by paragraphs, sections, or semantic boundaries produces more coherent chunks that embed and retrieve better
The right strategy depends on your content type. Code needs syntax-aware chunking, legal documents need clause-level splits, conversations need turn-based segmentation; one size does not fit all
Why does chunk size significantly affect RAG quality?
Now you understand how to prepare documents for RAG. Next up: The RAG Pipeline -- putting everything together into a complete system, from document ingestion to answer generation.