Neural networks cannot read text. They read numbers. Tokenization is the translation layer — the reason ChatGPT charges by token, the reason GPT-4 cannot count the r's in "strawberry", and the reason Korean and Hindi cost 2–4× more than English on the same API. Every model you use — Claude Sonnet 4.6, GPT-4o, Llama 3.3 — starts with this step.
Learning Objectives
After this lesson, you will be able to:
Understand why neural networks need numbers, not words, and how tokenization bridges the gap
Walk through the Byte-Pair Encoding (BPE) algorithm step by step
Compare three tokenization methods: BPE, WordPiece, and SentencePiece
See how vocabulary size (small vs. large) changes what the model can learn
Understand why splitting words into sub-pieces became the standard in modern LLMs
Spot common tokenization mistakes that quietly hurt model quality
Why not just split on spaces and assign each word a number? Three reasons:
Vocabulary explosion. English has over 170,000 words in common use. Add misspellings, proper nouns, code, URLs, and multilingual text, and you need millions of entries. Each entry is a row in the embedding matrix -- millions of rows means billions of parameters just for embeddings.
Unknown words. Any word not in your vocabulary becomes an <UNK> token, losing all meaning. "Transformerization" would be unknown to a word-level tokenizer.
Morphology blindness. "run," "runs," "running," and "runner" share a root but are four separate entries. The model must learn their relationship from scratch, with no structural hint that they are related.
Subword tokenization solves all three problems by splitting text into pieces that balance frequency and coverage.
Split every word in your training corpus into individual characters. The word "lower" becomes ['l', 'o', 'w', 'e', 'r']. Add a special end-of-word marker (often </w> or _) so the model knows where words end. Your initial vocabulary is just the set of all characters that appear in the corpus -- typically 256 bytes for a byte-level BPE.
Scan the entire corpus and count how often each pair of adjacent tokens appears. If "th" appears 1,000 times and "he" appears 900 times, those are the top pairs. This counting happens across all words, weighted by word frequency.
Take the pair with the highest count and merge it into a single new token. If "t" + "h" is the most frequent pair, create the token "th" and add it to the vocabulary. Replace all occurrences of "t" + "h" in the corpus with "th". The word "the" goes from ['t', 'h', 'e'] to ['th', 'e'].
Type any text above and watch BPE split it into tokens. Try these experiments:
Type a common word like "the" -- it should be a single token
Type an unusual word like "defenestration" -- watch it split into subword pieces
Type code like console.log("hello") -- notice how code tokens differ from English
Type a word in another language -- see how the tokenizer handles unfamiliar scripts
Try it! Before reading on, grab any sentence and try to guess where the BPE tokenizer would split it. Would "unhappiness" be one token or three? What about "ChatGPT"? Think about it, then scroll up to the interactive tokenizer and check your guesses.
Start with the raw text you want to tokenize: "lowest". This is just a string of characters -- the model cannot process it directly. We need to convert it to a sequence of integer token IDs from our learned vocabulary.
Break the word into individual characters: ['l', 'o', 'w', 'e', 's', 't']. Add a special end-of-word marker. This is our starting vocabulary -- every character that appears in the training corpus. For byte-level BPE, the initial vocabulary is the 256 possible byte values.
Scan the training corpus and count all adjacent character pairs. The pair ('e', 's') appears most frequently across words like "lowest," "newest," "widest." This pair is selected for merging. The count is weighted by how often each word appears.
If your training corpus is entirely English text, what happens when the model encounters Chinese characters at inference time?
Modern LLMs use byte-level BPE (used by GPT-2, GPT-4, LLaMA). Instead of starting with characters, they start with raw bytes (0-255). This means any input -- any language, any encoding, any binary data -- can be represented as a sequence of byte tokens. No <UNK> tokens ever. Rare scripts just require more bytes per character.
Talking about "bytes" in the abstract is one thing. Watching them on paper is what makes the algorithm click. The next few paragraphs take two simple strings, encode them to UTF-8, and walk through one merge step of a tiny BPE trainer so you can see exactly what a tokenizer sees.
Start with the word café. In ASCII the first three letters c, a, f each take a single byte: 0x63, 0x61, 0x66. But é is U+00E9, which lives outside the ASCII range, so UTF-8 spends two bytes on it: 0xC3 0xA9. The whole word is the 5-byte sequence [0x63, 0x61, 0x66, 0xC3, 0xA9] — one more byte than the four user-visible characters suggest.
The cost climbs sharply for emoji. The llama emoji 🦙 is U+1F999, well beyond the basic multilingual plane, so UTF-8 spends four bytes on a single glyph: [0xF0, 0x9F, 0xA6, 0x99]. A byte-level BPE tokenizer therefore starts with four tokens for one emoji before any merges happen, which is exactly why pre-merge sequence lengths are so different across scripts.
Now run a single training pass on a corpus of three words: cat (count 3), café (count 2), car (count 4). Treating each byte as a starting token, the initial split is:
cat → c a t × 3 = pairs (c,a) × 3, (a,t) × 3
café → c a f 0xC3 0xA9 × 2 = pairs (c,a) × 2, (a,f) × 2, (f,0xC3) × 2, (0xC3,0xA9) × 2
car → c a r × 4 = pairs (c,a) × 4, (a,r) × 4
Counting all pairs across the corpus: (c,a) appears 3 + 2 + 4 = 9 times — the most-frequent pair by a wide margin. BPE merges it into a new token ca. After the first merge the words become ca t, ca f 0xC3 0xA9, and ca r. The next round counts pairs again, sees (ca, r) × 4 leading the pack, and merges that to produce car as a single token. Three or four merges in, the most common surface forms of this toy corpus are already single tokens, while the rare bytes of é stay separate. That is the whole BPE algorithm; everything else is scale.
This is why GPT-4's cl100k_base is so efficient on English and on common code. "Hello, world!" lands in just 4 tokens (Hello, ,, world, !) because each of those substrings was frequent enough during training to win merges. Strings like def , return , and, and </div> are similarly compressed into single tokens, which is why GPT-4 reads Python and HTML almost as densely as prose. By contrast, Swahili text — which barely appeared in the merge-training corpus — averages roughly 3–4× more tokens per word than English, because most subword merges never fired and you end up paying near the byte rate. Hindi and Burmese can be 5–6× worse. The tokenizer is the same; the merge table is what is unbalanced.
The playground below lets you run real tiktoken (via Pyodide) on your own strings and see the fertility difference across languages firsthand.
Loading visualization...
Quick check
A multilingual chat product charges a flat per-message price. Why might a Swahili user effectively get 3× less context than an English user with the same byte-pair tokenizer?
WordPiece is similar to BPE but uses a different merge criterion. Instead of merging the most frequent pair, it merges the pair that maximizes the likelihood of the training data:
score(a,b)=freq(a)⋅freq(b)freq(ab)
WordPiece tends to produce slightly different vocabularies than BPE. BERT uses WordPiece with a vocabulary of 30,522 tokens. Subword pieces that are continuations of a word are prefixed with ## -- so "playing" might become ['play', '##ing'].
SentencePiece treats the input as a raw stream of Unicode characters (including spaces) rather than pre-tokenized words. It does not require language-specific pre-processing like splitting on whitespace. The space character is treated like any other character, often represented as ▁ (a special underscore). This makes it truly language-agnostic -- equally effective for English, Japanese, Thai (which has no spaces), and mixed-language text.
A model with vocabulary size 256 (just bytes) vs. one with vocabulary size 100,000 -- which needs fewer tokens to represent the same text?
Vocabulary size creates a fundamental tradeoff:
Small Vocabulary (< 10K)
Large Vocabulary (> 100K)
More tokens per sentence
Fewer tokens per sentence
Smaller embedding matrix
Larger embedding matrix
Better on rare/novel words
May waste capacity on rare tokens
Slower inference (more steps)
Faster inference (fewer steps)
Learns subword patterns well
Each token is more semantically complete
Modern LLMs have converged on roughly 100K-200K tokens as the sweet spot. GPT-4 uses ~100K (cl100k_base), GPT-4o uses ~200K (o200k_base), Llama 1 and 2 used 32K but Llama 3 and 3.1+ jumped to a 128K tiktoken-style vocabulary, Mistral models use 32K-128K, and Claude uses ~100K. The trend is toward larger vocabularies because the embedding matrix cost is small compared to the attention and FFN parameters in a multi-billion-parameter model, and larger vocabs reduce sequence length — directly cutting inference cost.
Tokenization is not just an implementation detail -- it directly affects model behavior:
Cost. API pricing is per token. Inefficient tokenization means you pay more for the same content. Code and non-English text often cost 2-3x more in tokens than English prose.
Context window. A 128K token limit holds different amounts of text depending on the tokenizer. Dense technical text with rare terms uses more tokens than simple English.
Arithmetic failures. LLMs struggle with arithmetic partly because numbers are tokenized unpredictably. "123456" might be one token, two tokens, or three tokens depending on the number and the tokenizer. The model never sees individual digits consistently.
Multilingual bias. Tokenizers trained primarily on English text produce more tokens for non-English text, making those languages more expensive and reducing effective context length.
BPE is iterative compression: the algorithm merges the most frequent adjacent byte pair into a new token, repeated until the vocabulary size target is reached — modern LLMs have converged on roughly 100K–200K token vocabularies
Tokenization is language-unequal: Latin scripts tokenize at roughly 1 token per word; CJK languages often require 2–3 tokens per character, making prompts 2–4x more expensive for the same semantic content
Vocabulary size is a Goldilocks problem: too small means poor compression and long sequences; too large means rare tokens have sparse training signal and poor representations
Use the real tokenizer, not approximations: tiktoken (OpenAI) and Anthropic's tokenizer API give exact counts; off-by-one estimates cause silent truncation in production
Prompt caching operates at token boundaries: understanding tokenization lets you structure system prompts for maximum cache hit rates, directly reducing API cost at scale
Now that we can convert text to tokens, the next question is: how does a model decide which tokens matter most? That is the attention mechanism -- the ability to focus on the relevant parts of the input.
Go back to Step 2 and count pairs again (now including the newly merged token). The pair "th" + "e" might now be the most frequent, creating the token "the". Repeat until you reach your desired vocabulary size -- typically 32,000 to 128,000 tokens for modern LLMs.
After thousands of merges, you have a vocabulary that includes single characters (for handling any input), common subwords ("ing", "tion", "pre"), full common words ("the", "and", "for"), and even multi-word fragments. Common words are single tokens; rare words get split into known subword pieces.
Merge 'e' + 's' into the new token 'es' everywhere in the corpus. The word now becomes ['l', 'o', 'w', 'es', 't']. The token 'es' is added to the vocabulary with a new ID. The merge rule ('e', 's') -> 'es' is recorded in the merge table.
Continue finding and merging the most frequent pairs. Next, 'es' + 't' might merge into 'est'. Then 'l' + 'o' into 'lo', and 'lo' + 'w' into 'low'. Each merge creates a new token and is added to the ordered merge table. Repeat until the vocabulary reaches the target size (32K-128K tokens).
After thousands of merges, the vocabulary contains: all individual characters/bytes (base tokens), common subwords like 'est', 'ing', 'tion', full common words like 'the', 'and', 'low', and the special tokens ([PAD], [UNK], [CLS]). Our word "lowest" is now tokenized as ['low', 'est'] -- two meaningful subword tokens.
To tokenize new text at inference time, apply the learned merge rules in order. Start with characters, then apply merge 1, merge 2, merge 3, and so on. The word "lowest" becomes ['low', 'est'] which maps to token IDs [4521, 891]. These integers are what the model actually sees. The merge table is deterministic -- the same text always produces the same tokens.