GPT-4 will not tell you what it was trained on. Llama 3 will: 15T tokens, a heroically filtered mix of web text, code, and "high-quality" sources. FineWeb, released by HuggingFace in 2024, made the full 15T-token open-data recipe public — and triggered an arms race over data curation that is now the single biggest lever in pretraining outside compute itself. This lesson is the unglamorous 80% of the story that scaling-laws papers gloss over: where the tokens come from, how they are cleaned, how duplicates are killed, and why the tokenizer choices you make on day one shape every dollar you pay at inference.
Learning Objectives
After this lesson, you will be able to:
Trace the lineage of open pretraining corpora — Common Crawl → C4 → The Pile → RefinedWeb → RedPajama → Dolma → FineWeb / FineWeb-Edu / DCLM — and identify the curation move that distinguishes each
Walk through a modern pretraining cleaning pipeline: language ID, boilerplate stripping, heuristic filters, KenLM perplexity filters, classifier filters, PII redaction, and the trade-offs of toxicity filtering
Apply MinHash + LSH near-duplicate detection on a small document set and explain why dedup is the single biggest lever in data-quality work
Reconcile Chinchilla (D ≈ 20N) with the modern over-training regime, and design a data-mixing schedule that weights code, math, and web text deliberately
Make defensible decisions on vocabulary size, byte-level fallback, tokenizer training corpus, and document packing — and explain tokenizer fertility's equity implications
Don't be misled by the headlines about model architectures — the 2024-2025 frontier-lab consensus is that data work matters more than architectural innovation for the next 1-2 generations. The Chinchilla rule tells you how much data; this lesson is the missing piece on which data and why.
Common Crawl publishes a monthly web snapshot as WARC files (Web ARChive). Each snapshot is around 200-400 TB compressed, with billions of pages. It contains HTML, JavaScript-rendered text, PDFs, code, structured data, and a long tail of garbage. The format records the URL, fetch timestamp, HTTP headers, and full response body. Two derived formats — WET (extracted plaintext) and WAT (metadata only) — exist for convenience but most serious pipelines re-extract from raw WARC because the default WET extraction is poor.
Modern training mixes include 5-20% code. The dominant open code corpus is The Stack (BigCode, 2022, 3 TB) and its successor The Stack v2 (2024, 67 TB across 600+ programming languages). These are crawled from GitHub with permissive-license filtering and PII redaction. Llama 3 disclosed 17.5% code in its mix; Claude and GPT-4 are believed to be in a similar 10-20% range. Heavy code in the mix improves not just code generation but general reasoning — a finding now well-replicated since Codex 2021.
Below is the canonical 2026 cleaning order — roughly the order FineWeb, Dolma, and DCLM all use, with minor variations. Skip any stage and you pay for it twice: in training compute and in downstream eval.
Use fastText (Facebook's lid.176 model) or CLD3 (Google) to label every document with a language and a confidence score. Keep documents above ~0.65 confidence for your target language(s). FineWeb keeps English-only at lid > 0.65; multilingual sets (mC4, OSCAR) keep a per-language threshold.
Why this matters: a 5% leak of low-confidence French into your "English" corpus quietly degrades your tokenizer fertility, distorts your downstream evals, and confuses your data-mix ratios.
Cheap rule-based filters that strip obvious garbage before anything expensive:
Filter
Threshold
What it catches
Length
<50 words OR >100K words
Stubs and dumps
Mean word length
<3 or >10 chars
Garbled OCR, machine-generated nonsense
Punctuation ratio
<0.12
List-only pages, code dumps
Symbol-to-word ratio
>0.1
Math/code disguised as prose, currency dumps
Ellipsis lines
>30% of lines end in "..."
Search-result excerpts
Repetition
top-2gram >0.2 of doc
Repeating boilerplate
Bullet/numbered lines
>90%
Stack Overflow listings, recipe sites
Stopword ratio
<2 of {the, of, and, a, in, to}
These are the Gopher filters (DeepMind 2021), refined by RedPajama-V2's quality signals, and now the de-facto standard. RedPajama-V2 ships 40+ quality signals per document so downstream users can pick their own thresholds.
Train a 5-gram language model (KenLM is the standard) on a high-quality reference corpus — typically Wikipedia. Score every web document by its perplexity under that model. Keep documents in the middle perplexity band: very low perplexity is template/boilerplate, very high is gibberish, the middle is real prose.
This is the CCNet filter (Wenzek et al. 2020) and it remains the standard preprocessing step for serious web data work. In our MathPlayground below, we will build a tiny n-gram version of this.
The 2024 quality-filter breakthrough: train a small classifier (DistilBERT-scale, sometimes a 1B Llama distillation) to predict "is this educational" on a few hundred thousand documents rated by a strong model. Then run it across the full corpus and keep the top X%.
FineWeb-Edu used Llama 3 70B to rate 500K documents on a 0-5 educational-value scale, trained a small classifier on those ratings, and kept the top ~30% of FineWeb. The result: a 1.3T-token subset that beats raw 15T FineWeb at small training budgets by 5-10 MMLU points. This is the single most impactful 2024 data-curation result.
DCLM-baseline-1.0 used a similar move with a slightly different rating axis. The pattern is now standard.
Run a PII detector — Microsoft Presidio, spaCy + custom regex, or a fine-tuned NER model — to redact emails, phone numbers, SSNs, credit cards, and addresses. Dolma redacts all of these; The Stack v2 redacts email addresses in commit messages. This is partly an ethical concern, partly a memorization-attack defense (Carlini et al. 2021 showed LLMs memorize and emit PII from training data verbatim).
The hardest stage to get right. Aggressive toxicity filtering (e.g., the C4 blocklist applied verbatim) disproportionately removes text about minority groups, AAVE, and LGBTQ+ topics — Dodge et al. 2021's "Documenting Large Webtext Corpora" measured this empirically. Most modern pipelines use a light toxicity filter (drop documents with >X% slur density) and rely on RLHF and safety post-training for the rest. FineWeb does no toxicity filtering by default and leaves it to downstream users.
Why dedup matters: a duplicate document trained 10 times is mostly wasted compute (the gradient updates collapse), inflates memorization risk (Carlini et al. 2022 showed memorization scales with duplication frequency), and skews downstream eval (because eval sets leak into training data via duplicates). Dedup is the most-replicated "free win" in pretraining data work.
The real problem: two news articles republished with a one-line attribution change, scraped versions of Wikipedia with different cookie banners stripped, paraphrased clickbait. Three families of solutions:
MinHash + LSH. The field standard. Hash document n-grams ("shingles"), compute a MinHash signature, bucket signatures via Locality-Sensitive Hashing, only do exact Jaccard inside buckets. Sublinear cost.
SimHash (Charikar 2002) — a single 64-bit fingerprint per document; near-duplicates have low Hamming distance. Used by Google for web dedup. Less common for LLM corpora than MinHash but cheaper at very large scale.
Suffix array methods (Lee et al. 2022, "Deduplicating Training Data Makes Language Models Better") — find all substrings of length k that appear more than once across the corpus, delete one copy. Strictly more thorough than MinHash, but harder to scale.
Cross-document. Different documents that are mostly the same. The big category. Catches near-duplicates of news articles, mirrored content, scraped copies.
Intra-document. Repeating phrases within one document. The Gopher repetition filters catch this; suffix array methods catch both at once.
The FineWeb release report documents a clean 20% downstream-score lift from a strong dedup pass (MinHash with 128 hashes, 14-band LSH at ~0.7 Jaccard threshold) over a URL-only dedup baseline. SlimPajama observed similar lifts over RedPajama. The 2025 consensus: a serious dedup pass is the highest-ROI data-work intervention available.
What Do You Think?
You have two 1000-word documents that share 950 words verbatim but the 50 unique words are different (e.g., a news article republished with a different byline and intro paragraph). Will exact-hash dedup catch them as duplicates?
Time to build MinHash + LSH from scratch on a small set of near-duplicate documents. This is the canonical "interview question that's actually how the field works" exercise.
Loading visualization...
What you should see when you run it: docs (0, 1) — the republished article — get a MinHash estimate of ~0.6-0.7 and survive LSH bucketing. Docs (2, 3) — sharing only the boilerplate footer — score around 0.05-0.10 and are correctly not flagged as duplicates. The whole point of LSH is that we computed Jaccard on a handful of candidate pairs, not all 21 — at 100M documents, this is the difference between "tractable on a laptop" and "rent a Spark cluster."
Quick check
A 100M-document corpus. URL-level dedup removes 30% of duplicates. Why bother with MinHash + LSH after that?
KenLM is the production tool, but you can build a tiny n-gram language model in pure numpy and demonstrate the CCNet filter behavior on a handful of documents.
Loading visualization...
The shape of the result: pure repetition gets very low perplexity (the model trivially predicts the next "the"), gibberish gets very high perplexity, real prose that resembles the reference lives in the middle. The CCNet filter keeps the middle band. This is exactly the heuristic that produced CCNet-100 (Wenzek et al. 2020) which became the standard preprocessing for everything downstream.
You read the Chinchilla rule in the scaling-laws lesson: D ≈ 20N is compute-optimal at a fixed training budget. The 2024-2025 evidence layered on top:
Llama 3 broke Chinchilla deliberately. Llama 3 8B was trained on 15T tokens — about 100x past Chinchilla-optimal. Llama 3 70B saw ~6T tokens, ~4x past. The reason: inference economics. Over-trained small models serve cheaper.
DCLM showed data quality can substitute for data quantity. DCLM-baseline at 4T high-quality tokens matches or beats RedPajama at 1.2T raw tokens on downstream eval — even at the same model size. Quality filtering can buy you 2-3x in effective tokens.
The "10x token rule" is the modern over-training heuristic. For a model that will be heavily served at inference, train it on ~10x the Chinchilla-optimal token count. A 7B model: ~1.4T optimal, ~14T in over-trained practice. Llama 3 went further still on 8B because the team had compute headroom and the curve kept moving.
The mix matters as much as the volume. The modern recipe is roughly:
Source
Share (typical)
Why
Web (CC-derived)
50-70%
Volume backbone
Code
5-20%
Reasoning gains, code capability
Math / academic (ArXiv, books)
5-10%
Reasoning, technical capability
Wikipedia / encyclopedic
3-5%
Factual grounding
Books / long-form
5-15%
Long-context coherence
StackExchange / Q&A
1-5%
Instruction-following baseline
Multilingual web
5-20%
Per-language eval
Curriculum scheduling is a 2024-2025 area of active work: train on noisier web data first, switch to higher-quality and code-heavy data in the final ~10% of tokens. Llama 3 disclosed a curriculum-style annealing phase; DeepSeek-V3 documented an explicit phase-shift in the data mix. The signal: late-stage data quality matters disproportionately because the late gradients shape downstream eval most heavily.
Quick check
You're filtering a 50T-token web crawl down to 5T tokens via FineWeb-Edu-style classifier filtering. Same 7B model trained on the 5T filtered set vs the 50T unfiltered set. Which wins on MMLU?
Now we get to the choices that haunt every downstream user of your model. The tokenizer is trained on a slice of your data, and those choices propagate forever.
Bigger vocab → fewer tokens per sentence → fewer model steps per character of text. But: the embedding matrix is vocab_size × d_model, so a 256K vocab at d=8192 is a 2.1B-parameter embedding table — significant for small models, negligible for 70B+.
The 2026 default: byte-level BPE (GPT-2-style) or SentencePiece with byte fallback. Guarantees no <UNK> for any input. The trade-off: rare Unicode characters (emoji, CJK ideographs) require multiple byte tokens.
This is the under-appreciated decision: train your tokenizer on a representative slice of your eventual training data. If your training data is 15% code, train your tokenizer on a 15%-code slice — otherwise you get bad code tokenization that costs you across the whole training run. Llama 3 specifically retrained its tokenizer with more code than Llama 2 to fix this.
This is opinionated: numbers get split into 1-3 digit chunks (which is why GPT-4 fails on long-number arithmetic), contractions get split off, and whitespace is handled explicitly. The choice of pre-tokenization regex propagates directly into model behavior — it is why "12345" and "1234567" tokenize very differently.
Fertility = tokens per word in a target language. The equity story from the tokenization lesson lives here. Llama 3 128K-vocab fertility (very approximate):
Language
Fertility (tokens/word)
English
~1.0
French
~1.2
Spanish
~1.2
Chinese
~1.5 (per character)
Japanese
~1.6 (per character)
Hindi
~2.5-3.0
Arabic
~2.5
Burmese / Khmer
~5-8
That fertility number directly translates into: cost per API call, effective context window, and number of forward passes the model must do for the same semantic content. Building a tokenizer that minimizes mean fertility across your intended user base is an equity decision dressed up as a hyperparameter.
What Do You Think?
Why is FineWeb-Edu's classifier-based filter so effective compared to heuristic + perplexity filters?
The final stage before bytes hit the GPU. Your training data is millions of documents of wildly varying length; your model trains on fixed-length sequences (2K, 4K, 8K, 32K depending on the model). Two choices:
Concatenate many short documents into one fixed-length sequence, separated by an EOS token. The model attends across documents in this scheme — attention is not blocked at document boundaries. Pros: simplicity, efficient GPU utilization. Cons: cross-document attention can teach the model spurious continuations ("end of cookie recipe" → "intro to chess opening theory") and adds noise.
Pack documents into a fixed-length sequence but mask attention so each document only attends to its own tokens. Plus optionally reset position embeddings at document boundaries. Pros: cleaner training signal. Cons: more bookkeeping in the data loader, and many open-source training scripts don't implement it correctly.
The 2024-2025 consensus drift: document-attention masks are worth the implementation effort for serious training runs. Llama 3, DeepSeek-V3, and Qwen 2.5 all use them. Early open-source models (Llama 1/2, Mistral) used EOS-separated packing and accept the noise.
Within a packed sequence, do you reset the position index to 0 at each document boundary? RoPE positional encodings make this cheap and clean. Most modern implementations reset. Some still don't, and the resulting "wraparound" position artifacts can show up as weird in-context-learning behavior near the start of the second document.
The open-data lineage runs C4 → The Pile → RefinedWeb → RedPajama → Dolma → FineWeb → DCLM. Each step is a curation move, and FineWeb's 15T-token release (April 2024) reset the open-source bar against Llama-3-scale closed data
Dedup is the highest-ROI data work. URL dedup is the easy 30% win, MinHash + LSH catches the long tail of near-duplicates, and FineWeb's report documents roughly 20% downstream-score lift from a strong dedup pass alone
Classifier-based quality filtering beats heuristic filtering by a wide margin. FineWeb-Edu's small classifier trained on Llama-3-rated educational value produces a 1.3T-token subset that beats the raw 15T set by 5-10 MMLU points at small budgets
Chinchilla (D ≈ 20N) is the floor; over-trained small models are the new default. Llama 3 8B at 15T tokens (~100x past Chinchilla) reflects inference economics, not a contradiction of the scaling law
Tokenizer choices are equity choices in disguise. A tokenizer trained on English-heavy data charges Hindi, Burmese, and Arabic users 2-5x more in tokens, API cost, and effective context; mixing the tokenizer training corpus to match your target users is the single most impactful equity intervention in pretraining
Why is MinHash combined with LSH (Locality-Sensitive Hashing) instead of just computing exact Jaccard for every document pair?
Now that you know where the tokens come from and how they get cleaned, the next lesson covers what happens to a pretrained model once it can't be made better just by adding more data — the post-training stack: supervised fine-tuning, RLHF, DPO, and Constitutional AI.