In 2020, OpenAI's Kaplan paper found that LLM loss falls as a clean power law in model size, data, and compute — for six orders of magnitude. That single empirical regularity is what convinced every lab to spend hundreds of millions on bigger models. Now in 2026, the pretraining curve is flattening — and o1, o3, DeepSeek R1, and Claude Opus 4 are betting on a completely new scaling axis: spend the compute at inference time, not training time.
Learning Objectives
After this lesson, you will be able to:
State the Kaplan power-law relationship between test loss and the three knobs (compute, parameters, data) — and why the exponents are nearly constant across orders of magnitude
Apply the Chinchilla compute-optimal rule (D ≈ 20N tokens) to decide how to split a fixed compute budget between model size and dataset size — and explain why GPT-3 was undertrained
Describe what 'emergent abilities' means, the Wei 2022 evidence, and the Schaeffer 2023 'mirage' counterargument — and pick a side defensibly
Distinguish three independent scaling axes — pretraining scale, post-training scale, and inference-time scale (o1/o3/R1) — and explain why the data wall is forcing labs to lean on the latter two
Don't worry if "scaling laws" sounds like distant frontier-lab business — the Chinchilla rule alone (D ≈ 20N) is the single most actionable fact for anyone training a transformer in 2026.
#Why Bigger Is Better: and What "Bigger" Should Mean
The big finding: across six or seven orders of magnitude, the loss falls as a power law in each of the three knobs.
L(N,D,C)≈L∞+NαNa+DαDb+CαCc
The key empirical claim from Kaplan: whichever resource is the bottleneck is the one that limits performance, and the relationship is a smooth power law all the way down. If you have unlimited compute but fixed data, loss flattens at the data-limited floor. Same for unlimited data, fixed parameters.
Kaplan's original paper concluded you should grow N and D together but underweighted D. Hoffmann 2022 retrained hundreds of small models with corrected methodology and produced the canonical 2026 rule:
Compute-optimal: D≈20⋅N tokens for N parameters
Drag along the compute-optimal frontier to see how model size and token count trade off at a fixed budget.
Loading visualization...
The companion FLOPs estimate gives you a direct compute number for any (N, D) pair:
F≈6⋅N⋅D(FLOPs for forward + backward)
What Do You Think?
You have a fixed compute budget of 6e22 FLOPs. Spend it on (A) a 175B model trained on 60B tokens (GPT-3-style: parameter-heavy, data-light) or (B) a 70B model trained on 140B tokens (Chinchilla-shaped at the same total FLOPs). Which gives lower test loss?
The answer is B. Chinchilla's empirical demonstration was that at the same FLOP budget, a smaller-but-more-trained model beats a larger-but-undertrained one. GPT-3 was the original cautionary tale — at 175B with only 300B tokens, it was sitting on capacity it never used. Chinchilla 70B (1.4T tokens) at the same budget outperformed it on most benchmarks.
#Emergent Abilities: Phase Transition or Optical Illusion?
In 2022, Wei et al. published "Emergent Abilities of Large Language Models", claiming many tasks show a phase transition: random-baseline performance up to some scale threshold, then sudden competence. Examples included multi-step arithmetic, IPA transliteration, and several BIG-bench tasks.
A 2026 view: scaling has fragmented into three independent axes, each with its own economics.
Axis
Scale lever
Example
Cost shape
Pretraining scale
params, tokens, compute
GPT-3 → GPT-4 → Llama 3 405B
Massive upfront; amortized across all queries
Post-training scale
RLHF/DPO data, instruction tuning, Constitutional AI
Claude 3.5 → Claude 4
Medium upfront; quality gate before deployment
Inference-time scale
CoT length, tree search, RL roll-outs at decode time
o1, o3, DeepSeek-R1
Per-query cost; scales with problem hardness
Total inference cost per query=Nactive⋅Ddecoded⋅2FLOPs per query
The 2024-2025 surprise was that inference scaling has its own scaling law. OpenAI's o1 blog showed log-linear improvement on AIME math problems as you increase test-time compute — a clean power law in compute-per-query, paralleling the pretraining law in compute-per-token.
Llama 3 405B was trained on 15T tokens. Web-scale text is roughly 50T-200T quality tokens depending on filtering aggression. We are running out of high-quality public text within 1-2 more model generations. Three responses being explored in 2026:
Synthetic data generation — use a strong model to produce training data for the next-gen model. Phi-3, Llama 3 distillation, Qwen-chat-of-thought all rely on this. Risk: model collapse if synthetic data drifts too far from real distribution.
Multimodal corpora — image-text-video-audio expand the effective data supply by orders of magnitude. Gemini, GPT-4V, Llama 3.2 Vision all train on multimodal data.
Repeated epochs with data curation — re-train on the same tokens with different orderings and filtering. Llama 3 used multiple epochs of high-quality data; the gains are smaller than fresh tokens but non-zero.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Tests · Verify the chinchilla rule produces D = 20*N tokens. Verify FLOPs = 6*N*D. Verify GPT-3 ratio is significantly less than 1.
Scaling laws are power laws in three variables — parameters, data, compute. Kaplan 2020 showed test loss falls as L ≈ L_∞ + a/N^α + b/D^β + c/C^γ across 8 orders of magnitude; the exponents are remarkably stable, which makes the relationship predictive
Chinchilla's rule (D ≈ 20N tokens) is the single most actionable scaling fact. Hoffmann 2022 showed compute-optimal training keeps model size and data in fixed proportion; GPT-3 was 7x undertrained by this measure, and every frontier model since has been Chinchilla-shaped or aggressively past-Chinchilla
Emergent abilities are real but rarer than the original catalog suggested. Wei 2022 and Schaeffer 2023 are both right: some capabilities really do appear discontinuously, but most "emergence" is binary-metric artifact of underlying smooth improvement
The data wall is forcing labs onto three new axes: synthetic data, multimodal corpora, and inference-time scaling — frontier pretraining is approaching the supply limit of high-quality public text within 1-2 generations; o1/o3/R1 represent a real paradigm shift toward spending compute at decode time instead of train time
Three scaling axes are now independent: pretraining, post-training, inference. Each has its own scaling law, its own cost shape, and its own deployment economics; the 2026 ML lead has to decide all three separately
You have a fixed FLOP budget. Chinchilla says you should spend it where?
Scaling laws tell you where to invest compute. The next lesson covers what to do at decoding time once your trained model has to actually generate text — greedy, beam, top-k, top-p, temperature, and the speculative-decoding trick that's quietly cut serving costs in half.