Track 04 · Deep Learning · 14 min
The neural networks that ate the world.
From a 1958 Cornell experiment to GPT-4 and AlphaFold — how stacks of matrix multiplications became the most important technology of the decade. Animated, visualized, and runnable, with five interactive demos that make backprop click.
“Deep learning will be able to do everything.”
#The hook
In 2012, Alex Krizhevsky's three-layer CNN (running on a pair of consumer GPUs) cut the ImageNet error rate in half overnight. By 2016 AlphaGo beat Lee Sedol. By 2022 ChatGPT broke into 100M users in two months. By 2024 AlphaFold won the Nobel.
#Why this matters in 2026 — the receipts
The deep learning takeover
The recipe that won everything
95%
ML papers using deep learning
arXiv 2025
12yr
Years from AlexNet to AlphaFold Nobel
2012 → 2024
1T+
Parameters in frontier 2026 models
GPT-4, Claude, Gemini
25K+
GPUs in a frontier training cluster
Meta + xAI 2025
#The four pieces of every neural network
The recipe
A neural network in four ingredients
1. Layers (matrix multiply + nonlinearity)
the building blocky = activation(W * x + b). That's one layer. Stack many.
- Matrix W transforms the input. Bias b shifts it. Nonlinearity (ReLU, GELU) makes it expressive.
- Without the nonlinearity, the whole stack collapses into one giant matrix — useless.
- Modern models stack hundreds of these layers.
2. Loss function
what wrong looks likeA single number that measures how wrong the prediction is.
- Cross-entropy for classification. MSE for regression.
- The loss is the mountain we walk down with gradient descent.
- Pick wrong loss → train wrong objective. The framing matters.
3. Backpropagation
how learning happensThe chain rule, applied carefully through the computation graph, to compute every parameter's gradient.
- Forward pass: input → output, store intermediate values.
- Backward pass: walk the chain rule backward, get a gradient for every parameter.
- PyTorch's autograd does this automatically. You almost never write it by hand.
4. Optimizer
how to stepOnce you have the gradient, decide how big a step to take. Adam is the default in 2026.
- SGD: take a step proportional to the gradient. Momentum: average recent gradients.
- Adam: per-parameter adaptive learning rate. Default for nearly every modern model.
- Learning-rate schedules (warmup, decay) matter as much as the optimizer itself.
17 pp
The error-rate drop AlexNet caused — overnight
In 2012, AlexNet beat the runner-up at the ImageNet competition by 17 percentage points. That single result triggered the deep-learning revolution. Within five years, every major tech company had a research lab. Within ten, the Nobel Prize.
Krizhevsky, Sutskever, Hinton — NeurIPS 2012
Vocabulary
Six deep-learning terms in every paper
Concept
Tensor
An N-dimensional array. Inputs, weights, activations all are tensors.
Like: A spreadsheet that goes 7 layers deep.
e.g. shape (32, 3, 224, 224) — batch of images
Concept
Activation
The nonlinearity between layers — what makes networks expressive.
Like: A neuron deciding to 'fire' or stay quiet.
e.g. ReLU, GELU, SiLU
Concept
Loss function
A scalar measuring how wrong the prediction is.
Like: Distance from the bull's-eye.
e.g. Cross-entropy, MSE
Concept
Backpropagation
The chain rule run backward through the computation graph.
Like: Tracing each weight's blame for the final error.
e.g. torch's .backward() does this
Concept
Optimizer
Algorithm that updates weights using gradients.
Like: How big a step to take down the hill.
e.g. SGD, Adam, AdamW
Concept
Regularization
Tricks that prevent overfit — dropout, weight decay, early stopping.
Like: Forcing the network to not memorize.
e.g. Dropout 0.1 in transformer blocks
#See the pieces — interactive
A neural network drawn without metaphors. Each circle is a unit; each line is a weight. Click neurons to see their activations.
#Backpropagation — the magic deflated
#CNNs — what made vision work
#Train a real net — runnable
# A real (tiny) neural network in PyTorch — runs in your browser via Pyodide
# Note: this needs micropip + pytorch package which Pyodide ships
import numpy as np
# Make a tiny classification dataset
np.random.seed(0)
X = np.random.randn(200, 4).astype(np.float32)
y = (X.sum(axis=1) > 0).astype(np.int64)
# Logistic regression as a 1-layer "neural network" via NumPy
W = np.random.randn(4, 2).astype(np.float32) * 0.1
b = np.zeros(2, dtype=np.float32)
def softmax(z):
z = z - z.max(axis=1, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=1, keepdims=True)
def forward(X):
return softmax(X @ W + b)
def loss(probs, y):
return -np.log(probs[np.arange(len(y)), y] + 1e-9).mean()
# Train with manual gradient descent — bare-bones backprop
lr = 0.1
for epoch in range(200):
probs = forward(X)
L = loss(probs, y)
# Backward pass — gradients of cross-entropy through softmax
one_hot = np.eye(2)[y]
dZ = (probs - one_hot) / len(y)
dW = X.T @ dZ
db = dZ.sum(axis=0)
W -= lr * dW
b -= lr * db
if epoch % 40 == 0:
acc = (probs.argmax(axis=1) == y).mean()
print(f"epoch {epoch:3d} loss {L:.3f} acc {acc:.2%}")
print("\nFinal accuracy:", (forward(X).argmax(axis=1) == y).mean())#What's been built on this
What deep learning has actually shipped
From AlexNet to AlphaFold
CNN that broke ImageNet
AlexNet (2012)
17%
Error reduction overnight
The paper that triggered the deep learning revolution. Krizhevsky, Sutskever, Hinton.
CNN
DL + reinforcement
AlphaGo (2016)
4-1
Defeated Lee Sedol
Combined deep CNNs with Monte Carlo tree search. The end of 'AI can't play Go'.
CNN + RL
Real-time perception
Tesla Vision
100Hz
Inference rate, on car
Custom silicon (HW4) running pruned neural nets at 100Hz on every Tesla. Pure deep-learning car.
CNN + custom HW
Vision + transformers
AlphaFold 2/3
200M+
Protein structures
Won the 2024 Nobel. A 50-year grand challenge solved with deep learning.
Hybrid net
Diffusion + transformers
Sora / Veo
60s
Coherent video, single prompt
Text-to-video crossed the 'looks real' line in 2024. Pure deep learning at scale.
Diffusion
Transformer LLMs
ChatGPT / Claude
1B+
Weekly users
All transformer architecture, all gradient descent, all the way down.
Transformer
#The 2026 frontier
Active research directions:
- Mamba / SSM models — replacing attention with state-space layers for linear-cost long context.
- Mixture of Experts (MoE) — sparsely activate parts of a giant model. DeepSeek V3, Mistral, OpenAI's GPT-4 all use this.
- FlashAttention 3 — making attention 2× faster on H100/B200.
- Distributed training (FSDP, ZeRO-3, 3D parallelism) — how 25,000-GPU training runs actually work.
#Where to go next
- Deep Learning track — 18 lessons: tensors, autograd, CNNs, RNNs, transformers, FSDP.
- Math Foundations — derivatives and the chain rule are the prerequisite.
- NLP & Transformers — the architecture that ate language.
- Generative AI — diffusion, GANs, video generation.
#Key takeaways
Key Takeaways
- A neural network is just stacked matrix multiplications with a nonlinearity in between, trained by gradient descent.
- Four ingredients: layers, loss function, backpropagation (chain rule), optimizer (Adam).
- Backpropagation is not magic — it's the chain rule applied carefully through a computation graph.
- CNNs broke vision (2012). Transformers broke language (2017). Same recipe, different architecture.
- Most practitioners fine-tune existing models, not train from scratch.
- The 2026 frontier: Mamba/SSMs, MoE, FlashAttention, distributed training at 25K GPUs.
#References & further reading
- Goodfellow, Bengio, Courville — Deep Learning (free at deeplearningbook.org). The textbook.
- Andrej Karpathy — Neural Networks: Zero to Hero (YouTube). Builds GPT from scratch.
- Krizhevsky, Sutskever, Hinton — ImageNet Classification with Deep CNNs (NeurIPS 2012). The foundational paper.
- FlashAttention 2 / 3 (Dao 2023, 2024). The attention speedup behind every modern LLM.
- 3Blue1Brown's neural network series (YouTube) — the most beautiful intro on the internet.