In 1986, Rumelhart, Hinton, and Williams published backprop and started the modern AI era. The math is calculus you already know — the chain rule. The insight is reusing intermediate results so you don't recompute the same gradient a billion times. Here's the algorithm that made every neural network you've ever used possible.
Learning Objectives
After this lesson, you will be able to:
Understand backpropagation as a systematic way to trace errors backward through the network using the chain rule
Compute gradients by hand for a small 2-layer network, step by step
Recognize why vanishing and exploding gradients happen and how activation function choice and depth play a role
Distinguish backpropagation (gradient computation) from gradient descent (weight update) and explain why they are separate steps
After the forward pass produces a prediction y-hat, the loss function measures how wrong it is: L = (y - y-hat)^2 (for MSE) or L = -log(y-hat[true class]) (for cross-entropy). This single number is the starting point for all gradient computation.
Compute how the loss changes with respect to the prediction: dL/dy-hat. For MSE: dL/dy-hat = 2(y-hat - y). For cross-entropy with softmax: dL/dz = y-hat - y_true. This tells us the direction and magnitude of the error at the output.
The gradient flows backward through the output activation. For softmax+cross-entropy, this simplifies beautifully: dL/dz2 = y-hat - y_true. For ReLU hidden layers, the gradient passes through unchanged where the neuron was active, and is zeroed where it was inactive.
The gradient for the output weights is: -- the outer product of the error signal and the previous layer's activations. Each weight's gradient reflects how much it contributed to the error. These gradients tell us exactly how to adjust W2.
Loading visualization...
Active Recall
Before we open the chain rule machinery: from the derivatives lesson, write the chain rule for y = f(g(x)) from memory in one line of notation. Then say in plain English what each factor represents. Do not scroll back to check — commit to your version first, then we'll use this exact rule for every layer below.
Write your answer in your own words — don't look back at the lesson. This is the most effective way to remember what you just learned.
Backpropagation can feel intimidating because of all the math notation. But at its core, it is just one idea: trace the blame for a wrong answer backward, one step at a time. If you can follow a chain of "because of this, because of that," you can understand backprop. You are closer than you think.
Before we touch neural networks, let us review the chain rule. If y = f(g(x)), then:
Try it! Take y = (3x + 1)^2. Compute dy/dx using the chain rule: the outer derivative is 2(3x+1) and the inner derivative is 3, so dy/dx = 6(3x+1). Verify by expanding and differentiating directly. The chain rule just saved you work -- and it saves even more work in a 96-layer network.
A neural network can be drawn as a computation graph -- a directed graph where each node is an operation (multiply, add, ReLU, etc.) and edges carry values. The forward pass flows left to right. Backpropagation flows right to left, carrying gradients.
What Do You Think?
In a computation graph for y = (a + b) * c, if the gradient flowing back to the multiplication node is dL/dy = 1, what gradient flows to a?
The multiplication node passes gradient c to the (a+b) branch and gradient (a+b) to the c branch. The addition node inside (a+b) passes the incoming gradient equally to both a and b. So dL/da = 1 * c * 1 = c.
Cross-entropy loss: L = -log(y_hat[true_class]) = -log(0.1) = 2.30. The network assigned only 10% probability to the correct class -- a bad prediction. This single number L is the starting point for backpropagation.
3
#Step 3: Gradient of Loss w.r.t. Output (dL/dy_hat)
For cross-entropy with softmax, the gradient has a beautiful simplification: dL/dz2 = y_hat - y_true = [0.1 - 1, 0.7 - 0, 0.2 - 0] = [-0.9, 0.7, 0.2]. The gradient is simply "predicted minus actual." Neurons that over-predicted get positive gradient (reduce them); those that under-predicted get negative gradient (increase them).
The step-by-step section above is symbolic. Now let us do it with real numbers and not a single Greek letter left to the imagination. We will pick a tiny network, push one input through it, then compute every single gradient by hand. At the end we verify against a finite-difference numerical estimate, just like a real autograd implementation would in its test suite.
The network. Two-dimensional input, one hidden layer of three ReLU units, scalar output, MSE loss:
x -> z1 = W1^T x -> h = ReLU(z1) -> y_hat = W2^T h -> L = 1/2 (y_hat - y)^2
We use no biases in this example (b1 = 0, b2 = 0) so every number on the page is a weight or an activation. The exact starting values are:
h = [1.1, 1.4, 0.0] -- the third unit is zeroed because z1[2] < 0.
Outputy_hat = W2^T h:
That is the entire forward pass: four lines of arithmetic, and now we know exactly how wrong the network is. The third hidden unit contributed nothing because ReLU killed it; remember that, because backprop is going to remember it too.
We walk back through the same computation graph in reverse, multiplying local derivatives. Save every intermediate -- we need them.
Output gradientdL/dy_hat = y_hat - y = -0.04 - 1.5 = -1.54. The prediction was 1.54 below the target, so the network needs to push y_hat upward.
Gradient for W2dL/dW2 = h * dL/dy_hat. Element-wise:
dL/dW2[0] = 1.1 * (-1.54) = -1.694
dL/dW2[1] = 1.4 * (-1.54) = -2.156
dL/dW2[2] = 0.0 * (-1.54) = 0.000
The dead ReLU unit gets a zero gradient on its outgoing weight -- it neither helped nor hurt the prediction.
Gradient passed back to hdL/dh = W2 * dL/dy_hat = [0.6, -0.5, 0.7] * (-1.54):
dL/dh = [-0.924, 0.770, -1.078].
Notice the entire third column of dL/dW1 is zero. Backprop has correctly figured out that the third hidden unit is currently a passenger and any gradient flowing into it is wasted -- which is the "dead ReLU" failure mode in disguise.
∂W2∂L=h⋅∂y^∂L,∂W1∂L=x⋅(∂z1∂L)T
ReLU′(z)={10z>0z≤0
What Do You Think?
Before reading on -- what should the gradient dL/dW2[0] be in this example? (W2[0] is the weight connecting the first hidden unit to the output.)
One backward pass, all six weights updated. If we now re-ran the forward pass with the new weights, y_hat would move from -0.04 toward 1.5, and the loss would drop. That is one SGD step on one example. Repeat for millions of examples and you have trained a network.
If the math above is right, perturbing one entry of W1 by a tiny epsilon and recomputing the loss should match the analytical gradient. We picked W1[0,0], whose analytical gradient is dL/dW1[0,0] = -0.924.
∂W1[0,0]∂L≈εL(W1[0,0]+ε)−L(W1[0,0])
The MathPlayground below runs exactly this check; the analytical and finite-difference gradients should agree to about five decimal places. If you ever write a custom autograd op in PyTorch or JAX, this is the test you write first -- it catches almost every backward-pass bug.
Loading visualization...
#Worked example 2: softmax + cross-entropy, the multi-class case
The first worked example was a regression network with MSE loss. The second-most-important loss in deep learning is softmax + cross-entropy for classification, and it has a famously clean gradient that is worth deriving by hand at least once. Pick a three-class problem so every number fits on one line.
Setup: logits z = [2.0, 1.0, 0.1] arriving at the output layer of some classifier, and the true label is class 0, encoded one-hot as y = [1, 0, 0].
The softmax converts logits to a probability distribution by exponentiating each entry and dividing by the sum:
exp(z) = [e^2.0, e^1.0, e^0.1] = [7.389, 2.718, 1.105]. The denominator is 7.389 + 2.718 + 1.105 = 11.213. So p = exp(z) / sum(exp(z)) = [0.659, 0.242, 0.099]. The model is 65.9 percent confident in the correct class.
Cross-entropy loss for one-hot targets simplifies to the negative log of the predicted probability assigned to the true class: L = -ln(p_0) = -ln(0.659) = 0.417. If the model were perfectly confident the loss would be zero; if it assigned the truth probability near zero, the loss would shoot to infinity.
Now the famous result: the gradient of cross-entropy loss with respect to the logits (not the probabilities) is just p - y.
Where does p - y come from? It is not magic, it is the softmax Jacobian times the cross-entropy gradient, telescoped. Cross-entropy with respect to the probability of the true class is dL/dp_0 = -1 / p_0. Softmax with respect to its own input logit is dp_0/dz_0 = p_0 (1 - p_0) along the diagonal and dp_i/dz_0 = -p_0 p_i off-diagonal. Chain-rule the true-class term: dL/dz_0 = (-1/p_0) * p_0(1-p_0) = -(1 - p_0) = p_0 - 1. With our numbers, 0.659 - 1 = -0.341. Check. For a wrong class i != 0, the only contribution is the off-diagonal one: dL/dz_i = (-1/p_0) * (-p_0 p_i) = p_i. With our numbers, dL/dz_1 = 0.242 and dL/dz_2 = 0.099. Check. The full vector is p - y and the per-component derivation matches.
Try it: Watch gradients flow backward and spot vanishing signalsInteractive
Loading visualization...
Try this: Watch the gradients flow backward through the network. Notice how the magnitude of gradients often decreases as they propagate to earlier layers -- this is the beginning of the vanishing gradient problem. Also notice how neurons that were zeroed by ReLU block the gradient entirely.
Try it: Compare sigmoid vs ReLU gradient flow across deep networksInteractive
Loading visualization...
Try this: Start with Sigmoid activation and 10 layers. Notice how neuron circles shrink and turn blue (cold) in earlier layers -- gradients are vanishing. Now switch to ReLU: the neurons stay large and warm-colored throughout. This is exactly why ReLU enabled deep learning. Increase network depth to 12 layers with Sigmoid and watch the first layer's gradient drop to near zero. Toggle "Show Values" to see the actual gradient magnitudes.
The symbol delta^(l) is the error signal at layer l -- it encodes how much each neuron in that layer contributed to the final loss. The circle-dot operator is element-wise multiplication.
The sigmoid function squashes values to (0, 1). Its derivative has a maximum of 0.25 -- meaning each layer can multiply the gradient by at most 0.25. After 10 layers: 0.25^10 = 0.000001. After 20 layers, the gradient is essentially zero. This is why sigmoid networks cannot be trained deep.
ReLU's derivative is exactly 1 for positive inputs and 0 for negative inputs. Active neurons pass gradients through with no attenuation. This is why ReLU revolutionized deep learning -- it solved the vanishing gradient problem for the forward direction, enabling networks with dozens or hundreds of layers.
Modern frameworks like PyTorch and JAX do not require you to implement backpropagation manually. They use automatic differentiation (autograd): every operation records itself on a computation graph during the forward pass, and calling .backward() traverses this graph in reverse, applying the chain rule automatically.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
Tests · Verify loss decreases over training. Verify predictions for x>0.5 approach 1 and x<=0.5 approach 0.
Backpropagation is the chain rule applied systematically to a computation graph. It propagates error signals backward from the loss through each layer, computing how much each weight contributed to the error
Gradients flow backward through the network. Each layer computes its local gradient and multiplies it by the gradient flowing in from above, building up the full derivative through the chain rule
Vanishing gradients make deep networks hard to train. When gradients pass through many sigmoid or tanh activations, they shrink exponentially, making early layers learn extremely slowly
Exploding gradients destabilize training. When gradients grow exponentially through layers, weight updates become enormous, causing the loss to diverge; gradient clipping is the standard fix
Automatic differentiation handles all of this for you. Frameworks like PyTorch record the computation graph during the forward pass and automatically apply backpropagation when you call .backward()
What mathematical rule is backpropagation based on?
Key Terms8 terms
Algorithm that computes the gradient of the loss with respect to every parameter in a network by applying the chain rule in reverse through the computation graph.
A directed graph where nodes are operations and edges carry values. Forward pass flows left-to-right; the backward pass walks the same graph in reverse to accumulate gradients.
Calculus rule for differentiating composed functions: dy/dx = (dy/du) * (du/dx). It is the mathematical engine behind backprop because a neural net is a long chain of composed functions.
The partial derivative of the loss with respect to a layer's pre-activation z. It encodes how much each neuron contributed to the final error and is propagated backward through the network.
When gradients shrink exponentially as they propagate to earlier layers (common with sigmoid/tanh), making early weights barely update. Solved by ReLU, residual connections, and normalization.
When gradients grow exponentially through layers (common in deep RNNs), causing unstable updates and NaN losses. Mitigated by gradient clipping (torch.nn.utils.clip_grad_norm_).
Technique that tracks every operation on a tape during the forward pass, then mechanically applies the chain rule during .backward() to compute exact gradients without manual derivation.
Memory-saving technique that discards some intermediate activations during forward pass and recomputes them during backward pass — trades ~30% more compute for ~60% less memory.
Where This Matters
OpenAI / Anthropic / Meta
Training GPT / Claude / LLaMA
Every weight in a frontier LLM is updated by backprop. A single GPT-4-scale training run runs trillions of backward passes across thousands of GPUs synchronized via AllReduce.
↑
Without backprop, LLMs literally cannot learn — all training stops
Meta
PyTorch Autograd (Meta)
PyTorch's autograd implements reverse-mode automatic differentiation — the engineering realization of backpropagation. Every loss.backward() call replays the tape to compute gradients for millions of parameters.
↑
Powers research at virtually every AI lab and company today
DeepSpeed / FSDP
Gradient Checkpointing at Scale
Training models with hundreds of billions of parameters on limited GPU memory uses gradient checkpointing plus mixed precision — engineered variants of backprop — to fit the backward pass in memory.
↑
Enables training 100B+ param models on current-generation GPUs
Interview Practice
Backpropagation is the engine of learning in neural networks. You now understand how gradients flow backward through the computation graph. Next: Activation Functions -- the nonlinear functions that give neural networks their power, and why the choice of activation matters enormously.
To continue backward, compute: dL/da1 = W2^T * dL/dz2. The transpose of the weight matrix distributes the error signal back through the same connections used in the forward pass. The error signal splits and flows to all neurons in the hidden layer.
Through the ReLU derivative and chain rule: dL/dW1 = dL/dz1 * x^T, where dL/dz1 = dL/da1 * ReLU'(z1). Each hidden weight's gradient is computed. The chain rule has traced responsibility for the error all the way back to the first layer.
Apply gradient descent to every parameter: W = W - lr * dL/dW, b = b - lr * dL/db. One backward pass computed all gradients. One update step adjusts all weights. The network has learned from its mistake and will make a slightly better prediction next time. Repeat millions of times.
By the chain rule: dL/dW2 = dL/dz2 * dz2/dW2. Since z2 = W2*a1 + b2, the derivative of z2 w.r.t. W2 is a1. So: dL/dW2 = (dL/dz2) * a1^T. This is an outer product of the error signal and the previous layer's activations. Each weight's gradient reflects how much it contributed to the error.
Even simpler: dL/db2 = dL/dz2. The gradient for the bias is just the error signal itself, because the derivative of z = Wa + b with respect to b is 1.
6
#Step 6: Pass the Gradient Backward to Layer 1 (dL/da1)
To continue the chain backward, we need the gradient w.r.t. the input to layer 2, which is the output of layer 1: dL/da1 = W2^T * dL/dz2. The transpose of the weight matrix "distributes" the error signal back to the previous layer. This is the key step -- the error signal propagates backward through the same weights used in the forward pass.
ReLU's derivative is simple: 1 if the input was positive, 0 if it was negative. So: dL/dz1 = dL/da1 * ReLU'(z1). For neurons that were active during the forward pass (z1 > 0), the gradient passes through unchanged. For neurons that were zeroed by ReLU (z1 <= 0), the gradient is killed -- set to zero. Dead neurons contribute zero gradient.
Same pattern as layer 2: dL/dW1 = (dL/dz1) * x^T. The gradient for each weight is the product of the error signal reaching that layer and the input to that layer.
Now we have gradients for every parameter. Apply gradient descent: W1 = W1 - lr * dL/dW1, b1 = b1 - lr * dL/db1, W2 = W2 - lr * dL/dW2, b2 = b2 - lr * dL/db2. One backward pass computed all the gradients. One update step adjusts all the weights. Repeat.