Deep Learning From Scratch Lab
loss.backward() and optimizer.step() in one line each. That is convenient until a loss turns into nan, or a model trains and learns nothing, and no message says why. This lab has eight exercises. Each is a short function you write yourself in numpy, then check with hidden tests that run in your browser. Each one starts with a plain-words setup and a trace by hand, so you know the right number before you type a line.After this lesson, you will be able to:
- Write sigmoid and ReLU with their slopes, and make sigmoid safe for large negative inputs
- Write the forward pass of a dense layer on a batch and keep the shapes straight
- Write softmax and cross-entropy so that big scores cannot overflow, and write their combined gradient
- Write backpropagation for a two-layer network and prove it right with a numerical gradient check
- Write the SGD with momentum update and the Adam update, including Adam's bias correction
- Write batch normalization in training and evaluation mode, and inverted dropout forward and backward, and name the traps that break them
Before You Start
#How This Lab Works
TODO marks. It has a Run button and a Check my work button, and under it sits a Show solution button. The hints appear one at a time, so you can take the smallest nudge that gets you moving.Check my work runs your code, then runs a list of hidden tests, one at a time. It tells you how many passed and shows the message of the first one that failed. The starter is built to fail on purpose, so a fresh cell always says some checks failed. When you finish the TODO lines, all of them pass.
Three things about the tests are worth knowing before you start.
- They are tolerance based. A result close enough to the right answer passes, so a harmless rounding difference never fails you.
- They are deterministic. Any random numbers come from a fixed seed, so a test that fails today fails the same way tomorrow.
- They check more than the one example in the hand trace. Some compare your answer with a second route to the same number, such as a slow loop version, or a slope found by nudging a number a tiny bit each way. If your code only matches the trace, it will still fail those.
The cells use numpy only. The browser has no torch, so none of this needs a graphics card, and each cell runs in a moment.
A raw score from the last layer of a network, one per class, before it is turned into a probability. It can be any number.
How much the loss changes when one number changes a little. Training moves each weight a small step against its gradient.
Testing a gradient by nudging one number a tiny bit up and down, recomputing the loss, and dividing the change by the gap. It needs no calculus and is slow, so it is used only to test.
A running pile of past gradients. Steps in a steady direction add up, and steps that flip back and forth cancel.
Adam starts its running averages at 0, which makes the early ones too small. Dividing by a known factor fixes the early steps.
The mean and variance that batch normalization keeps while training, to use later when no batch is available.
Some layers act differently when learning and when answering. Batch normalization and dropout are two of them.
An array of the same shape as the data, with one number per entry. Multiplying by it keeps some entries and zeroes others.
#The Conventions Used Here
Many mistakes in this area are not about maths. They are about which of two conventions a sentence uses. Fix them before writing code. This table is the single source for the whole lab, and the tests assume it.
| Quantity | Meaning in this lab |
|---|---|
| Batch | X has shape (n, d): n examples as rows and d numbers per example as columns. |
| Layer | W has shape (d_in, d_out) and b has shape (d_out,). A layer computes X @ W + b. |
| Labels | y is a 1-D array of whole numbers from 0 to k - 1, one per row. It is not one-hot. |
| Loss | The loss is the mean over the batch, so every gradient carries one factor of 1 / n. |
| Gradient names | dW is the gradient of the loss with respect to W. A gradient always has the same shape as the thing it belongs to. |
| Slope of ReLU at 0 | Taken as 0. |
| Momentum | velocity = beta * velocity + grad, then w = w - lr * velocity. This is the form PyTorch uses. Some books fold lr into the velocity, which gives different numbers. |
| Adam | The step counter t starts at 1. The small number eps is added after the square root. |
| Batch norm | The batch variance divides by n. decay is the weight on the OLD running value. PyTorch's own momentum argument is the weight on the NEW value, so a decay of 0.9 here matches a momentum of 0.1 there. |
| Dropout | p is the chance of dropping a number. The mask holds 0 for a dropped number and 1 / (1 - p) for a kept one. |
When a convention differs from the book you learned from, the tests follow this table. Neither is wrong, and a small change of convention gives different numbers, so name yours before you compare results.
#Exercise 1: Sigmoid, ReLU and Their Slopes
s * (1 - s). The slope of ReLU is 1 for a positive input and 0 for a negative one.1 / (1 + exp(-x)) asks for exp(1000) when x is -1000. That number is too big for a 64-bit float, so numpy reports an overflow. The test treats an overflow as a failure, because it is a warning that the next operation may produce nan. You write the version that never makes the huge number.Sigmoid, ReLU and their slopes
Write sigmoid, sigmoid_grad, relu and relu_grad in numpy. sigmoid must not overflow for very large or very small inputs. Press Check my work to run the hidden tests.
The test that catches most wrong answers is the nudge test. Move x by 0.000001 up and down, divide the change in output by the gap, and compare with your slope function. It works for any function that has a slope, which is why exercise 4 uses it on a whole network. The slope of ReLU at exactly 0 is not defined, because the graph has a corner there. This lab takes 0, which is a common choice.
#Exercise 2: A Dense Layer on a Batch
X is (n, d_in), W is (d_in, d_out), so X @ W is (n, d_out). The bias b has d_out numbers and is added to every row. After that the activation is applied. You write the layer with three activations: relu, sigmoid and linear, where linear means "leave it alone".A dense layer on a batch
Write dense_forward(X, W, b, activation) that returns the raw output Z and the activated output A for a whole batch. Support the activations 'relu', 'sigmoid' and 'linear'. Press Check my work to test it on several shapes.
Most dense-layer bugs are shape bugs. If the shapes do not line up, numpy tells you at once. The dangerous bugs are the ones where the shapes happen to line up and the meaning is wrong, such as a square weight matrix used the wrong way round. That is why the test uses a non-square W and also compares with a slow loop that adds up one product at a time.
#Exercise 3: Softmax and Cross-Entropy, Done Safely
exp of a big score overflows. The fix is to subtract the row's biggest score before exponentiating. That does not change the softmax, because the same factor appears on top and bottom and cancels, and it makes the biggest exponent exp(0) = 1. Second, if a probability rounds to 0, its log is minus infinity. So compute the log-probabilities directly as shifted - log(sum(exp(shifted))), never as the log of a softmax. You also write the gradient, which has a famously simple form: softmax minus one-hot, divided by n.Softmax and cross-entropy
Write softmax, cross_entropy and softmax_ce_grad for a batch of logits. They must work for scores like 1000 without overflow. Press Check my work to test the values, the stability and the gradient.
p - one_hot, many libraries offer a single function that takes raw logits and does both steps in a stable way. The reason to prefer it is exactly the stability you just wrote.#Exercise 4: Backprop Through a Two-Layer Network
a, weights W and output gradient dz: the weight gradient is a.T @ dz, the bias gradient is dz added up over the rows, and the gradient sent back to the input is dz @ W.T. For the sigmoid in the middle, multiply the incoming gradient by the slope a1 * (1 - a1). The loss end starts you off: dz2 = (p - one_hot(y)) / n.How do you know the backward pass is right? Nudge one weight a tiny bit up and down, recompute the loss both times, and divide the change by the gap. That number is the true slope for that weight, found without any calculus. The hidden tests do this for every weight of your network and compare. If your backprop is right, the two agree to many digits.
Backprop through a two-layer network
Finish two_layer_loss_and_grads. The forward pass and the loss are written. Add the backward pass that returns the gradients for W1, b1, W2 and b2. Press Check my work: it compares every gradient with a numerical gradient.
Two of the tests are not about the gradient values at all. One feeds every example twice and demands the same loss and gradients, which catches a sum used where a mean belongs. The other checks the loss against a slow loop version of the same network, so that a wrong loss with a matching wrong gradient cannot slip through. A gradient check only proves the gradient matches the loss you wrote. It cannot tell you that the loss itself is the one you meant.
#Exercise 5: SGD With Momentum
velocity = beta * velocity + grad, then w = w - lr * velocity, where beta is how much of the old velocity to keep. A value of 0.9 is a common starting point. You write one step, then a short loop that uses it.SGD with momentum
Write sgd_momentum_step(w, grad, velocity, lr, beta), which returns the new weights and the new velocity, and use it in the minimize loop. Press Check my work to test single steps, the steady-gradient limit and a stretched bowl.
velocity = beta * velocity - lr * grad and then w = w + velocity. It describes the same idea with lr folded into the velocity, and it gives different numbers for the same lr. The tests follow the table at the top of the lab.#Exercise 6: The Adam Update
m, is an average of the gradient and says which way to go. The second, v, is an average of the squared gradient and says how large the gradients usually are. The step is the learning rate times m divided by the square root of v, so a weight with large gradients takes a smaller step than a weight with tiny gradients.m by 1 - beta1^t and v by 1 - beta2^t, where t is the step number, counting from 1. The full update is w = w - lr * m_hat / (sqrt(v_hat) + eps). The usual starting values are lr = 0.001, beta1 = 0.9, beta2 = 0.999 and eps = 0.00000001.The Adam update
Write adam_step(w, grad, m, v, t, ...) that returns the new weights and the two new running averages, with bias correction. Press Check my work to test the first steps, tiny and huge gradients, and a short run on a bowl.
The most surprising test is the one that feeds gradients of 0.5, 50 and -0.003 on the first step and expects steps of almost exactly lr, with the sign of the gradient reversed. That is Adam's defining property: the size of the first step does not depend on the size of the gradient. Another test uses a gradient of 0.000000001, so small that eps matters, to check that eps was added after the square root and not inside it.
#Exercise 7: Batch Normalization Forward
sqrt(var + eps). The tiny eps stops a division by zero. Then it rescales with two learnable numbers per feature: gamma multiplies and beta shifts, so the network can undo the normalization if that helps.new = decay * old + (1 - decay) * batch. In evaluation, a single example may arrive alone, so there is no batch to measure. The layer uses the running averages that it saved during training, and leaves them unchanged.Batch normalization forward
Write batchnorm_forward(x, gamma, beta, running_mean, running_var, training, decay, eps). Return the output and the new running statistics. Press Check my work to test both modes and the edge cases.
The sharpest test feeds data whose columns have very different means and spreads, then demands that each output column has mean 0 and variance close to 1 while the mean of each row is not forced to 0. Normalizing across a row instead of a column is layer normalization, a different layer. Another test adds a constant to a whole column and expects no change in the output, which is true only if the batch mean is subtracted.
#Exercise 8: Inverted Dropout, Forward and Backward
p. If you only zeroed numbers, the layer's output would be smaller on average than at test time, when nothing is dropped. Inverted dropout fixes this inside training: the numbers that are kept are multiplied by 1 / (1 - p), so the average stays the same, and at evaluation time the layer does nothing at all.1 / (1 - p) for a kept one. The forward pass is x * mask. The backward pass multiplies the incoming gradient by the same mask, so no gradient flows through a dropped number. The random numbers must come from the generator you are given, so that a fixed seed gives a fixed mask.Inverted dropout
Write dropout_forward(x, p, rng, training) and dropout_backward(dout, mask). Training drops numbers at random and scales the kept ones up. Evaluation returns x unchanged. Press Check my work to test the mask values, the average, the seeding and the gradient.
The test that most often catches a wrong answer averages the output over many numbers and expects the input average back. A version that only zeroes numbers, without the scale-up, comes out about half as large with p = 0.5, and the test says so. A second test makes two calls with the same generator and demands different masks, which catches code that creates a fresh generator inside the function.
#Where the Traps Are
Every exercise has one common wrong answer that looks right. They fall into four families.
exp of a large number overflows, and log of a number that rounded to 0 is minus infinity. Subtract the maximum before exp, and compute log-probabilities as shifted minus log-sum-exp. Keep eps where the formula puts it: inside the square root for batch norm, outside it for Adam.| Symptom | Likely cause |
|---|---|
| The loss becomes nan or inf after a few steps | exp overflowed in softmax or sigmoid, or log of a probability that rounded to 0 was taken |
| Training runs but the model learns very slowly, or the gradient check is off by a constant factor | The loss is a mean but the 1 / n was left out or applied twice |
| A shape error, or a result of the right shape with the wrong meaning | A transposed weight matrix, or a sum along the wrong axis |
| Training accuracy is good and test accuracy is poor | Evaluation mode was not switched on, so batch norm still uses the batch and dropout still drops |
| The same example gets a different answer in a different batch | Batch norm used the batch statistics at evaluation instead of the running ones |
| Adam's first step is far larger or smaller than the learning rate | Bias correction is missing, or t started at 0 |
| Dropout makes outputs about half as large | The kept numbers were not scaled up by 1 / (1 - p) |
#Capstone Checklist
Use this list to check that you are ready to read and write real training code. Tick each item only if you can do it without looking.
- I can write sigmoid so that it does not overflow, and ReLU, with both of their slopes
- I can write a dense layer on a batch and say the shape of every array in it
- I can write softmax and cross-entropy that survive scores of 1000, and write their gradient
- I can write backprop for a two-layer network and prove it with a numerical gradient check
- I can write one SGD with momentum step and say what the velocity settles at under a steady gradient
- I can write one Adam step, with bias correction, and say why its first step is about lr
- I can write batch normalization in both modes and say which statistics each mode uses
- I can write inverted dropout forward and backward and say why evaluation does nothing
- Given a training loop I have never seen, I can find the axis, the mean-or-sum, the stability and the train-or-eval choices in under a minute
If you can tick every line, you can read the training code of a library and spot a convention mismatch. If you cannot, go back to the exercise for that line, open the solution, and write the function again without it.
Learning representations by back-propagating errors
David Rumelhart, Geoffrey Hinton, Ronald Williams (1986)
The chain rule through a network, as in exercise 4.
Adam: A Method for Stochastic Optimization
Diederik Kingma, Jimmy Ba (2015)
The update with bias correction from exercise 6.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe, Christian Szegedy (2015)
The training and evaluation behaviour from exercise 7.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov (2014)
Dropout, of which exercise 8 writes the inverted form.
#Key Takeaways
- Sigmoid must be written with
exp(-abs(x))so it never overflows. Its slope is s * (1 - s), at most 0.25, and ReLU's slope is a step that is 0 at 0 in this lab - A dense layer is
X @ W + bwith X (n, d_in), W (d_in, d_out), b (d_out,). Test it with a non-square W and a slow loop - Softmax subtracts the row maximum, and cross-entropy uses shifted minus log-sum-exp. The gradient of the mean loss with respect to the logits is (softmax - one-hot) / n
- Backprop is the chain rule written one layer at a time. A numerical gradient check, nudging each weight both ways, proves it, and a mean loss carries one factor of 1 / n
- SGD with momentum keeps
velocity = beta * velocity + grad, and under a steady gradient the velocity settles at grad / (1 - beta) - Adam divides a bias-corrected gradient average by the square root of a bias-corrected squared-gradient average, so its first step is about lr whatever the gradient size
- Batch norm uses batch statistics in training and saved running averages in evaluation. Inverted dropout scales kept numbers by 1 / (1 - p) and does nothing in evaluation
#Quick Check
Why does the safe sigmoid use exp(-abs(x)) instead of exp(-x)?