Optimization & Gradient Descent

#The Blindfolded Hiker

If you only learn one algorithm from this track, make it this one: feel the slope, step downhill, repeat.

From a model with two settings to one with billions, they all learn this way. Enough picture. Let us run it.

#Your First Loop: Gradient Descent in 10 Lines

Take the simplest landscape there is: the loss L(x) = x², a bowl whose lowest point is x = 0. The slope at any x is 2x (the derivative, from the previous lesson). Start at x = 4 and use a step size η = 0.1. (η is the Greek letter eta, the learning rate: the fraction of the slope we step each time.)

Every step does the same two things: measure the slope, then move x by minus η times the slope. Here are the first four steps by hand:

x shrinks towards 0, and so does the slope, so the steps get smaller on their own: the ground flattens as you near the bottom. That is the whole algorithm: measure the slope, step against it, repeat. In code it is ten lines.

Try it! Press Run, then change lr to 0.5 and run again, then to 1.1.

The printed values are 7.131 for lr = 0.01, 0.002127 for lr = 0.1 and 2.352e+04 (that is 23,520) for lr = 1.1. Read the three curves:

• Too small (0.01): the loss falls in the right direction, but only from 16 to 7.131 in 20 steps. It will get there eventually, after far too long.
• Good (0.1): the loss drops from 16 to 0.002 in the same 20 steps.
• Too big (1.1): the loss goes up, not down. It runs 16, 23.04, 33.178, 47.776 and keeps climbing. Each step overshoots the bottom and lands higher up the opposite wall. This is what "diverging" looks like on a loss curve.

Before the explanation, commit to a guess.

#When Does the Loop Converge?

Write one step as algebra. On L(x) = x² the slope is 2x, so

This matches the curves. At η = 1.1 the multiplier is −1.2, so x grows by 1.2 each step and the loss x² grows by 1.2² = 1.44: 16, then 16 × 1.44 = 23.04, then 33.178. Every bowl has its own speed limit. For L(x) = a·x² the slope is 2ax, the multiplier is 1 − 2aη, and the limit is η < 1/a: a steeper bowl (bigger a) allows only a smaller step.

#The Setup: Loss Landscapes

For a model with two parameters, the loss landscape is a 3D surface you can draw. For a model with billions of parameters it has billions of dimensions. You cannot visualise it, but the math works the same way.

#Convex vs Non-Convex Loss

Here λ (Greek lambda) is any number from 0 to 1 that slides a point along the line between x and y. In the example above, λ = ½ is the midpoint.

A convex function has at most one valley bottom, and that local minimum is automatically the global minimum. Gradient descent on a convex loss reliably converges when the step size is small enough, exactly as you derived for x². The first models you will fit in the next lessons, such as linear regression, have convex losses.

Click to drop a starting point on each landscape and watch the scoreboard: on the single bowl every drop reaches the same bottom, while on the multi-valley surface the landing spot depends on where you dropped. The non-convex case is what every neural network actually faces.

#The Gradient: Which Way Is Downhill?

#The Algorithm: The Most Important Equation in Modern ML

Name what you have been doing. Write θ (Greek theta) for all the parameters together. In the x² loop θ was just x. In the two-parameter example θ was (w₁, w₂). One step is:

Where:

• theta_t are the current parameters
• eta (Greek letter eta) is the learning rate, the step size
• nabla L is the gradient of the loss, the direction
• theta_ are the updated parameters

#Explore: The Gradient Descent Playground

Pick a surface, press play and watch the ball roll downhill, then change the learning rate and watch how the path changes.

#The Learning Rate: The Most Important Hyperparameter

The learning rate is a hyperparameter: a setting you choose before training, rather than one the model learns. It controls how big each step is, and you saw it decide between the three loss curves above. This single number has an outsized effect on whether training succeeds or fails.

#Too Small (lr = 0.0001)

#Too Large (lr = 1.0)

With a huge learning rate, the model takes giant leaps. Instead of gently descending into a valley, it overshoots the minimum entirely, landing on the opposite mountainside. The next step overshoots again. The loss oscillates wildly or explodes to infinity. On L(x) = x² that starts above η = 1; for other losses the limit is different, and steeper losses have a smaller limit.

#Just Right (lr = 0.001 to 0.01 typically)

The ideal learning rate is large enough to make meaningful progress but small enough to not overshoot. In practice, finding this sweet spot requires experimentation or learning rate finders. Modern optimizers like Adam (introduced below) adapt the effective step size automatically for each parameter, but the initial learning rate still matters.

#Learning-Rate Schedules

Linear warmup starts at zero and ramps up. With η_max = 0.001 and a warmup of 4 steps, the rates are 0, 0.00025, 0.0005, 0.00075, then 0.001 from step 4 onward.

Cosine decay then lowers the rate smoothly. With η_max = 0.001, η_min = 0 and T = 100 steps, the rate at step 50 is exactly half of η_max, 0.0005.

#Gradient Clipping

When the gradient occasionally becomes enormous (in very deep networks, or in 16-bit arithmetic, which overflows easily), clip it back into a safe range. Take a gradient vector g = [3, 4]. Its length (L2 norm) is 5. With a threshold τ = 1 (Greek tau) we scale the whole vector by 1/5, giving [0.6, 0.8], which has length 1 and the same direction. A gradient already shorter than τ, like [0.3, 0.4] with length 0.5, is left alone.

This single line rescues training from NaN cascades. In mixed-precision training, an occasional overflow can make a gradient component infinite. Without clipping, the next parameter update sends the weights to NaN and the model is dead. Clipping at τ = 1.0 keeps step sizes bounded regardless. Clipping is not a free lunch: if your gradients are exploding consistently, you have a real problem (bad initialization, learning rate too high, missing layer norm) and clipping is just hiding it.

#Reading a Loss Curve

In practice you do not watch the parameters, you watch the loss curve. Five shapes cover most of what you will see:

The rising and NaN rows are the same story at different stages: on the 1.1 curve the loss was 23,516 after 20 steps and still growing by 1.44 times per step. A few hundred more steps and it overflows to infinity, and then to NaN. The next lesson applies this loop to real data, so you will read exactly these curves there.

#Step-by-Step Walkthrough

#Variants of Gradient Descent

#Batch Gradient Descent

#Stochastic Gradient Descent (SGD)

#Mini-Batch SGD (What Everyone Actually Uses)

#Momentum: Building Speed Downhill

First the numbers. Keep a velocity v that mixes the old velocity with the new slope: v = 0.9 × (old v) + (new slope). Suppose the slope says "downhill" with the same size, 1, four steps in a row. With η = 0.1 the step is η times v:

A steady push builds speed: the steps keep growing. Now a zigzag, where the slope alternates +1, −1, +1, −1, +1, −1. The velocity goes 1, −0.1, 0.91, −0.181, 0.8371, −0.2466. It never grows past 1 in size, because each push is partly cancelled by the opposite one. Consistent directions pile up, zigzags cancel. That is all momentum does.

Now the formula. β (Greek beta, here 0.9) is the fraction of the old velocity you keep:

#Adam: The Modern Default

1. First moment (m): a running average of the gradients. Like momentum, it tracks direction.
2. Second moment (v): a running average of the squared gradients. It tracks how big the gradients typically are.

Then it divides the first by the square root of the second. Here is the very first step for one parameter, with every number shown. Take gradient g = 0.5, decay rates β₁ = 0.9 and β₂ = 0.999 (Greek beta, the "keep fraction" for each average), step size η = 0.001, and ε = 10⁻⁸ (epsilon, a tiny number that prevents division by zero). Both averages start at 0.

Why the "bias correction"? Both averages start at 0, so early on they are too small: m₁ = 0.05 is a tenth of the true gradient 0.5. Dividing by (1 − β) undoes that. Without it, the first step would be m ÷ √v = 0.05 ÷ 0.0158 = 3.16, about three times too big. With it the ratio is 1 and the first step is almost exactly η.

Now the payoff. Repeat the same arithmetic with g = 5, or with g = 0.005. The ratio m̂ ÷ √v̂ comes out as 1 (to six decimals) every time, so the first step is again about η. A parameter with huge gradients and one with tiny gradients both move by about the same amount. That is per-parameter scaling.

This means parameters with consistently large gradients take smaller steps relative to their gradient (preventing explosions), and parameters with small gradients take relatively larger steps (ensuring progress). Adam adapts automatically, which is why it is the default optimizer for most deep learning research and for training virtually all large language models.

#The Optimizer Race

Press play and watch SGD, Momentum, RMSProp, and Adam start from the same point on the same loss surface. (RMSProp is Adam's squared-gradient scaling without the momentum part.) Compare the shapes of their paths, not just who finishes first.

⚡ Playground: Gradient Descent Explorer → — adjust learning rate, momentum, and optimizer and watch convergence live on a 2D loss surface.

#Try It Yourself

You have run the loop and seen the speed limit. Now write the loop yourself and find the limit by experiment. Fill in the TODOs in the starter code, then compare with the solution.

When you run the solution, the first block prints x after 500 steps of 0.000164 for lr = 0.01 and for lr = 0.99 (both converge, the first creeping down from one side and the second zigzagging across 0), 1.4e-48 for 0.1, 0 for 0.5, 4 for 1.0 (it never moved closer, bouncing between 4 and −4), and 1.56e+40 for 1.1. The multiplier rule agrees with the experiment on all six rates. On the loss 3x² the rates 0.30 and 0.33 converge (x about 1.4e-48 and 0.000164) while 0.34 and 0.35 blow up to 1.31e+09 and 1.99e+21, so the limit sits just above 0.33, matching 1/3.

#Live: Gradient Descent in 2D, From Scratch

At lr = 0.1 the distance to the minimum after 40 steps is 0.000532. At lr = 0.2 it is 5.51e-09. Just under the limit, at lr = 0.249, it is still 0.7252 after 40 steps because y flips sign every step while shrinking only slightly. At lr = 0.26 it is 21.72: the y coordinate grows and the run diverges. One learning rate has to serve every parameter, and the steepest direction sets the limit. That is the problem Adam addresses.

#Key Takeaways

• Gradient descent is "feel the slope, step downhill". The algorithm iteratively updates parameters by moving in the direction opposite to the gradient, reducing the loss at each step
• On f(x) = x² each step multiplies x by 1 − 2η, so the loop converges exactly when 0 < η < 1. Too large and the loss curve climbs, too small and it crawls; before redesigning your architecture, try changing the learning rate by factors of 10
• A loss curve is your main dial: smooth fall is healthy, flat means too small or a bug, jagged is normal for mini-batches, rising means too big, NaN means it exploded
• Mini-batch SGD is what everyone actually uses. Computing gradients on small random batches (32-512 examples) balances speed and accuracy, and the stochastic noise helps escape saddle points and sharp minima
• Momentum smooths out oscillations. A steady push builds speed (1, 1.9, 2.71, 3.439) while a zigzag cancels itself
• Adam adapts step sizes per parameter. Dividing the averaged gradient by the square root of the averaged squared gradient makes every parameter move by about η, which is why it is the default optimizer for most deep learning

#Quick Check