How a Network Learns: Start Here

#Learning Is Guess, Check, Adjust

Picture an archer on a practice range, with a friend standing at the target. The archer shoots, the friend calls out "two paces left", and the archer turns a small knob on the bow sight a little to the right. Then the archer shoots again. Nobody explains the physics. Shoot, hear how far off it was, adjust, shoot again. After enough rounds, the arrows land close.

To keep the numbers tiny, we use a machine with exactly one dial. It is told an input called x and has to guess an answer. Its guess is the input multiplied by the dial, which we call w. So the guess is w times x.

We also have some examples where we know the right answer:

You can see the pattern: the right answer is always 2 times the input. The machine does not know that. It only has the table and a dial that starts at 0. A good result would be a dial that ends up near 2.

In one sentence: learning is a loop of guess, measure how wrong, adjust a dial, and guess again.

#How Wrong Is a Guess?

To adjust the dial in a sensible way, the machine needs one number that says how wrong it is across all the examples. Here is a simple recipe. For each example, find the miss, which is the guess minus the right answer. Square each miss, meaning multiply it by itself. Then take the average of the squares.

Why square? Two reasons. A miss of minus 3 and a miss of plus 3 are equally bad, but if you just added them you would get 0 and think all was well. Squaring makes every miss positive. And squaring makes a big miss count much more than a small one: a miss of 4 gives 16, while a miss of 1 gives only 1.

Look at the shape. The how-wrong number falls as the dial goes from 0 to 2 and rises again from 2 to 4. It is like a smooth bowl, and learning means rolling down to the bottom of the bowl. The bottom, w = 2, is where the machine matches the data.

The machine in this lesson cannot read a table of five values and see the shape. It can only try the dial where it is. So it needs a way to find which way is downhill from the spot where it stands. That is the next step.

In one sentence: the how-wrong number is the average of the squared misses, and learning means moving the dial to make it smaller.

#Which Way Is Downhill?

Imagine you are standing on a hillside in thick fog. You cannot see the valley, but you can feel the ground under your feet. If you shift your weight a little to the right and you go down, then downhill is to the right. You do not need to see the whole hill. A tiny test step tells you the direction.

The machine does the same with the dial. It tries a tiny nudge, adds 0.01 to w, and sees what happens to the how-wrong number.

new w = old w minus step size times slope

The step size is a number you choose. The usual name for it is learning rate. Take a step size of 0.1. At w = 0 the slope is minus 18.62, so the new w is 0 minus 0.1 times minus 18.62, which is 1.862.

Notice how the steps shrink by themselves. When the dial is far from the bottom, the slope is steep and the step is big. Near the bottom the slope is gentle and the steps get small. Nobody told the machine to slow down. The slope did it.

In one sentence: try a tiny nudge, see which way the how-wrong number moves, and take a step the other way.

#Watch It Learn

Below is the same machine. The curve is the bowl of how-wrong numbers. The red dot is the dial. The green dashed line marks w = 2, the bottom. Press "Nudge once" and the dot makes one step, and the table under the chart records the step.

Before you press, predict the first step with a step size of 0.10 and the dial at 0: you worked it above.

Do: Leave the starting dial at 0 and the step size at 0.10. Press "Nudge once" four times.

Notice: The dial goes 1.862, 1.986, 1.994 and 1.995, and the how-wrong number falls from 18.667 to 0.089 and then toward 0. Each step is smaller than the one before.

Why: The slope is steep at the start and gentle near the bottom, and the step is the slope times the step size. So the steps shrink on their own as the dial nears 2.

Do: Press Reset. Set the step size to 0.20 and press "Nudge once" four times. Then Reset and try 0.30.

Notice: At 0.20 the dial jumps past 2 to about 3.7, then back to about 0.5, then past 2 again, and it is still zig-zagging. At 0.30 it jumps farther each time and flies off the chart.

Why: A step that is too big overshoots the bottom and lands higher on the other side of the bowl. If it overshoots by more than it started away, each step gets worse.

In one sentence: with a sensible step size the dial rolls down into the bottom of the bowl, and with a too-big step it bounces or flies away.

#Run the Loop in Code

# Learn one dial by nudging it. The right answer is always 2 times the input.
xs = [1, 2, 3]
ys = [2, 4, 6]

def how_wrong(w):
    total = 0
    for x, y in zip(xs, ys):
        miss = w * x - y
        total = total + miss * miss
    return total / len(xs)

def slope_by_nudging(w, nudge=0.01):
    return (how_wrong(w + nudge) - how_wrong(w)) / nudge

w = 0.0
step_size = 0.1
for step in range(4):
    s = slope_by_nudging(w)
    print(f"step {step}: w = {w:.4f}, how wrong = {how_wrong(w):.4f}, slope = {s:.3f}")
    w = w - step_size * s
print(f"after 4 steps: w = {w:.4f}, how wrong = {how_wrong(w):.4f}")
print(f"guess for a new input x = 10: {w * 10:.2f}")

Do: Run the cell. Then change step_size to 0.05 and run it again.

Notice: At 0.1 the printed dial values are 0.0000, 1.8620, 1.9861 and 1.9944, matching your hand steps, then 1.9950 after the fourth update, and the guess for x = 10 is 19.95. At 0.05 the dial climbs more slowly and is still short of 2 after four steps.

Why: A smaller step size takes smaller steps, so it needs more of them. The last line shows the point of learning: the dial was found from three examples and it also gives a sensible guess for an input of 10 that it never saw.

In one sentence: the code is the hand method repeated, and its printed numbers match the steps you worked on paper.

#Too Small, Just Right, Too Big

The step size has to be chosen well, and the choice matters a lot. Here is a second experiment. We run the loop for five steps with four different step sizes, starting from w = 0, and print where the dial lands after each step. Make a guess before you run it.

# What happens when the step size is too small or too big?
xs = [1, 2, 3]
ys = [2, 4, 6]

def how_wrong(w):
    return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)

def slope_by_nudging(w):
    return (how_wrong(w + 0.01) - how_wrong(w)) / 0.01

for step_size in (0.05, 0.1, 0.2, 0.3):
    w = 0.0
    path = []
    for step in range(5):
        w = w - step_size * slope_by_nudging(w)
        path.append(round(w, 2))
    print("step size", step_size, "-> w after each step:", path)

Do: Run the cell. Then put 0.01 in front of the list of step sizes, making it (0.01, 0.05, 0.1, 0.2, 0.3), and run again.

Notice: Step size 0.05 gives 0.93, 1.43, 1.69, 1.83, 1.91 and is closing in on 2. Step size 0.1 gives 1.86, 1.99, 1.99, 1.99, 1.99. Step size 0.2 gives 3.72, 0.5, 3.29, 0.87, 2.97 and bounces. Step size 0.3 gives 5.59, minus 4.47, 13.63, minus 18.95, 39.69, getting worse each time.

Why: Each step multiplies the distance from 2 by a fixed factor. At 0.1 the factor is small, so the distance collapses. At 0.2 the factor is about minus 0.87, so the dial flips side and loses only a little distance each time. At 0.3 it is about minus 1.8, so the distance grows.

This is the main practical lesson of training: if a network's loss goes up or jumps around instead of falling, the first thing to suspect is a step size that is too big.

In one sentence: a step size that is too small wastes time, and one that is too big overshoots and can run away.

#From One Dial to Millions

Our machine had one dial and three examples. A real network has a weight on every connection. The walk neuron from the last lesson had 4 numbers: three weights and a bias. Small image networks have millions. Large language models have billions.

The loop is the same. Guess on the examples. Measure how wrong. For every dial, find which way to turn it. Turn each dial a small step. Repeat.

Keep one more thing in mind. A network is judged on inputs it has not seen, like the x = 10 in our code, not only on the examples it trained on. A network that learns the training examples perfectly but fails on new ones has not really learned. Later lessons on regularization and on practical training tricks are about this.

In one sentence: a real network repeats the same nudge loop on millions of dials, and backpropagation is the fast way to find all the slopes at once.

#Where to Go Next

You can now explain training in plain words, work the nudge loop by hand, and see why the step size matters.

• Next: Tensors & Automatic Differentiation. It shows how real tools store the numbers and find all the slopes for you.
• Then: Perceptron & Multilayer Networks, Activation Functions and Loss Functions put names and more detail on the pieces you met.
• After that: Backpropagation explains how all the slopes are found in one pass.

This track has levels. The first two lessons are Level 0. After them come the foundations, then training, then vision, then sequences and attention, and then generative and advanced models.

In one sentence: read the tools lesson next, then work through the foundations in order.

#Key Takeaways

• Learning is a loop: guess, measure how wrong, nudge the dial, repeat
• The loss is one number for how wrong all the guesses are, and learning means making it smaller
• A tiny test nudge shows which way is downhill, and the slope says how steep
• The step is the step size times the slope, so steps shrink on their own near the bottom
• A step size that is too small is slow and one that is too big bounces or runs away
• A real network is the same loop on millions of dials, with backpropagation finding the slopes quickly

#Quick Check