Deep Learning: Start Here, a Neuron by Hand

#What a Neural Network Is Made Of

Think of a panel of three friends deciding whether to go to a film. Each friend has a view, and some friends have a louder voice than others. The panel has to turn three views into one decision.

Here is the plan for this lesson. First we work out what one neuron does, with numbers you can add in your head. Then we put three together. Finally we ask how a network gets the right numbers to begin with, which is the next lesson.

In one sentence: a neural network is many small deciders, and each decider multiplies, adds and compares.

#One Neuron: Multiply, Add, Compare

Let us make the deciding concrete. You are deciding whether to go for a walk. Three facts matter, and each is a yes or a no, which we write as 1 or 0.

• Sunny: 1 if the sun is out, 0 if not.
• Homework done: 1 if it is finished, 0 if not.
• Friend free: 1 if a friend can come, 0 if not.

The neuron follows three steps. Multiply each input by its weight. Add the three products and the bias. If the total is greater than 0, the answer is 1 (go). Otherwise the answer is 0 (stay in).

• Products: 1 times 2 is 2, 1 times 3 is 3, and 0 times 1 is 0.
• Total: 2 plus 3 plus 0 plus the bias of minus 4 is 1.
• 1 is greater than 0, so the answer is 1. Go for a walk.

• Products: 2, 0 and 1.
• Total: 2 plus 0 plus 1 minus 4 is minus 1.
• Minus 1 is not greater than 0, so the answer is 0. Stay in.

Your hand answers match the idea that homework matters most. Without the homework, the sun and a friend together reach only 3, which does not beat the head start of 4.

The picture is the same story for any case. The numbers on the arrows are the ones from Case A. Change the inputs and the numbers change, but the four steps never do.

In one sentence: a neuron multiplies each input by its weight, adds the products and the bias, and answers 1 only when the total is greater than 0.

#Try the Neuron Yourself

The panel below is the same neuron. The circles on the left are the three inputs. The numbers on the lines are the weights. The circle in the middle shows the total, and the last circle shows the answer. A table underneath shows every product so you can check the sum with a pencil. Before you press anything, predict the answer for the case that is already set up (sunny, homework done, no friend): you worked it as Case A.

Do: Press "Not sunny, homework done, friend free". Then drag the Bias slider from minus 4 up to minus 3.

Notice: At minus 4 the total is exactly 0 and the answer is 0. At minus 3 the total becomes 1 and the answer flips to 1.

Why: The answer is 1 only when the total is greater than 0. The bias is the head start. Moving it by one step pushed this case across the line.

Do: Press "Sunny, homework not done, friend free". Then raise the weight on Friend free from 1 to 4.

Notice: First the total is minus 1 and the answer is 0. With the weight at 4 the total is 2 and the answer is 1.

Why: A bigger weight makes one input count for more. Here a friend became important enough to outvote the missing homework.

In one sentence: change a weight or the bias and the same inputs can give a different answer, and the table shows exactly why.

#What a Weight and a Bias Mean

You have now used both, so here are the plain meanings.

Notice what the weights and the bias are: they are just numbers. A neuron has no secret inside. If you know its inputs, its weights and its bias, you can predict its answer with a pencil, and that is exactly what you did.

In one sentence: weights say how much each input counts, the bias is the bar the total has to clear, and the activation turns the total into a yes or a no.

#Run the Same Neuron in Code

You have worked two cases by hand. Now ask the computer for those two and three more. The code below does the same multiply, add and compare. Read it first and see that each line is a step you already did.

# One neuron: decide whether to go for a walk
def neuron(inputs, weights, bias):
    total = bias
    for x, w in zip(inputs, weights):
        total = total + x * w
    return total, 1 if total > 0 else 0

weights = [2, 3, 1]      # sunny, homework done, friend free
bias = -4

cases = {
    "sunny, homework done, no friend": [1, 1, 0],
    "sunny, homework not done, friend free": [1, 0, 1],
    "sunny, homework done, friend free": [1, 1, 1],
    "not sunny, homework not done, friend free": [0, 0, 1],
    "not sunny, homework done, friend free": [0, 1, 1],
}
for name, x in cases.items():
    score, out = neuron(x, weights, bias)
    print(f"{name}: score {score}, output {out}")

Do: Run the cell. Then change the bias to -3 and run it again.

Notice: With the bias at -4 the scores are 1, -1, 2, -3 and 0, and the outputs are 1, 0, 1, 0 and 0. The first two lines match Case A and Case B. With the bias at -3 every score goes up by 1, and the last case now has score 1 and output 1.

Why: The bias is added once to every total, so changing it shifts all five scores by the same amount. Only the cases that were at 0 or just below can flip.

In one sentence: the code is the hand method written out line by line, so a hand trace tells you what the code will print.

#One Neuron Cannot Do Everything

A single neuron is useful, but it has a limit, and the limit explains why networks have more than one neuron. Take two inputs, each 0 or 1. There are four cases. Here are three different goals.

One neuron can do AND. Use weights 2 and 2 with bias minus 3. The totals for the four cases are minus 3, minus 1, minus 1 and 1, so only the last case answers 1. One neuron can also do OR. Use weights 2 and 2 with bias minus 1. The totals are minus 1, 1, 1 and 3, so only the first case answers 0.

Now try the exclusive-or column. We want the answer 1 when exactly one input is on, and 0 when both are on or neither is. Try weights 2 and 2 with bias minus 1. The case 1, 1 gives a total of 3, so the neuron answers 1, and that is wrong. Try bias minus 3 instead. Now 1, 1 gives a total of 1, which answers 1 and is still wrong, and the cases 1, 0 and 0, 1 give minus 1 and answer 0, which is wrong as well.

You could keep trying numbers, but here is why it never works. Call the four totals T00, T10, T01 and T11, named after the inputs. With weights w1 and w2 and bias b, the totals are b, w1 plus b, w2 plus b, and w1 plus w2 plus b. Add T10 and T01: you get w1 plus w2 plus 2b. Add T11 and T00: you get the same thing, w1 plus w2 plus 2b. So T10 plus T01 always equals T11 plus T00. But exclusive-or needs T10 and T01 both greater than 0, so their sum is greater than 0, while it needs T11 and T00 both at 0 or below, so their sum is at 0 or below. One number cannot be both. So no choice of two weights and one bias gives the exclusive-or answer.

In one sentence: one neuron handles patterns like AND and OR but cannot handle exclusive-or, so networks use several neurons working together.

#Three Neurons Together

Here is the trick. Build exclusive-or from three neurons arranged in two steps. The first step has two neurons that both look at the inputs. The second step has one neuron that looks at the first two neurons' answers.

• Neuron OR: weights 2 and 2, bias minus 1. It says 1 when at least one input is on.
• Neuron AND: weights 2 and 2, bias minus 3. It says 1 only when both inputs are on.
• Neuron OUT: weights 2 and minus 2, bias minus 1, and it looks at the answers of OR and AND. Its weight on AND is negative, so AND being on argues against yes.

Exclusive-or means "at least one, but not both". OR says "at least one" and AND says "both", so OUT should say yes when OR is on and AND is off. Check it by hand. Fill in the table row by row.

Take the last row slowly. Both inputs are on. OR totals 2 plus 2 minus 1, which is 3, so OR answers 1. AND totals 2 plus 2 minus 3, which is 1, so AND answers 1. Now OUT sees 1 and 1. Its total is 1 times 2, plus 1 times minus 2, plus its bias of minus 1, which is minus 1. That is not greater than 0, so OUT answers 0. Exactly right: both on means exclusive-or says no.

Now check your table against the computer.

# Three neurons that together compute exclusive-or (exactly one of the two inputs is on)
def neuron(inputs, weights, bias):
    total = bias
    for x, w in zip(inputs, weights):
        total = total + x * w
    return 1 if total > 0 else 0

def tiny_network(a, b):
    h_or = neuron([a, b], [2, 2], -1)         # fires if at least one input is on
    h_and = neuron([a, b], [2, 2], -3)        # fires only if both are on
    out = neuron([h_or, h_and], [2, -2], -1)  # on when OR is on and AND is off
    return h_or, h_and, out

print("a b | or and | out")
for a in (0, 1):
    for b in (0, 1):
        h_or, h_and, out = tiny_network(a, b)
        print(a, b, "|", h_or, " ", h_and, " |", out)

Do: Run the cell. Then change the last neuron's bias from -1 to -3 and run it again.

Notice: The first run prints the out column as 0, 1, 1, 0, which is the exclusive-or pattern, and the or and and columns match your table. With -3 the out column prints 0 on every row.

Why: The last neuron's biggest possible total is 2 (OR on, AND off), and 2 minus 3 is below 0. A bias that is too negative leaves a neuron that can never say yes.

In one sentence: three neurons in two layers do what one cannot, because the first layer builds a better view of the inputs for the last neuron to use.

#Where Learning Fits In

So here is the picture to carry forward. A network is a pile of neurons in layers. Each neuron multiplies, adds and compares. The numbers inside, the weights and biases, decide everything it does. Training is how the right numbers are found.

In one sentence: you chose the weights by hand here, and in real networks training finds them.

#Where to Go Next

You can now work a neuron on paper, read a tiny network, and check it in code. The next lesson answers the question this one left open: how does a network find good weights?

• Next: How a Network Learns: Start Here. It shows what "how wrong" means, and how a network improves by small nudges.
• Then: Tensors & Automatic Differentiation gives you the tools real networks are built with, and Perceptron & Multilayer Networks goes deeper on what you met here.

This track has levels. This is the start, Level 0. After it come the foundations, then training, then vision, then sequences and attention, and then generative and advanced models.

In one sentence: read the learning lesson next, then work through the foundations in order.

#Key Takeaways

• A neural network is many small deciders called neurons, and each one multiplies, adds and compares
• A neuron multiplies every input by its weight, adds the products and the bias, and answers 1 only if the total is greater than 0
• A weight says how much an input counts, and a bias is the bar the total has to clear
• One neuron can do AND and OR but not exclusive-or, and three neurons in two layers can
• A hand trace gives you a prediction, and the code tells you whether you were right
• The weights are just numbers, and training is how a real network finds good ones

#Quick Check