A neural network is a stack of matrix multiplications with a sprinkle of non-linearity. That's it. By the end of this lesson, you'll trace a forward pass through a 2-layer net by hand, prove for yourself that XOR is impossible with one neuron, and understand exactly why depth is the secret ingredient behind every model from ResNet to GPT-4.
Learning Objectives
After this lesson, you will be able to:
Trace how biological neurons inspired the McCulloch-Pitts model and Rosenblatt's perceptron, and where the analogy breaks down
Build a perceptron from scratch and prove for yourself why a single perceptron cannot solve XOR (Minsky-Papert 1969)
Stack neurons into a multilayer perceptron and follow the matrix-form forward pass: h = σ(Wx + b)
Vectorize the forward pass across a batch of inputs (H = σ(XWᵀ + b)) and verify dimension compatibility at every layer
State the Universal Approximation Theorem (Cybenko 1989, Hornik 1989) and explain why depth wins over width in practice
Diagnose the 'collapse to one linear layer' bug when nonlinearity is missing between hidden layers
You are about to learn the single most important building block in modern AI. If anything feels confusing on first read, that is normal -- researchers took decades to get this right. Take it one step at a time, and the pieces will click.
σ = activation function (sigmoid, tanh, ReLU, ...)
y = output
The weights and bias are the parameters that the network learns during training. They start random and are nudged by gradient descent until the neuron produces useful outputs.
Try it! Implement the neuron function from the code above. Feed in x = [1, 2], w = [0.5, -0.3], b = 0.1, and use np.maximum(0, z) (ReLU) as your activation. Change the weights and watch how the output changes. You are manually tuning a one-neuron neural network.
That's literally the logistic regression model from the Classical ML track. A 1-neuron neural network IS logistic regression. Everything new in this track is what happens when you start stacking these units.
What Do You Think?
What happens if you remove the activation function entirely (i.e., use the identity, σ(z) = z)?
Without an activation function, a neuron computes y = w^T x + b, which is a linear function. Stack a thousand linear neurons in a hundred layers, and the entire network is still just one big linear function -- the composition of linear maps is linear. The activation function is what gives neural networks their power.
If predicted 0 but true label is 1: add the input to the weights (w = w + x). This rotates the decision boundary toward correctly classifying this point.
If predicted 1 but true label is 0: subtract the input (w = w - x). This rotates it away.
#Stacking Neurons: The Multilayer Perceptron (MLP)
A neural network is neurons organized in layers:
Input layer: receives the raw data (not really "neurons" -- just the data passed through)
Hidden layers: the computational layers where learning happens
Output layer: produces the final prediction
In a fully-connected (or "dense") layer, every neuron connects to every neuron in the next layer. Each connection has its own weight.
Try it: Add neurons and layers and watch how the network's capacity growsInteractive
Loading visualization...
Try this: Watch how data flows from input through hidden layers to the output. Notice how each layer transforms the data -- the first hidden layer typically separates broad patterns, while deeper layers extract finer distinctions.
Computing one neuron at a time is fine for intuition. In practice, we group all the neurons in a layer into a matrix and compute them in one shot.
h=σ(Wx+b)
For a layer with 128 inputs and 64 outputs, W has 128 × 64 = 8,192 parameters, plus 64 biases. Each of the 64 neurons computes a different weighted combination of all 128 inputs in parallel, all expressed as one matrix multiply.
In practice, we don't process one input at a time -- we process a batch of N inputs as a matrix.
H=σ(XWT+b)
Processing 32 inputs in a batch is barely slower on a GPU than processing 1, because the GPU parallelizes across the batch dimension. This is why batch size is an important hyperparameter -- larger batches squeeze more out of GPU memory.
A feedforward network with one hidden layer of sufficient widthcan approximate any continuous function on a compact subset of Rn to arbitrary accuracy.
Proven independently by George Cybenko (1989) and Kurt Hornik (1989, 1991), this theorem is both reassuring and misleading:
Reassuring: neural networks are theoretically capable of representing any pattern.
Misleading: it says nothing about how many neurons you need, how to train them, or whether gradient descent will find the right weights.
In practice, deeper networks (more layers with fewer neurons each) work dramatically better than wider networks (one layer with many neurons). Depth allows hierarchical feature extraction -- each layer builds on abstractions from the previous one.
#Train vs Inference: The Forward Pass Plays Both Roles
The forward pass is one direction: input flows to output. By itself, no learning happens -- the forward pass just makes a prediction using the current weights.
Inference (prediction time): only the forward pass runs. Input → output, done.
Training (learning time): forward pass produces a prediction → loss function measures error → backward pass (next lesson) computes gradients → weights get updated.
Some layers behave differently in train vs inference mode (Dropout, BatchNorm). That's a topic for the regularization and normalization lessons later in this track. For now, internalize this: the forward pass is the same operation in both modes; what differs is what happens around it.
#Build It Yourself: Perceptron + 2-Layer XOR Solver
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
Tests · Verify the perceptron converges on AND. Verify it does NOT converge on XOR. Verify the 2-layer MLP returns [0, 1, 1, 0] for the four XOR inputs.
An artificial neuron computes y = σ(w·x + b). A weighted sum followed by a nonlinear activation. This single formula, repeated billions of times in stacked layers, is the entire computational substrate of modern AI.
A single perceptron can only solve linearly separable problems. It draws one straight line through feature space. XOR breaks it. The fix is depth: stacking layers with nonlinear activations creates curved boundaries.
A layer's forward pass is one matrix multiply: h = σ(Wx + b). Stacking L layers gives an L-step matrix-multiply chain. Vectorizing across a batch (H = σ(XWᵀ + b)) is what GPUs are built to accelerate.
The Universal Approximation Theorem says one hidden layer can in principle approximate anything — but in practice, depth gives exponential efficiency gains over width, and modern networks are deep, not just wide.
Without nonlinear activations between layers, an N-layer network collapses to 1 layer. Composition of linear functions is linear. Activation functions are not optional decoration; they are what make depth meaningful.
You now understand the unit (the artificial neuron), the architecture (stacked layers), and the operation (matrix-form forward pass). But the activation function is the secret ingredient — choose wrong and your network won't train. Next lesson: Activation Functions.
Cycle through all training examples. Each pass is an epoch. If the data is linearly separable, the Perceptron Convergence Theorem (Rosenblatt 1958) guarantees the algorithm converges in a finite number of steps.
The perceptron can only learn linear decision boundaries -- straight lines in 2D, flat planes in 3D. If the data requires a curved boundary (like XOR), the perceptron will never converge. This was Minsky and Papert's devastating result.