Stack a thousand linear layers and you still have one linear layer — the algebra collapses. Activation functions are the bent piece of glass between layers that turns straight lines into curves. Get this choice wrong (sigmoid in a 50-layer net) and training fails silently. Get it right (GELU in a transformer) and you get GPT-4.
Learning Objectives
After this lesson, you will be able to:
Compare the main activation functions (sigmoid, tanh, ReLU, GELU, SiLU) -- their shapes and when to use each
Understand why ReLU became the default and why newer models prefer GELU or SiLU
Spot the 'dead neuron' problem with ReLU and know the three standard fixes
Select the correct activation for hidden layers vs. output layers based on task type (classification, regression, binary)
Activation functions might sound like a small detail, but they are one of the most consequential choices in designing a neural network. Getting comfortable with them now will pay off in every lesson that follows. You have got this.
Try it! Compute sigmoid(0), sigmoid(5), and sigmoid(-5) by hand or in Python. Notice how the output is always between 0 and 1, and how extreme inputs get "squashed" to nearly 0 or 1. Now compute the derivative at each point -- see how the gradient nearly vanishes at the extremes?
Pros: Eliminates vanishing gradients for active neurons (derivative = 1), computationally trivial (just a comparison), enables training of very deep networks, induces sparsity (many neurons output zero).
Cons: Dead neurons -- if a neuron's input is always negative (due to bad initialization or a large gradient update), it outputs zero forever and never recovers. Its gradient is zero, so no learning signal reaches it. In extreme cases, up to 40% of neurons can die.
What Do You Think?
In a network with ReLU activations, what percentage of neurons typically have zero output for any given input?
Approximately 50% of neurons output zero for any given input (assuming roughly symmetric weight initialization). This sparsity is actually beneficial -- it means only a subset of neurons activate for each input, creating a sparse, efficient representation. But permanently dead neurons (zero for ALL inputs) are a problem.
Pros: Smooth everywhere (no kink at zero), combines gating and activation, empirically superior for transformers. Used in GPT, BERT, and most modern LLMs.
Cons: More expensive to compute than ReLU. The approximation formula is complex.
By 2023, the FFN sublayer in every flagship LLM had migrated from a plain Linear -> activation -> Linear to a gated variant. Llama, PaLM, Mistral, Gemma, and DeepSeek all use SwiGLU. The change is small in code and substantial in quality, so it is worth knowing the actual formula rather than treating it as a black box.
The original Gated Linear Unit (GLU), proposed by Dauphin et al. 2017, replaces the single projection of a feed-forward layer with two parallel projections and an elementwise sigmoid gate:
GLU(x)=(W1x)⊙σ(W2x)
SwiGLU (Shazeer 2020) keeps the structure identical and only swaps the sigmoid gate for a Swish (a.k.a. SiLU) gate. GeGLU swaps it for GELU. In both cases, the gating activation is now smooth and non-monotonic, which empirically beats plain sigmoid by a small but consistent margin and stacks with the standard "smooth activation in transformers" intuition you already have from GELU vs ReLU.
The FFN dimension trick. A standard transformer FFN is Linear(d -> 4d) -> activation -> Linear(4d -> d) -- two matrices of size d x 4d, total 8 d^2 parameters. Swapping in SwiGLU adds a third matrix (the gate projection), which would naively inflate parameter count by 50 percent. Production architectures compensate by shrinking the hidden dimension from 4d to (2/3) x 4d, i.e. roughly 2.67 d. The three matrices of size d x (2.67 d) then sum to 8 d^2 parameters again -- matching the plain FFN budget. This 8/3 hidden-dim ratio is what you see hard-coded in Llama's source: intermediate_size = int(2 * 4 * dim / 3) rounded to a multiple of 256 for kernel alignment.
Compare ReLU and sigmoid side by side. Notice how sigmoid's derivative (the orange dashed line) is always below 0.25 -- this is why gradients vanish. Then switch to ReLU and see the derivative is exactly 1 for positive inputs. Finally, look at GELU -- notice how it smoothly transitions, unlike ReLU's sharp kink at zero.
Without activation functions, deep networks collapse to a single linear transformation. Nonlinear activations between layers are what give neural networks the power to learn complex, curved decision boundaries
ReLU became the default because it avoids vanishing gradients. Its gradient is either 0 or 1, enabling efficient training of deep networks, unlike sigmoid/tanh which saturate and shrink gradients
Dead neurons are ReLU's biggest weakness. Once a neuron's input becomes permanently negative, its gradient is always zero and it never learns again; Leaky ReLU and ELU fix this by allowing small gradients for negative inputs
GELU and Swish are preferred in transformers. Their smooth, non-monotonic shape provides better gradient flow and empirically improves performance in attention-based architectures like GPT and BERT
Why did ReLU replace sigmoid as the default hidden layer activation?
You now understand the nonlinear functions that give neural networks their power. Next: Training Dynamics -- batch normalization, dropout, learning rate scheduling, and all the tricks that make deep network training actually work in practice.