Before 2010, training a 20-layer network was nearly impossible — the gradient would shrink to ~10⁻³⁰ before reaching the first layer. The fix wasn't a new architecture or a smarter optimizer. It was changing the starting numbers. Glorot in 2010, He in 2015. Two papers, one idea: pick your initial weights so that signal variance stays constant through the network. That alone unlocked depth.
Learning Objectives
After this lesson, you will be able to:
Explain why gradients can vanish or explode as they flow backwards through deep networks, and how that explains why deep nets failed to train before 2010
Derive the variance-propagation argument behind Xavier/Glorot init for sigmoid/tanh and He/Kaiming init for ReLU — and pick the right one for the activation you are using
Spot 'dying ReLU' and 'symmetry not broken' bugs by looking at activation distributions per layer instead of just the loss curve
Use orthogonal init for RNNs and the right gain factor for non-default activations, and know when batch normalization makes init less critical (and when it still matters)
Don't worry if "variance preservation" sounds intimidating — once you see what happens to a signal when each layer multiplies it by 0.5 thirty times, the whole thing becomes obvious.
The whole game is keeping the variance of activations stable as data flows forward, and the variance of gradients stable as they flow backward. If you can preserve both, deep training works. If you can't, it doesn't.
Consider a single linear layer: y = W x + b. Assume the inputs x are zero-mean with variance σ²ₓ, the weights W are zero-mean with variance σ²_W, and they're independent. Then for each output unit y_j:
Var(yj)=nin⋅Var(W)⋅Var(x)
So the rule for the forward pass is Var(W) = 1/n_in. But there's a second constraint coming from backprop. The gradient flowing backward through a linear layer is multiplied by Wᵀ, so by symmetric reasoning we need Var(W) = 1/n_out to keep gradient variance stable.
You can't satisfy both unless n_in == n_out, so Glorot and Bengio compromised: average the two. That's Xavier init.
Xavier (Glorot):Var(W)=nin+nout2
The Xavier derivation assumes the activation is roughly linear near zero — true for sigmoid and tanh. Not true for ReLU, which kills half the neurons (those with negative pre-activations output zero). When half the neurons are dead, the effective fan-in is halved, so we need to double the variance to compensate.
He (Kaiming):Var(W)=nin2
What Do You Think?
You build a 50-layer ReLU network and initialize the weights using Xavier init (Var(W) = 1/n_in instead of 2/n_in). What happens to the activation variance as you go deeper?
The factor of 2 is not cosmetic. With Xavier on ReLU, every layer drops activation variance by half. After 50 layers you're at 10⁻¹⁵ -- numerically zero. The network has no signal to train on, and no fancy optimizer can rescue it.
Sweep the weight scale and activation choice to watch signal variance hold steady, vanish, or explode as it propagates layer by layer.
Before any of the variance math kicks in, init has one more job: break symmetry. If every weight in a layer starts at exactly the same value (say, all zeros, or all 0.01s), every neuron in that layer computes the same function. They get the same gradient. They update by the same amount. They stay identical for all of training. You effectively have one neuron per layer.
This is why you cannot initialize weights to zero. Bias terms are different — you can zero those because the input dimensionality already breaks their symmetry — but weights must start with random asymmetry. Every modern init scheme is "draw from some random distribution with the right variance."
Tests · Verify that the zero-init + sigmoid configuration produces zero variance at every layer. Verify that ReLU + Xavier collapses to ~10^-15 by layer 30. Verify that ReLU + He stays in the 0.1 - 10 range across all 30 layers.
Even with perfect init, deep sigmoid/tanh networks struggle because their derivative saturates -- a sigmoid neuron with input 5 has derivative ≈ 0.007, and that derivative gets multiplied into every gradient flowing through. ReLU's derivative is exactly 1 for any positive input, which means gradient magnitude is preserved through the chain rule (rather than being attenuated by 0.007 per layer).
But ReLU brings its own pathology: dying ReLU. If a neuron's pre-activation drifts negative for the entire training distribution, its output is always zero, its gradient is always zero, and it can never recover. Deep ReLU networks routinely have 30-50% dead neurons by the end of training. Fixes:
Leaky ReLU: small negative slope (f(x) = max(αx, x) with α=0.01) -- gradient is never exactly zero
ELU / SELU: smooth negative tail -- self-normalizing under specific init
GELU (transformers' favorite): probabilistic gating that's mostly ReLU but smooth near zero
Swish / SiLU (mobile-friendly): x · sigmoid(x), smooth and self-gated
Each of these comes with its own ideal init gain. PyTorch's torch.nn.init.calculate_gain(nonlinearity) looks up the right multiplier for you: 5/3 ≈ 1.667 for tanh, sqrt(2) ≈ 1.414 for ReLU, and 1.0 for 'linear', 'sigmoid', and the conv families. Tanh is the one people misremember as 1.0 — it isn't, and hardcoding 1.0 there will quietly shrink your activations layer over layer. When in doubt, call calculate_gain rather than typing a constant.
The standard story breaks down in two important regimes:
Recurrent networks apply the same weight matrix W over many time steps. After 50 time steps, you're effectively raising W to the 50th power. If W has any eigenvalue with magnitude ≠ 1, the result either vanishes or explodes -- and Xavier/He are derived for one application of W, not 50. The fix: orthogonal initialization (Saxe et al. 2013). An orthogonal matrix has all eigenvalues with magnitude exactly 1, so Wⁿ stays bounded for any n.
Skip-connection architectures (ResNet, transformers) add the input directly to the output: y = x + F(x). This is great because it gives gradients a "highway" -- the gradient can flow back through the skip connection without any weight multiplication, so vanishing is impossible no matter how deep. He init still matters for the residual branch F(x), but the network's depth-tolerance comes mostly from the skip itself. Fixup init (Zhang et al. 2019) showed you can train arbitrarily deep skip-connection nets without batch normalization if you scale the residual branches correctly at init -- a hint that BN is mostly a crutch for poorly-initialized networks.
Once batch normalization (next lesson) re-centers and re-scales activations after every layer, init starts to matter less. BN absorbs the variance-preservation job that init was doing, and you can get away with less careful schemes. This led to a brief era (~2016) where people stopped thinking about init.
But init is making a comeback in two contexts:
Layer normalization (transformers) doesn't re-scale across the batch -- it normalizes per token. So BN's "automatic correction" doesn't fully apply, and init still matters.
Very large models (>10B parameters) often skip BN/LN entirely on the residual branch and rely on careful init + scaling instead. Fixup, T-Fixup, and ReZero are all post-2019 schemes that revisit init for billion-parameter models.
The practical takeaway: init still matters, just in narrower regimes than it used to. And when it bites you, it bites silently -- you don't get a Python error, you get a model that trains 5% worse for reasons no one can explain. Logging activation variance is the cheapest diagnostic in deep learning.
Vanishing/exploding gradients are a variance-propagation problem, not a fancy optimizer problem — when forward signal variance shrinks layer-by-layer, the same matrices in backprop shrink the gradient by the same factor; no optimizer can recover what isn't there
Xavier init for sigmoid/tanh, He init for ReLU and friends. The factor of 2 difference comes from ReLU killing half the neurons; using the wrong one for your activation produces silent failure, not a crash
Symmetry must be broken at init. Zero-initialized weights mean every neuron in a layer learns identically, collapsing the layer's effective capacity to 1; bias terms can be zeroed but weights cannot
RNNs need orthogonal init, skip-connection nets need scaled-residual init. The standard variance arguments break down when you apply a weight matrix many times (RNNs) or stack hundreds of residual blocks (very deep ResNets / transformers)
Batch norm reduces but does not eliminate the importance of init. Modern transformers, billion-parameter models, and LayerNorm-only architectures still need careful init; logging per-layer activation variance is the cheapest diagnostic you can add
Why does ReLU need He initialization (Var(W) = 2/n_in) instead of Xavier (Var(W) = 2/(n_in + n_out))?
Init is the silent enabler of every deep network you train. Next: how batch normalization, layer normalization, and group normalization stabilize activation distributions during training — and why they make some init choices forgiving and others unforgivable.