A 100M-parameter network has more capacity to memorize than your dataset has examples. Without regularization it will memorize, and your 99% training accuracy will collapse to 60% on real data. Dropout, weight decay, label smoothing, Mixup — they all do the same job: punish the network for overfitting and force it to generalize. Each one is a single line of PyTorch and a 2x improvement on real benchmarks.
Learning Objectives
After this lesson, you will be able to:
Apply weight decay correctly in deep networks — and know why it must be decoupled from the gradient (AdamW), not folded into L2 like the original Adam
Use dropout the right way: invert-scale at training time, switch off at inference, and pick a sensible p for your network size and dataset
Soften your training targets with label smoothing to stop your model from screaming '99.99% confident' when it should not be — and explain why this also calibrates probabilities
Combine modern data augmentation tricks (Mixup, CutMix, RandAugment) and stochastic depth so your deep network actually generalizes instead of memorizing the training set
Don't worry if the toolbox feels overwhelming at first — most projects start with weight decay + a couple of augmentations, and you only reach for the heavier tricks (Mixup, stochastic depth) when you actually have a deep enough network for them to matter.
A deep network with a few million parameters can fit a few thousand training images perfectly. That is not learning — that is a lookup table dressed up in tensor form. Regularization is what stops the model from settling for the lookup table and pushes it toward representations that transfer to new data.
In track-03 you saw L2 regularization for linear and logistic regression: add λ‖w‖² to the loss and the optimizer pulls weights toward zero. In deep networks the same idea applies — every layer's weights are penalized for growing too large.
Ltotal=Ltask(θ)+2λi∑θi2
The catch: for adaptive optimizers like Adam, baking L2 into the loss interacts badly with the per-parameter learning rate. The "weight decay" you specified gets scaled by Adam's adaptive denominator, so parameters with large historical gradients get less decay than parameters with small gradients — the opposite of what you want.
θt+1=θt−ηm^t/(v^t+ϵ)−ηλθt
Practical defaults: for transformers and modern fine-tuning, AdamW with weight_decay=0.01–0.1 is the default. For SGD on CNNs, weight_decay=5e-4 is the long-running ResNet/ViT recipe. Always use AdamW, not Adam-with-L2.
The math reason inverted dropout works: at training, each neuron's output is multiplied by 1/(1-p)only when the mask keeps it. The expectation across the random mask is (1-p) · x · 1/(1-p) + p · 0 = x. So the activation distributions match between train and eval — no need to scale anything at inference.
Where to put dropout: traditionally between fully-connected layers (FC → dropout → FC). Modern practice has shifted: in transformers, dropout sits between attention output and the residual add, plus inside the MLP block. In CNNs, dropout often goes between blocks (or is replaced by stochastic depth, see below). Dropout on the input layer is rarely useful — use augmentation instead.
What Do You Think?
Your training accuracy is 78% and your validation accuracy is 85% — yes, validation is HIGHER than training. You have dropout=0.3 in the model. What's happening?
The answer: dropout actively damages training-time forward passes by zeroing 30% of activations. At evaluation, every neuron is back. So the model can score higher on val than train during training. As epochs accumulate, training accuracy climbs and eventually overtakes val (because of overfitting). Don't panic if early epochs show val > train when dropout is high.
When you train classification with one-hot targets and cross-entropy, the model is rewarded for outputting probability 1.0 on the correct class and 0.0 on every other class. To do that, the logit for the correct class has to be infinitely larger than every other logit. Models try — and the result is over-confident, badly calibrated probabilities (every prediction is 99.999% something).
qi′=(1−ε)⋅1[i=y]+K−1ε
Label smoothing has two effects worth knowing:
Calibration: The model's predicted probabilities become closer to actual frequencies. A 90% prediction is right ~90% of the time, instead of the 99% confidence / 90% accuracy mismatch you get with one-hot. This is huge for production where downstream systems (alerts, thresholds, costs) depend on probabilities meaning what they say.
Tighter clusters: Müller, Kornblith, and Hinton 2019 showed label smoothing pulls all examples of the same class toward the same point in the penultimate layer — the embedding space gets cleaner clusters with bigger gaps. It also makes knowledge distillation worse (a known tradeoff).
When NOT to use label smoothing: if you are training a teacher model that will distill into a student, smoothing erases the soft logit information the student would learn from. Otherwise, default it on at 0.1.
Train until validation loss stops improving, then stop. That is early stopping. It is so simple it sounds trivial, but it is one of the most reliable regularizers ever invented — Goodfellow's textbook calls it "the most commonly used form of regularization in deep learning" and it has zero hyperparameter tuning beyond a patience value.
pythonreference · read-only
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
best_val_loss = float("inf")
patience, patience_counter = 5, 0
for epoch in range(max_epochs):
train_one_epoch(model, train_loader)
val_loss = evaluate(model, val_loader)
if val_loss < best_val_loss:
best_val_loss = val_loss
torch.save(model.state_dict(), "best.pt") # checkpoint the best
patience_counter = 0
else:
patience_counter += 1
if patience_counter >= patience:
print(f"Early stop at epoch {epoch}, best val loss {best_val_loss:.4f}")
break
model.load_state_dict(torch.load("best.pt")) # restore the winner
The trick is always restore the best checkpoint, not whatever weights you happened to have at the moment of stopping. Once validation loss starts climbing, the model is overfitting; the best generalizing weights are already behind you on disk.
#Data Augmentation: The Quietly Strongest Regularizer
In practice, the single most impactful regularizer for vision and audio is data augmentation — apply random transformations to inputs at training time so the model sees a slightly different version of every example every epoch.
Photometric: color jitter (brightness/contrast/saturation/hue), Gaussian blur, grayscale conversion
Erasing: RandomErasing (Zhong et al. 2020) — black out a random rectangle, forces robustness to occlusion
Learned policies: AutoAugment (RL-searched augmentation pipeline), RandAugment (just pick N random ops with magnitude M — works almost as well, far simpler), TrivialAugment (one random op, even simpler, often best)
For text: random token deletion, word swap, back-translation (translate to French and back), synonym replacement (NLPAug, TextAttack libraries).
For audio: time masking, frequency masking (SpecAugment), pitch shift, time stretch, additive noise.
The principle is universal: augment along the dimensions that should not change the label. Flipping a cat image horizontally is still a cat. Translating "the movie was great" to French and back is still positive sentiment. Anything that changes the label (rotating a "6" upside down to a "9") is a mistake.
#Mixup and CutMix: Convex Combinations of Examples
x~=λxa+(1−λ)xb,y~=λya+(1−λ)yb,λ∼Beta(α,α)
CutMix (Yun et al. 2019) is the patch-swapping cousin: instead of pixel blending, cut a random rectangle from image B and paste it onto image A. The label blend ratio is the area ratio of the patches. CutMix tends to outperform Mixup on object-detection-style tasks because patches preserve local texture instead of averaging it out.
Stochastic Depth / DropPath (Huang et al. 2016): in deep ResNets, randomly drop entire residual blocks during training (i.e. let x + F(x) become just x). The network has to learn to function with arbitrary subsets of layers — equivalent to training a shallow ensemble. Default in modern ViT and ConvNeXt training recipes.
What Do You Think?
You train an MLP on a 200-row tabular dataset with dropout=0.5 between every layer. After 100 epochs, training accuracy is 60% and validation accuracy is 58%. What's the most likely problem?
The answer: dropout 0.5 zeroes half of every layer's activations every step. On a tiny dataset with a small network, this leaves so little signal that the model cannot even fit the training set. Drop dropout to 0.1 or remove it entirely; rely on weight decay + data augmentation instead. Regularization should match the gap between training and validation loss — if training loss is high, you don't have an overfitting problem and regularization is hurting you, not helping.
Tests · Verify the manual dropout matches PyTorch's official scaling. Confirm train mode produces stochastic outputs and eval mode produces deterministic ones.
#Putting It Together: A Modern Regularization Stack
For a fresh CIFAR-10 ResNet from scratch in 2025, the typical recipe is:
Data augmentation: random crop, horizontal flip, RandAugment(N=2, M=9), Mixup(alpha=0.2), random erasing
Architecture-level: stochastic depth (linearly increasing drop probability with depth, max 0.1)
Training-time noise: dropout=0.1 inside each block
Loss / optimizer: cross-entropy with label_smoothing=0.1, AdamW with weight_decay=0.05
Schedule: warmup 5 epochs, cosine decay over 200 epochs, early stopping with patience=20
For a fine-tuning task on a small custom dataset, dial almost all of this back: AdamW with weight_decay=0.01, light flip+crop augmentation, label_smoothing=0.1. Skip dropout (the pretrained model is already a heavy regularizer for your scale), skip Mixup (you don't have enough data for it to help).
Weight decay belongs in AdamW, not in the loss. Only AdamW applies decoupled weight decay correctly; vanilla Adam-with-L2 is broken because adaptive denominators warp the decay strength per parameter
Dropout is inverted-scaled at training time so inference needs no special handling. But you MUST call model.eval() before inference, or the random masks keep firing and your predictions become non-deterministic noise
Label smoothing fixes over-confident probabilities. Replacing one-hot targets with (1-ε, ε/(K-1), ..., ε/(K-1)) keeps logits bounded and produces calibrated outputs that downstream systems can actually trust
Data augmentation is usually the strongest regularizer in vision and audio. RandAugment + Mixup + horizontal flip will beat any architecture tweak, because it directly tells the model which transformations should not change the label
Match regularization strength to your overfitting gap. Heavy regularization on a small dataset with a small model means you can't even fit the training set; start light, add only when validation accuracy plateaus while training keeps climbing
You're training a transformer with Adam and `weight_decay=0.01` set on the optimizer. Why might switching to AdamW improve generalization without changing the value?
Regularization is the contract between your model and reality: don't memorize, generalize. Up next: Normalization — the family of tricks that keep activations and gradients in a sweet zone where every layer of a deep network can actually learn.