LeNet (1998), AlexNet (2012), VGG (2014), Inception (2014), ResNet (2015), EfficientNet (2019), ConvNeXt (2022). Each one was the new state-of-the-art for about 18 months, then someone published a better idea. The history is short and worth knowing — every trick that works in modern CNNs (skip connections, bottlenecks, depthwise convolutions, compound scaling) came from one of these papers, and they all influence the architectures you use today.
Learning Objectives
After this lesson, you will be able to:
Trace the CNN family tree from LeNet (1998) to ConvNeXt (2022) and explain what each architecture contributed — not just what it looked like
Understand why ResNet's skip connections solved the depth degradation problem and why almost every modern network borrowed them — including transformers
Compare parameter count, FLOPs, and accuracy across architectures so you can pick the right backbone for a given latency/accuracy budget
Apply EfficientNet's compound scaling principle (depth × width × resolution) when you need to make a model bigger or smaller without doing it ad-hoc
Recognize when MobileNet-style depthwise separable convolutions are the right tool for edge deployment, and when ConvNeXt-style modernized convs beat ViTs at the same FLOPs
Don't worry if the names blur together at first — every one of these architectures was the answer to one specific question, and once you know the question, the answer makes sense.
About 60K parameters. Trained on a CPU for days. The recipe — alternating convolutions and pooling, then flatten and feed to a classifier — has not changed in 26 years. What changed is everything else: bigger filters, more channels, more layers, better activations, normalization, residual connections, and a thousand training tricks.
ReLU instead of tanh — trained ~6× faster with no saturation problem
Dropout in the FC layers — first practical use, prevented overfitting on 1.2M images
Data augmentation (random crops, horizontal flips, PCA color jitter)
Two GPUs with model parallelism (NVIDIA GTX 580, 3GB each) — the model literally did not fit on one card
Local Response Normalization (later abandoned, but it worked at the time)
The result: 15.3% top-5 error on ImageNet vs. 26.2% for the runner-up. A 10-point gap on a benchmark where last year's improvement had been 1 point. The CV community switched paradigms in 12 months.
VGG made a single bet: just stack 3×3 convolutions and go deep. Instead of mixing filter sizes (5×5, 7×7, 11×11 like AlexNet), VGG used only 3×3 filters, but stacked many of them. Two stacked 3×3 convs see the same area as one 5×5 conv, but with fewer parameters and an extra non-linearity in between.
VGG-16: 16 weighted layers, 138M parameters
VGG-19: 19 weighted layers, 144M parameters
All filters 3×3, all max-pools 2×2
VGG was the canonical proof that depth matters more than filter design. It also became the workhorse feature extractor for years — perceptual loss in style transfer, GAN losses, and early object detectors all reached for VGG-16 features.
The cost: VGG was huge and slow. The FC layers alone had 100M+ parameters. Modern networks dropped them in favor of global average pooling.
#GoogLeNet / Inception (2014): Width Without the Bill
GoogLeNet (codenamed Inception, like the movie) won ImageNet 2014 with a different philosophy: why pick one filter size when you can use them all in parallel? The Inception module ran 1×1, 3×3, 5×5 convolutions and a 3×3 max-pool side by side, then concatenated their outputs along the channel axis.
The trick that made this affordable: 1×1 bottleneck convolutions before the expensive 3×3 and 5×5 paths. A 1×1 conv reduces the channel count cheaply (no spatial computation), letting the 5×5 filters operate on a slimmer input.
22 layers deep, but only 5M parameters (compared to VGG's 138M)
Auxiliary classifiers attached to intermediate layers, gradient injection during training
The 1×1 conv idea was reused everywhere thereafter (ResNet bottleneck, MobileNet, transformer FFN)
Going deeper than ~20 layers in a plain network actually hurt accuracy — and not because of overfitting. He et al. observed that a 56-layer plain CNN had higher training error than a 20-layer one, even though the deeper network could in principle just learn identity functions for the extra layers. The optimizer was failing to find the right weights.
The fix was disarmingly simple. Instead of asking each block to learn an arbitrary transformation H(x), ask it to learn the residualF(x) = H(x) − x, and add x back:
y=F(x,{Wi})+x
With residual connections, He et al. trained networks 152 layers deep and won ImageNet 2015. Then they trained a 1001-layer version on CIFAR for fun — and it still trained.
ResNet variants you will see in the wild:
ResNet-18 / 34: basic residual blocks, no bottleneck
ResNeXt: groups of parallel residual paths (cardinality > 1) — Facebook 2016
Wide ResNet: shorter but wider, sometimes beats deeper variants on small datasets
Try it! Open the Python REPL and type these lines yourself: import torchvision.models as m; r = m.resnet50(weights="IMAGENET1K_V2"); print(sum(p.numel() for p in r.parameters())/1e6, "M params") — you get a 25.5M-parameter ImageNet-pretrained ResNet-50 in two lines.
DenseNet took the skip connection to its logical extreme: every layer connects to every later layer in the block. Layer 5 sees the concatenated outputs of layers 1, 2, 3, 4 — not just layer 4. Feature maps are concatenated, not summed (unlike ResNet).
Strong gradient flow (every layer is one hop from the loss)
Strong feature reuse (early features always available to later layers)
Surprisingly parameter-efficient — DenseNet-121 matches ResNet-50 with half the parameters
Memory-hungry to train (concatenation grows the tensor) — that's the tradeoff
DenseNet is less common in production than ResNet, but its core idea — exposing all features to all later layers — shows up in U-Net skip connections, transformer cross-attention, and modern feature pyramid networks.
MobileNet asked a different question: how do we run a CNN on a phone? The answer was the depthwise separable convolution, which factorizes a standard convolution into two cheaper steps:
Depthwise: one 3×3 filter per input channel, applied independently. No mixing across channels.
Pointwise: a 1×1 conv that mixes channels.
The cost savings are dramatic. A standard 3×3 conv with M input and N output channels on a D×D feature map costs 9 · M · N · D² multiplications. The depthwise separable replacement costs 9 · M · D² + M · N · D² — typically 8–9× cheaper for the channel counts used in real networks.
By 2019 the field had a confusing problem: people were making CNNs deeper (ResNet-200), wider (Wide ResNet), or higher-resolution (ImageNet-21K), but each axis was tuned ad-hoc. Tan & Le ran a neural architecture search and discovered a simple rule: scale all three dimensions together with a fixed ratio.
depth:d=αϕwidth:w=βϕresolution:r=γϕsubject to α⋅β2⋅γ2≈2,α,β,γ≥1
EfficientNet-B0 (the smallest) was the result of an architecture search; B1–B7 are the same network scaled with this compound rule. EfficientNet-B7 hit 84.4% top-1 ImageNet accuracy with 66M parameters — beating much larger predecessors. The compound-scaling principle later became the template for scaling LLMs (Chinchilla scaling laws, GPT-4-class budgets).
By 2021, Vision Transformers (ViT) were the new hotness — and the conventional wisdom was that pure convolutions were obsolete. Liu et al. at FAIR pushed back with a deceptively simple paper: what if convs aren't the problem? What if it's just that ResNet was using 2015 training tricks?
They took a vanilla ResNet-50 and modernized it piece by piece, borrowing transformer-era tricks:
LayerNorm instead of BatchNorm
GELU instead of ReLU
Larger 7×7 depthwise kernels (matching the receptive field of an attention head)
Inverted bottleneck (channel expansion in the middle, like MobileNet V2)
Fewer activations and norms per block
Each change added 0.1–0.5%. Stacked together, ConvNeXt-B beat Swin-B (a strong ViT) at the same FLOPs. The lesson: architecture choices and training tricks are deeply coupled, and "transformers beat convs" was at least partly "modern recipes beat old recipes."
What Do You Think?
You train a 50-layer plain CNN on ImageNet from scratch. Then you train a 50-layer ResNet (same depth, same params) with the same hyperparameters. Both networks have enough capacity to overfit the data. What happens?
The right answer is the third one. Plain deep networks suffer from optimization-side degradation, not over-parameterization. Skip connections do not add parameters (the addition is free) — they change the optimization landscape so gradients can flow. Train the same depth with skip connections and the network learns; without them, it stalls.
Each CNN architecture answered one specific question. LeNet (do convs work?), AlexNet (do they scale?), VGG (does depth help?), ResNet (how do we go really deep?), MobileNet (can they fit on a phone?), EfficientNet (how do we scale principled?), ConvNeXt (are convs really obsolete?).
ResNet's skip connection is the single most-borrowed idea in deep learning. It lives in every transformer block, every diffusion U-Net, AlphaFold, WaveNet, and modern recurrent nets; the trick is changing the optimization landscape so gradients can flow, not adding capacity.
Compound scaling beats one-axis scaling. EfficientNet's depth · width · resolution rule replaced "just make it deeper," and the same principle later guided LLM scaling laws.
Depthwise separable convolutions are an ~8× FLOP saver for mobile but check wall-clock latency on real hardware — they can be bandwidth-limited on GPUs.
"Transformers beat convs" was partly a training-tricks story. ConvNeXt showed that modernizing a ResNet with LayerNorm, GELU, and large kernels closes the gap; architecture and recipe are deeply coupled.
Why did plain networks (no skip connections) deeper than ~20 layers actually perform WORSE than shallower ones in 2014, before ResNet?
The architecture is only half the story — the other half is what someone else already trained and is willing to share. Next up: Transfer Learning & Fine-Tuning, where you skip 90% of the work by standing on someone else's GPU bill.