Supervised learning was the dominant paradigm from roughly 2012 (AlexNet on ImageNet) until 2018 (BERT). It works — but it has a fatal scaling problem.
ImageNet has 1.2M labeled images. That's about the upper limit of what a research lab can afford to label. But unlabeled images? Common Crawl alone has billions. YouTube uploads more video every day than a human could watch in a lifetime.
The question SSL asks: can we design a learning objective that uses no human labels, yet produces representations as good as supervised ones?
For roughly a decade the answer was "kinda, but worse." Then between 2018 and 2022, SSL methods caught up to supervised learning and passed it on most downstream benchmarks. The recipe settled into three steps:
Pretrain a large model with an SSL objective on web-scale data (months on thousands of GPUs).
Fine-tune or linear-probe for the actual downstream task (hours on one GPU).
Optionally distill into a smaller production model.
This is how GPT works. This is how CLIP works. This is how DINOv2 works. This is how everything works now.
Modern SSL splits cleanly into three families. Each defines a different "pretext task" — a problem the model solves with no human labels, in hopes that the representations it builds along the way transfer to real tasks.
Idea: Two views of the same image; one network predicts the other's representation. No negatives needed.
BYOL (2020). Online network predicts target network's output; target is an EMA of online.
SimSiam (2021). Same idea, even simpler: just stop-grad on the target branch.
DINO (2021). Self-distillation with sharpening + centering tricks for ViT.
What Do You Think?
You're running SimCLR with batch size 32. For each anchor image, how many negative examples does the contrastive loss see?
#3. MAE: Masked Autoencoders Are Scalable Vision Learners
He et al., 2022. Possibly the single most influential vision SSL paper after CLIP.
The setup
Split an image into non-overlapping patches (e.g., 16×16 ViT patches).
Mask 75% of them, randomly.
Feed only the visible 25% to a ViT encoder.
A small ViT decoder takes (encoded visible patches + mask tokens) and predicts the raw pixels of the masked patches.
LMAE=∣M∣1i∈M∑∥xi−Dec(Enc(xvis))i∥22
Why 75% masking? This is the paper's most surprising result. In NLP, BERT masks 15% — because language is information-dense and 50% would destroy the sentence. Images are the opposite: highly redundant. A patch of sky predicts the next patch of sky. To force the network to learn something non-trivial, you have to delete most of the image.
Asymmetric encoder/decoder. The encoder is large (ViT-L or ViT-H) and operates on only 25% of patches — making pretraining ~3× cheaper. The decoder is small and discarded after pretraining; only the encoder is used downstream.
Chen et al., 2020. The paper that re-legitimized contrastive SSL by showing it could match supervised ResNet on ImageNet linear probe — given enough batch size and compute.
The pipeline
Take an image x. Apply two random augmentations (crop, color jitter, blur, flip) → x_i, x_j.
Encode both with a shared ViT/ResNet → h_i, h_j.
Project through a small MLP "projection head" → z_i, z_j.
Compute NT-Xent loss: pull z_i and z_j together, push z_i away from all other 2N − 2 views in the batch.
Why huge batches? Each anchor sees 2N − 2 negatives. SimCLR's strongest results used batch size 4096 or 8192 — meaning each anchor was contrasted against ~8000 negatives per step. Smaller batches give a weaker, noisier signal.
Temperature τ. Low τ (0.05–0.1) makes the softmax sharp, focusing the loss on the hardest negatives. Too low → unstable. Too high → loss becomes flat, indistinguishable from random.
Loading visualization...
#5. MoCo: Contrastive Learning Without Huge Batches
He et al., 2020. Same year as SimCLR; different solution to the same problem.
SimCLR's weakness: you need batch size 4096+ to get enough negatives. That requires TPUs or a small GPU cluster. MoCo replaces the giant batch with two tricks:
Momentum encoder. Keep two copies of the encoder: a "query" encoder (trained normally) and a "key" encoder (slowly updated as an EMA of the query). The key encoder produces stable representations of past examples.
Queue of negatives. Maintain a FIFO queue of ~65k key embeddings from previous batches. Each anchor contrasts against the entire queue.
The result: contrastive learning with batch 256, and 65,000 negatives per step.
Why momentum encoder, not just the same encoder? If you encoded queued negatives with the constantly-changing query encoder, their embeddings would be inconsistent across time. The momentum encoder evolves slowly (typical momentum: 0.999), so queue embeddings stay coherent for thousands of steps.
Grill et al., 2020. The result that broke the contrastive-SSL consensus.
The shocking claim: You don't need negative pairs at all. Just predict one augmented view from another.
Why this should not work. Without negatives, the trivial solution is "output the same vector for every input." Loss = 0. No information. This is called representation collapse, and it's the boogeyman of all SSL design.
Yet BYOL doesn't collapse. Here's the architecture:
BYOL has no negative pairs. What stops the network from collapsing to the trivial solution where every output is the same constant vector?
Quick check
A new paper claims a non-contrastive SSL method that drops both the predictor MLP and the stop-gradient. Should you believe it works?
#7. DINO: Self-Distillation Lights Up Object Parts
Caron et al., 2021. Same family as BYOL, but with a ViT and some specific tricks that produced one of the most visually striking results in vision SSL: emergent object segmentation without any segmentation labels.
The recipe
Two networks: student (trained) and teacher (EMA of student).
Multi-crop: 2 large "global" crops + several small "local" crops.
Student sees all crops; teacher sees only global crops.
Loss: cross-entropy between student's softmax output and teacher's softmax output.
Two tricks against collapse: centering (subtract running mean from teacher logits) and sharpening (lower teacher temperature than student).
The emergent result: the attention maps of the final ViT block, when visualized, segment foreground objects (and even object parts) cleanly. No segmentation labels were used. This is one of the most striking demonstrations that SSL representations capture meaningful structure.
DINOv2 (Oquab et al., 2023) scaled this to 142M images and a ViT-g/14 with 1.1B parameters. The features it produces are now used as a frozen backbone for many downstream tasks (depth estimation, semantic segmentation, instance retrieval) — often beating fully supervised baselines.
Radford et al., 2021. The single most consequential SSL paper for multimodal AI.
The objective: Given a batch of (image, caption) pairs, train an image encoder and a text encoder so that matching pairs have high cosine similarity and non-matching pairs have low similarity. It's literally NT-Xent across two modalities.
Training data: 400M (image, alt-text) pairs scraped from the web. No human annotation. The web's own captions are the supervision signal.
Zero-shot classification
To classify an image with CLIP:
Encode the image: v = image_encoder(img).
Encode each class name as a prompt ("a photo of a [class]"): t_i = text_encoder("a photo of a cat"), etc.
Predict the class with highest cos(v, t_i).
No fine-tuning. No class-specific training data. Just compare embeddings. CLIP achieved 76% zero-shot ImageNet accuracy — matching the original supervised ResNet-50, without ever seeing an ImageNet label.
This was the proof that SSL + scale + web data could leapfrog the supervised paradigm entirely. Every modern vision-language model descends from CLIP: Flamingo, Gemini, GPT-4V, Claude's vision.
What Do You Think?
MAE masks 75% of image patches. What happens if you push the mask ratio to 95%?
The latest generation of SSL methods is moving away from pixel-space reconstruction and toward prediction in representation space.
JEPA (Joint Embedding Predictive Architecture), LeCun 2022+. Don't predict pixels. Don't even predict patches. Predict the embedding of the masked region from the embedding of the visible region. This sidesteps two problems:
Pixel-level reconstruction wastes capacity on irrelevant detail (texture, lighting).
I-JEPA (Assran et al., 2023). JEPA for images. Pick a target block of patches, mask it, predict its representation from a visible context block. No pixel reconstruction, no negatives, no augmentations.
V-JEPA (Bardes et al., 2024). Same idea for video. Predict masked spatiotemporal blocks in representation space. The first method that genuinely learns motion and physics from raw video without labels.
DINOv2 (2023). Scaled DINO + I-JEPA-style objectives + curated 142M-image dataset. Produces features that work as a near-universal frozen backbone.
AudioMAE, VideoMAE. MAE generalizes naturally: mask audio spectrogram patches, mask video tubelets. Both achieve SOTA on their respective modalities.
Predictive coding revival. A 1980s neuroscience idea (Rao & Ballard 1999) reborn: representations should let you predict the next moment in sensory input. JEPA is essentially a modern, deep-learning-friendly form of predictive coding.
Where it's heading: the consensus is that the future is prediction-in-representation-space + world models + multimodal SSL. The 2026 frontier is no longer "pretrain a backbone" — it's "pretrain a foundation that simulates the world well enough that downstream tasks become trivial."
Loading visualization...
#10. Why SSL Works: Theory, Intuition, Open Questions
Empirically SSL works spectacularly. Theoretically the story is still developing. A few key ideas:
Both contrastive and reconstructive SSL force the encoder to compress inputs into representations that retain task-relevant information (predicting the masked patch, distinguishing positive from negative) while discarding irrelevant detail (exact noise, exact crop position). This is the information-bottleneck principle in action.
In contrastive SSL, the choice of augmentation is the inductive bias. SimCLR's authors explicitly say: SimCLR is learning to be invariant to "things that the augmentation pipeline considers irrelevant" (color jitter → color invariance; crop → location invariance). Different augmentations → different features.
A real subtlety: SSL representations are not automatically superior to supervised ones. They tend to be:
Better for transfer learning (especially when downstream data is small).
Better at OOD generalization in many vision benchmarks.
Comparable or worse on the original pretraining-distribution task.
For tasks where you have plenty of labels and inference distribution matches training, supervised may still win.
Quick check
You're choosing an SSL objective for pretraining a vision backbone that will be fine-tuned end-to-end on a medical segmentation task. Which family is most likely the strongest choice?
SSL is the dominant pretraining paradigm now. Every frontier model is built on self-supervised pretraining: GPT, Claude, Gemini, CLIP, DINOv2, MAE, SAM, JEPA. Supervised learning is now a fine-tuning step, not a pretraining strategy.
The three families have different sweet spots. Generative SSL (BERT, MAE) tends to win at fine-tuning on dense tasks. Contrastive SSL (SimCLR, CLIP) tends to win at linear probe, retrieval, and multimodal. Non-contrastive SSL (BYOL, DINO) wins at instance-level features without needing huge batches.
Collapse is the enemy. Every non-contrastive SSL design is, in some way, an answer to the question: what stops the network from outputting a constant? Architectural asymmetry (BYOL), distribution sharpening (DINO), explicit variance regularization (VICReg) — all are anti-collapse mechanisms.
The frontier is moving toward representation-space prediction (JEPA family) and world-model-style SSL. By the time you next read about SSL, MAE will probably look as quaint as word2vec does today.