In 2020, Dosovitskiy's team at Google asked a heretical question: what if you treat an image like a sentence, chop it into 16×16 patches, and shove them through a vanilla transformer? CNNs had ruled computer vision for a decade. ViT replaced them. Every multimodal model you use — GPT-4o, Claude Sonnet 4.6, Gemini 2 Pro — uses a Vision Transformer (or close cousin like SigLIP) as the eye that lets the LLM see.
Learning Objectives
After this lesson, you will be able to:
Slice an image into 16x16 patches, embed them as 'visual words,' and feed them to a standard transformer encoder — the entire ViT architecture in one breath
Diagnose when ViT will lose to a CNN (small data, edge inference) versus when it wins decisively (300M+ images, multimodal models, foundation backbones)
Pick the right ViT variant for the job — vanilla ViT for scale, DEiT when data is scarce, Swin when you need hierarchical CNN-like locality, MAE for self-supervised pretraining
Read attention maps from a trained ViT to understand what the model is looking at — and use that to debug failures or build interpretable medical/scientific systems
Don't worry if "treat an image like a sentence" sounds weird — once you see the patch-embedding step, the rest of ViT is the same transformer encoder you already understand from BERT.
The Vision Transformer (ViT) was introduced in 2020 by Google Research with a paper titled "An Image is Worth 16x16 Words." The deliberately provocative title summarizes the architecture exactly: take an image, slice it into patches, treat each patch as a word, and run BERT.
Take a 224x224 RGB image. Divide it into a grid of 14x14 patches, each 16x16 pixels. That gives 196 patches total. Each patch has 16 × 16 × 3 = 768 raw pixel values. Flatten each patch into a 768-dim vector and project it through a learned linear layer to produce a d_model-dim patch embedding.
Prepend a learnable [CLS] token to the sequence (its final hidden state will represent the whole image — used for classification). Then add a learned 1-D positional embedding to each token, including the CLS token.
Feed z_0 through L transformer encoder blocks (typically 12 for ViT-Base, 24 for ViT-Large, 32 for ViT-Huge). Each block is exactly the BERT block: multi-head self-attention → add & LayerNorm → FFN → add & LayerNorm.
After the final layer, take the CLS token's hidden state, pass it through a linear head, and you have your image classifier. That's the entire architecture.
The 2021 surprise was not that transformers can do vision — researchers had tried that before. The surprise was that with enough data, ViT outperforms the best CNNs. Specifically:
Trained on ImageNet alone (1.2M images): ResNet-152 beats ViT-Base.
Pretrained on ImageNet-21k (14M images): roughly tied.
Pretrained on JFT-300M (300M images): ViT beats every CNN on every benchmark.
What Do You Think?
You have 5,000 labeled medical images of skin lesions and need to build a classifier. Which is most likely to win?
The right answer is DEiT or DINOv2 fine-tuning. With only 5K images, training any transformer (ViT or Swin) from scratch wildly underperforms — you don't have the data for a transformer to learn the visual basics. A pretrained ViT, however, has those basics from millions of images and only needs to specialize. Fine-tuning a DEiT-Base on 5K medical images routinely outperforms training a ResNet from scratch by 5-15% accuracy.
The original ViT paper had an inconvenient asterisk: the JFT-300M dataset is private to Google. Without it, ViT was inferior to CNNs on every public benchmark — making the architecture a research curiosity for most of the field.
DEiT (Touvron et al. 2021, Facebook AI) fixed this with three tricks:
Strong augmentation (RandAugment, mixup, cutmix, random erasing) — every transformer needs this for vision.
Knowledge distillation from a CNN teacher — they added a special [DIST] token next to [CLS], trained to match a RegNet teacher's output. The CNN's inductive bias gets transferred through the distillation loss.
Tuned hyperparameters for ViT specifically (learning rate, weight decay, augmentation strength).
Result: DEiT-B trained only on ImageNet-1k matches ViT-B trained on ImageNet-21k. The distillation token is dropped at inference.
Plain ViT uses global attention at every layer — every patch attends to every other patch. That's O(N² · d) per layer, which becomes painful at high resolution (e.g. 1024x1024 images give 4096 patches → 16M attention scores per head per layer).
Swin Transformer (Liu et al. 2021) makes attention local:
Divide the image into windows of, say, 7x7 patches.
Inside each window, run normal multi-head self-attention. Cost: O(N · M² · d) where M is window size — linear in N.
Shift the windows by M/2 between alternating layers. Patches that were in different windows in layer k end up in the same window in layer k+1. Information crosses boundaries without ever needing global attention.
Patch merging between stages downsamples the resolution and increases channels — exactly like CNN downsampling stages.
ΩViT=4ND2+2N2DΩSwin=4ND2+2M2ND
This makes Swin the de-facto choice for dense prediction tasks — object detection, instance segmentation, semantic segmentation — where high-res inputs matter.
#MAE: Self-Supervised Pretraining, Cheap and Effective
Supervised pretraining (training a ViT to classify ImageNet labels) requires labels. Labels are expensive. Worse, classification-pretrained features often don't transfer well to dense tasks.
Masked Autoencoder (He et al. 2021) uses self-supervision instead:
Randomly mask 75% of the patches of an input image.
Run the encoder (a standard ViT) on only the visible 25% of patches.
A small decoder receives the encoder output plus learnable mask tokens at the masked positions and tries to reconstruct the missing pixels.
After pretraining, throw away the decoder. The encoder is now a strong general-purpose backbone.
Why MAE works:
The 75% masking ratio is much higher than BERT's 15%. Vision is much more spatially redundant than text — you can recognize a cat from any quarter of the image. With low masking, the model just interpolates from neighbors and learns nothing useful.
The decoder is small (≤ 8 layers) and only used during pretraining. The encoder, which is the part you keep, processes only 25% of tokens — making pretraining 3-4x cheaper than supervised pretraining at the same model size.
The result: ViT-Huge pretrained with MAE on ImageNet (no labels) outperforms ViT-Huge trained with supervised classification, on every downstream task.
If MAE is the most efficient pretrainer, DINOv2 (Meta 2023) gives the strongest features. It uses a self-distillation loss without negative samples: a student network learns to match a teacher network (an EMA of past student weights) on multiple augmented crops of the same image. The features that result are remarkably general — frozen DINOv2 features beat fine-tuned ResNet-50 features on most evaluation benchmarks.
Today, when someone says "use a ViT backbone for X," they almost always mean a DINOv2 or SigLIP encoder. ResNet-50 has been retired from the same role.
Tests · Verify that the output shape is (batch_size, num_patches + 1, d_model) and that swapping cls_token / pos_embed for zeros breaks classification accuracy.
ViT treats images as sequences of patches. Slice into 16x16 patches, flatten, embed, add CLS token + position, run a standard transformer encoder. The architecture is BERT applied to image patches.
ViT needs scale to win. At small data (≤1M images, ≤10K samples for fine-tuning) CNNs still beat ViT trained from scratch. ViT only wins when pretrained on >14M images. Always start from a pretrained backbone.
DEiT solved ViT's data hunger. Knowledge distillation from a CNN teacher made ViT trainable on plain ImageNet. The distillation token is the trick.
Swin brings CNN-like hierarchy back. Shifted local windows make attention linear in image size and bring back inductive bias for spatial structure. The default for detection and segmentation.
Self-supervised pretraining (MAE, DINOv2) is the modern default. Frozen DINOv2 features outperform fine-tuned ResNet-50 features on most downstream tasks. Self-supervision, not labels, is now where ViTs get their strength.
You have a 224x224 RGB image and patch size 16. How many tokens enter the transformer encoder?
Now you understand how transformers see. Next up: how vision and language are stitched together — CLIP, LLaVA, Flamingo, and the multimodal models that turned LLMs into systems that can describe a photo, read a chart, or watch a video.