Classification answers "what is this picture?" Detection answers "what is in this picture, where, and how many?" Segmentation answers "for every single pixel, what does it belong to?" The architectures that solve these three problems share a backbone — CNN or ViT — and then layer on increasingly clever heads. This lesson follows that ladder. We start with Vision Transformers as a bridge from convolutions, then climb through R-CNN to YOLO to DETR for detection, and through U-Net to SAM for segmentation. By the end you will know which architecture to reach for when the task is no longer "is it a cat?"
You have spent the prior lessons mastering the CNN family — from LeNet through ResNet to ConvNeXt — all aimed at one task: assign a single label to a single image. Almost no real-world computer-vision system stops there.
A self-driving car needs to know there are three pedestrians in the frame, and where each one is, to plan a path.
A radiologist's tumor-screening tool must highlight the exact pixels of a suspicious lesion, not just say "this scan looks bad."
An agricultural drone counting corn cobs across a field needs to localize and count every cob, regardless of how many appear.
A satellite-imagery analyst tagging deforestation needs per-pixel land-cover labels across thousands of square kilometers.
Adobe Photoshop's generative fill needs to know exactly which pixels the user wants replaced when they paint a rough lasso.
All four of these are detection or segmentation problems, not classification. Different architectures, different losses, different metrics. This lesson maps the territory.
The transformer self-attention lesson and the dedicated Vision Transformers lesson in the NLP-and-Transformers track cover the ViT mechanics in depth. We will not duplicate them — instead, this section sketches the architecture quickly so you can recognize it inside detection and segmentation pipelines.
Try it: Slice an image into ViT-style patchesInteractive
Loading visualization...
The patch grid is the only place ViT differs from a BERT encoder. After patch embedding, everything you already know about multi-head self-attention applies directly.
Recall the contrast with a CNN, which slides small filters over local neighborhoods and grows its receptive field only with depth:
Compare: a 3x3 CNN filter scanning an imageInteractive
Loading visualization...
Watch the filter visit every position one stride at a time. A CNN with three convolutional layers has a receptive field of seven pixels. A ViT — from layer one — has every patch attending to every other patch. The patch in the top-left can directly influence the prediction for the patch in the bottom-right, with no intermediate layers required.
Smaller patches (8x8 or 4x4) capture finer detail but explode the sequence length. A 224x224 image with 16x16 patches gives 196 tokens. With 8x8 patches it gives 784 tokens. Self-attention costs scale quadratically with sequence length, so halving the patch size roughly quadruples the compute. ViT papers default to 16x16 for that reason. Variants like ViT-B/8 exist for high-resolution medical and satellite tasks where detail matters more than throughput.
The ViT encoder stack: same blocks as BERTInteractive
Loading visualization...
Each block is multi-head self-attention plus a feed-forward network, wrapped in residual connections and LayerNorm — identical to a text transformer. The only difference is the input: 197 patch embeddings instead of 197 word embeddings.
What Do You Think?
You have 10,000 labeled training images. You train ResNet-50 from scratch and ViT-Base from scratch with the same compute budget. Which model wins on a held-out test set?
Locality — the 3x3 filter only looks at nearby pixels, so the network has to find local patterns first.
Translation invariance — the same filter is applied at every position, so a feature learned in the top-left works in the bottom-right for free.
A ViT has neither. Its self-attention can connect any patch to any other patch, and its positional embeddings are learned per-position — there is no enforcement that patch (1, 2) and patch (5, 6) should be treated the same way. The flip side: with enough data, the model discovers the right inductive bias, and the discovered bias is often more flexible than the one a CNN has hard-coded. This is why ViT pretrained on 300M+ images beats every CNN at scale.
Five years after the original ViT paper, the field has converged on a practical truth:
At parity of compute and data, CNN and ViT are roughly tied on standard benchmarks. ConvNeXt-V2 trades blows with Swin-V2 on ImageNet. EfficientNet-V2-L is competitive with ViT-L.
ViT wins on flexibility. It is trivially extended to multimodal tasks (text + image), to long videos (just add tokens), and to variable input sizes. CNN architectures need explicit redesign for each new modality.
CNN wins on edge deployment. MobileNet and EfficientNet-Lite remain the default for phones, doorbells, and battery-constrained devices. ViT requires more memory bandwidth per FLOP.
For foundation models, ViT has won. CLIP, SigLIP, DINOv2, SAM — every general-purpose vision backbone shipping in 2024-2026 is a ViT or a near-relative. That is the architecture choice you will inherit whenever you fine-tune a pretrained model.
What does a ViT attention head look at?Interactive
Loading visualization...
Pretrained ViTs develop attention heads that focus on object regions, texture boundaries, and even semantic parts — often without any segmentation supervision. This is what makes them excellent foundations for detection and segmentation heads.
The patch embedding step is the only piece of ViT that differs from a text transformer. Let's implement it from scratch on a tiny 8x8 grayscale image, then run a single self-attention block over the resulting patches.
Loading visualization...
This is the entire ViT pipeline in 50 lines. Everything else is depth (12 blocks for ViT-Base) and scale (bigger d_model, more heads, real datasets). The hard part is the data — not the math.
Classification asks "what is in this image?" and outputs one label. Detection asks "what is in this image, where, and how many of each?" and outputs a variable-length list of (label, bounding box, confidence) triples.
The first generation of deep-learning detectors stuck close to traditional computer vision: first find regions that might contain objects, then classify each one.
R-CNN (2014). Run selective search (a hand-crafted algorithm) to propose ~2000 candidate regions per image. Resize each region to 224x224. Run a CNN on each region independently to extract features. Classify each region with an SVM. Slow — 47 seconds per image at test time, because every region requires a full CNN forward pass.
Fast R-CNN (2015). Run the CNN once on the whole image to produce a feature map. Then crop region-of-interest features directly from that feature map using RoI Pooling. One CNN forward pass per image instead of per region. 25x faster than R-CNN.
Faster R-CNN (2015). Replace selective search with a learned Region Proposal Network (RPN) that shares the backbone CNN. Proposals are now generated by the network itself, end-to-end trainable, ~10x faster than Fast R-CNN. This is still the canonical two-stage detector and a strong baseline in 2026.
Mask R-CNN (2017). Add a third head on top of Faster R-CNN that predicts a binary segmentation mask for each region. Now you get instance segmentation almost for free. Still widely used in industrial inspection and medical imaging.
YOLO ("You Only Look Once") asked a heretical question in 2015: why split detection into two stages at all? The whole pipeline can be one CNN that outputs boxes and classes directly.
The YOLO trick. Divide the input image into an SxS grid (originally 7x7, modern versions 13x13 or 20x20). For each grid cell, predict a fixed set of anchor boxes — typically 3 to 5 — each with (x, y, w, h, objectness, class probabilities). A 13x13 grid with 3 anchors and 80 classes outputs 13 * 13 * 3 * (5 + 80) = ~43,000 candidate boxes per image. Most are background. Non-Max Suppression (NMS) prunes them down to the actual detections.
What Do You Think?
A YOLO model processes a 416x416 image with a 13x13 grid and 3 anchors per cell. How many candidate predictions does it emit before non-max suppression?
Variants over time. YOLOv2 added anchor boxes and multi-scale training. YOLOv3 used three different output scales for small/medium/large objects. YOLOv4 piled on every training trick (mosaic augmentation, CIoU loss, Mish activation). YOLOv5 was a major engineering refactor that made YOLO trivially usable from PyTorch. YOLOv7 and YOLOv8 squeezed more accuracy at the same speed. YOLOv9 (2024) introduced "Programmable Gradient Information" to keep gradient flow healthy in deep one-stage detectors.
SSD (Single-Shot MultiBox Detector, 2016). Like YOLO but predicts at multiple feature-map scales — small objects from high-resolution early features, large objects from low-resolution late features. A natural fit with Feature Pyramid Networks.
RetinaNet (2017). Solved a subtle but devastating problem in one-stage detectors: extreme class imbalance. With 100,000 candidate boxes per image and only ~10 real objects, the loss is dominated by easy background boxes. RetinaNet introduced Focal Loss, which down-weights easy examples and lets the model focus on hard ones.
Focal Loss(pt)=−(1−pt)γlog(pt)
Quick check
Why does Focal Loss help one-stage detectors but is rarely needed in two-stage detectors?
DETR (2020). Detection Transformer. Facebook AI's radical reformulation: skip the grid, skip the anchors, skip the NMS. Output a fixed-size set of N=100 predictions (each a class + box). Match predictions to ground-truth objects with bipartite matching (the Hungarian algorithm) during training. The set-prediction loss directly penalizes the matched pairs.
Every detection paper assumes you know these terms cold:
Anchor box. A pre-defined reference box (e.g., 32x32, 64x128, 128x64) placed at every grid cell. The network predicts an offset from the anchor rather than absolute coordinates. Anchors encode the prior that "objects come in roughly these shapes" — pedestrian anchors are tall and thin, car anchors are wide and short.
IoU (Intersection over Union). The fraction of overlap between two boxes. IoU = area_of_intersection / area_of_union. IoU=1 means perfect overlap, IoU=0 means no overlap. The standard cutoff for "this prediction matches this ground-truth box" is IoU >= 0.5.
NMS (Non-Max Suppression). After the model emits thousands of candidate boxes, NMS keeps only the highest-confidence box from any cluster of overlapping boxes. The procedure: sort all boxes by confidence; pick the top one; remove all other boxes with IoU >= 0.5 against it; repeat until empty.
mAP (mean Average Precision). The standard detection metric. For each class, compute the precision-recall curve as you sweep the confidence threshold. Average Precision (AP) is the area under that curve. Mean AP averages across all classes. COCO mAP averages further across IoU thresholds 0.5 to 0.95 — much stricter than the older PASCAL VOC mAP at IoU=0.5.
Detection metrics feel abstract until you compute them on a tiny example. The next playground walks through both — IoU between two boxes, then AP across a small set of predictions.
Loading visualization...
That's the entire mAP pipeline on a single image and a single class. Real benchmarks (COCO) compute this for 80 classes across 5,000 validation images at IoU thresholds from 0.5 to 0.95 in steps of 0.05, then average everything together — but the core arithmetic is exactly what you just ran.
By 2024 the detection landscape shifted again. Why train a model on a fixed list of 80 COCO classes when language models can describe any object in natural text?
Grounding DINO (2023) combines a DINO detector backbone with text-image alignment. You pass a free-form prompt like "a small red object next to the laptop" and the model returns matching boxes. Trained at scale, it generalizes to objects it never saw labeled — open-vocabulary detection.
OWLv2 (2023) from Google scales the same idea up: prompt the model with a class name, get boxes for that class. No fine-tuning per class needed.
SAM 2 + GPT-4V (2024) — even more general: use a multimodal LLM to describe what to detect, then SAM to find it pixel-perfectly.
These models are slower than YOLO and not yet the right tool for self-driving or real-time vision. But for "find every loose screw in this assembly-line photo," they make zero-shot a viable answer.
Quick check
Which detector should you reach for first when you have 200 labeled examples of a custom class and need to ship a demo in two days?
Segmentation is detection taken to the pixel level. Instead of a bounding box around the cat, you label every cat pixel as "cat" — and every other pixel as something else.
Semantic segmentation. Every pixel gets a class label. All cat pixels are "cat" — but the model does not distinguish between this cat and that cat. Output: an H x W label map.
Instance segmentation. Every pixel gets a class label and an object ID. Pixels of cat A are "cat 1," pixels of cat B are "cat 2." Output: a list of (class, mask) pairs.
Panoptic segmentation. Combines the two: countable "things" (cat, car, person) get instance IDs; uncountable "stuff" (sky, road, grass) gets a single semantic label per class. Output: a unified per-pixel (class, instance_id) map.
Quick check
Two cats sit side by side in a photo. Semantic segmentation outputs ___ ; instance segmentation outputs ___ ; panoptic segmentation outputs ___ .
FCN (Fully Convolutional Network, 2015). The first deep-learning approach to semantic segmentation. Take any CNN classifier (VGG, ResNet), strip the final fully-connected layer, and replace it with a 1x1 convolution that outputs C classes per spatial position. The result is a low-resolution prediction map (e.g., 7x7 if you started with 224x224). Upsample it back to the input resolution with a transposed convolution. Crude but it worked.
U-Net (2015). The architecture that runs medical imaging in 2026. Ronneberger et al. designed U-Net for cell segmentation when they had only ~30 training images. The trick: encoder-decoder with skip connections between symmetric layers.
The encoder downsamples through standard convolutions and max-pooling, growing the receptive field but losing spatial detail.
The decoder upsamples back to the input resolution with transposed convolutions.
At each decoder layer, the matching high-resolution feature map from the encoder is concatenated along the channel axis.
That skip path is the whole point. The encoder learns what is in the image at the cost of spatial precision; the decoder reuses the encoder's spatially precise early features to recover where exactly the objects are. Without the skips, the upsampled output is blobby and misses fine boundaries.
What Do You Think?
Why does U-Net put skip connections from the encoder directly to the decoder, instead of just upsampling the bottleneck features?
DeepLab (2015-2018). The other lineage of semantic segmentation, built on dilated (atrous) convolutions. Instead of downsampling aggressively, DeepLab keeps a higher spatial resolution throughout the network and uses dilated convs to expand receptive field without losing detail. The signature module is Atrous Spatial Pyramid Pooling (ASPP): parallel dilated convs at multiple rates (1, 6, 12, 18) capture context at multiple scales. DeepLab-V3+ remains a strong baseline on Cityscapes and PASCAL VOC.
Mask R-CNN. Already mentioned in the detection section — it adds a small FCN head on each region of interest that outputs a binary mask. This is the canonical recipe for instance segmentation.
SAM (Segment Anything Model, 2023). Meta AI's foundation model for segmentation. Trained on 1.1 billion masks across 11 million images, SAM is promptable: pass it a point, a box, or a coarse mask, and it returns a high-quality segmentation around that prompt. It is class-agnostic — it does not know what a cat is, but it knows where the boundary of the thing you pointed at lies. Photoshop's generative-fill lasso, Apple's "lift subject from background," and countless medical-imaging tools shipping in 2024-2026 are all built on SAM or its successor SAM 2.
Classification uses cross-entropy. Segmentation often uses Dice Loss or IoU Loss instead, especially when classes are imbalanced (tumor pixels are 0.1% of the image; background is everything else).
Combined losses are common: 0.5 * Cross-Entropy + 0.5 * Dice is a popular recipe for medical segmentation, giving stable training (from CE) and class-imbalance robustness (from Dice).
Three reasons U-Net is the default in radiology, pathology, and microscopy:
Small data. Medical datasets are often hundreds, not millions, of images. U-Net's encoder-decoder design with skip connections trains stably from scratch on small data because the parameter count is modest and the skips give a strong inductive bias for spatial reconstruction.
Boundary precision matters. A tumor segmentation that is off by 5 pixels at the boundary is clinically meaningful. The skip connections preserve the fine spatial detail needed for accurate boundaries.
Three-dimensional extension is easy. A 3D U-Net just swaps every 2D convolution for a 3D one. CT and MRI volumes — the bread and butter of medical imaging — are inherently 3D, and U-Net's 3D variant remains the go-to architecture for organ and lesion segmentation. nnU-Net (2021) is the auto-configuring 3D U-Net that won most of the medical-imaging challenges in 2021-2023.
Three threads are reshaping computer vision faster than any new architecture:
Foundation models give features for free. A pretrained DINOv2 or SAM encoder gives you features that beat ImageNet-pretrained ResNet on nearly any downstream task. The new question is not "what architecture do I train?" but "which pretrained backbone do I freeze?" Fine-tune a small head, ship the model.
Open-vocabulary everything. Classification (CLIP), detection (Grounding DINO, OWLv2), and segmentation (CLIPSeg, OVSeg) are all going text-conditioned. You prompt the model with a natural-language description and it returns matching outputs. For one-off prototypes and long-tail object types, this collapses weeks of data labeling into a single prompt.
Vision-language models eat the stack. GPT-4V, Claude Sonnet, Gemini, and the open-source LLaVA family take an image plus a text prompt and produce text — answering questions, describing scenes, reading documents, locating objects, and even outputting bounding-box coordinates as JSON. We cover the architecture of these models in the next track (Track 5: Transformers, "Multimodal Models" lesson). For now, recognize that the same ViT encoder you saw in Part 1 is the eye that lets these LLMs see, and that detection and segmentation increasingly happen inside a single conversational model rather than a dedicated pipeline.
The result: classical detection and segmentation pipelines still dominate where latency, accuracy, or auditability matters — self-driving, industrial inspection, medical diagnosis. Foundation-model approaches dominate where flexibility and zero-shot generalization matter — content moderation, image editing, exploratory science. Knowing both stacks is the modern requirement.
ViT treats an image as a sequence of patches and runs a standard transformer encoder over them — same architecture as BERT, different inputs. The trade is no built-in locality or translation-invariance in exchange for global modeling from layer one. CNNs win at small data; ViT wins at scale.
Detection is classification plus regression at variable count. Two-stage detectors (Faster R-CNN, Mask R-CNN) propose then classify regions; one-stage detectors (YOLO, SSD, RetinaNet) predict at every grid cell directly; DETR-style transformers output a fixed-size set matched to ground truth via Hungarian assignment.
The detection vocabulary is non-negotiable. Anchor boxes, IoU, NMS, mAP, COCO metrics — you cannot read a detection paper or debug a model without them. Focal Loss is the trick that makes one-stage detectors competitive despite extreme class imbalance.
Segmentation comes in three flavors. Semantic (per-pixel class), instance (per-pixel class + object ID), and panoptic (both). U-Net with skip connections dominates medical imaging because it preserves fine spatial detail; SAM is the foundation model that segments anything you point at, regardless of class.
Foundation models are reshaping the pipeline. Pretrained ViT backbones (DINOv2, CLIP), promptable segmentation (SAM), and open-vocabulary detection (Grounding DINO) collapse weeks of architecture and labeling work into a frozen encoder plus a tiny task-specific head.
Next up: Generative models. We have spent every lesson so far asking "what is in this image?" — now we ask the opposite question. Given a description, can the network draw the image?