Stable Diffusion runs on your laptop. DALL-E 3 powers ChatGPT. Midjourney generates 30M images a day. The one trick that made all of this affordable: don't diffuse on pixels — diffuse on an 8× compressed VAE latent. That single architectural decision cut compute by 64× and turned image generation from a cloud-only luxury into a commodity.
Learning Objectives
After this lesson, you will be able to:
Understand why working in compressed latent space makes diffusion 10-100x faster without losing quality -- and quantify the exact spatial compression (8x) that Stable Diffusion's VAE provides
Follow the complete Stable Diffusion pipeline from text prompt to final image: text encoder → VAE encoder → U-Net denoiser → VAE decoder
Explain how classifier-free guidance (CFG) is computed and why it doubles the per-step cost
Trace how cross-attention connects CLIP text embeddings to every U-Net layer to steer generation toward the prompt
Latent diffusion is where everything in this track comes together: autoencodersGAN & VAE FoundationsGANs train a generator-discriminator pair adversarially; VAEs learn an encoder-decoder with a KL-regularized latent prior. The two pre-diffusion deep generative families and the architectures the generative-AI track builds on.Learn more → (compression), diffusion (generation), and CLIPCLIPCLIP aligns text and image embeddings in a shared space using contrastive learning, enabling zero-shot classification from text descriptions alone.Learn more → (understanding text). If you have followed those lessons, you already have all the building blocks. This lesson just shows how they snap together into the system that powers Stable Diffusion, DALL-E, and every AI art tool you have used.
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
That is the insight behind Latent Diffusion Models (LDMs). Running the diffusion process directly on 512x512x3 images (786,432 dimensions) is like designing with individual bricks -- computationally enormous and wasteful, because most of those pixels are highly correlated (neighboring pixels in a sky are nearly identical).
Instead, a VAE compresses the image to a much smaller latent space (e.g., 64x64x4 = 16,384 dimensions), and diffusion operates there. A 48x reduction in spatial size means roughly 48x less compute for each U-Net forward pass. This single insight made high-resolution image generation practical on consumer GPUs.
Before latent diffusion, pixel-space diffusion models like DDPM and Guided Diffusion produced excellent results but required enormous compute. Generating a single 256x256 image took minutes on a high-end GPU. Scaling to 512x512 or 1024x1024 was prohibitively expensive. Latent Diffusion Models, introduced by Rombach et al. (2022), changed everything by operating in compressed latent space.
Try it! Open Stable Diffusion (or any free alternative like Playground AI). Generate the same image with CFG guidance scale = 1 (low), 7 (default), and 20 (high). Watch how low guidance gives creative but off-topic results, while high guidance follows your prompt precisely but may look over-saturated. The sweet spot is usually 7-12.
The text prompt is converted into a sequence of embedding vectors by a pretrained text encoder. These embeddings capture the semantic meaning of the prompt and guide the diffusion process through cross-attention.
A pretrained VAE compresses images from pixel space (512x512x3) to latent space (64x64x4). The compression is lossy but preserves the essential visual information. The diffusion process operates entirely in this compressed space.
The U-Net operates entirely in latent space. It takes a noisy latent z_t, the timestep t, and the text conditioning c, and predicts the noise (or v, in newer versions):
After denoising is complete (z_0 is recovered), the VAE decoder maps back to pixel space:
x^0=VAEdec(z^0)∈R512×512×3
Try it: Explore the Stable Diffusion PipelineInteractive
Loading visualization...
What Do You Think?
The VAE in Stable Diffusion was trained SEPARATELY from the U-Net. Why is this a good design choice?
Decoupling the VAE from the diffusion model is a powerful architectural decision. The VAE provides a general-purpose image compression/decompression capability that can be:
Trained once on a large dataset and reused
Shared across different diffusion models (SD 1.5, SDXL, etc.)
Upgraded independently (better VAE decoders can improve all existing models)
Used for other tasks (image editing, inpainting, super-resolution)
Classifier-free guidance (CFG) is the technique that makes text-to-image generation actually follow prompts. Without it, the model generates plausible images that may ignore the text. With it, the model strongly adheres to the prompt.
ϵ^=ϵθ(zt,t,∅)+w⋅(ϵθ(zt,t,c)−ϵθ(zt,t,∅))
The critical detail: CFG requires two U-Net forward passes per step -- one unconditional and one conditional. This doubles the compute cost. Training with CFG requires randomly dropping the text conditioning (replacing it with a null embedding) during some fraction of training steps (typically 10%), so the model learns both conditional and unconditional generation.
The prompt is encoded by CLIP into conditioning embeddings. Separately, random noise is initialised in a compressed 64x64 latent space — not in pixel space, which is what makes this tractable. A U-Net then denoises that latent step by step, at each step attending to the text embeddings via cross-attention so the image is pulled toward the prompt. Once the latent is clean, a decoder expands it back into a full-resolution image.
The user provides a natural language prompt -- for example, "a cat on mars." This plain-text string is the only input to the entire pipeline. Everything that follows is the system translating this sentence into a photorealistic image.
The prompt is tokenized and passed through a pretrained CLIP text encoder (or T5 for newer models). Output: a sequence of 77 embedding vectors of dimension 768 (SD 1.5) or larger. These embeddings capture the semantic meaning of "cat," "on," and "mars" and their compositional relationship. This takes ~5ms and is cached for all denoising steps.
The text embeddings do not enter the U-Net as a simple input -- they guide generation through cross-attention layers at every resolution level. At each denoising step, the noisy latent features (queries) attend to the text embeddings (keys and values). This mechanism lets the model compose objects, styles, and spatial relationships described in the prompt.
A random latent tensor z_T of shape 64x64x4 is sampled from N(0, I). Diffusion operates entirely in this compressed latent space -- 48x smaller than pixel space. The U-Net does not see pixels; it sees latent features. This is the key efficiency insight of Latent Diffusion Models.
The U-Net iteratively denoises the latent tensor over 20-50 steps. At each step, it predicts noise conditioned on both the timestep and the text embeddings. Classifier-free guidance (CFG) combines a conditional prediction (with text) and an unconditional prediction (without text) to amplify prompt adherence. The scheduler computes z_(t-1) from z_t and the guided noise prediction. This loop consumes 95% of total generation time.
The fully denoised latent z_0 is passed through the VAE decoder, which upsamples from 64x64x4 back to 512x512x3 pixel space. This single forward pass (~20ms) adds the fine pixel-level details -- textures, sharp edges, color gradients, and sub-pixel information -- that the latent space had abstracted away.
The output is a photorealistic 512x512 image of a cat sitting on the Martian surface. Total pipeline time: ~3 seconds on an A100 (50 steps), ~8 seconds on an RTX 3090. The seed, prompt, guidance scale, and scheduler settings can be saved and shared for exact reproduction. The image is novel -- it matches the prompt but was never in the training data.
ControlNet (Zhang et al., 2023) adds spatial control to Stable Diffusion by conditioning on additional inputs: edge maps, depth maps, pose skeletons, segmentation masks, and more.
ControlNet works by creating a trainable copy of the U-Net encoder blocks and injecting their outputs into the original U-Net through zero-initialized convolutions. This means:
The original model weights are frozen (no catastrophic forgetting)
The ControlNet learns to modulate the generation based on the spatial input
Multiple ControlNets can be stacked (edges + depth + pose simultaneously)
The Stable Diffusion / latent-diffusion family has evolved rapidly. The table below covers the open-weights image lineage from 2022 through 2024-2025, plus the leading video-diffusion analogs.
Model
Release
Backbone
Params
Resolution
Text Encoder
Key Innovation
SD 1.4
Aug 2022
U-Net
~860M
512x512
CLIP ViT-L/14
First public latent diffusion release
SD 1.5
Oct 2022
U-Net
~860M
512x512
CLIP ViT-L/14
The de-facto SD 1.x checkpoint everyone fine-tunes
SD 2.0
Nov 2022
U-Net
~865M
768x768
OpenCLIP ViT-H
OpenCLIP encoder, NSFW filter, retrained data
SD 2.1
Dec 2022
U-Net
~865M
768x768
OpenCLIP ViT-H
Fixes to 2.0's data filtering, v-prediction
SDXL 1.0
Jul 2023
U-Net
~2.6B + 6.6B refiner
1024x1024
CLIP-L + OpenCLIP-G
Two text encoders, base + refiner pipeline
SDXL Turbo
Nov 2023
U-Net
~2.6B
512x512
CLIP-L + OpenCLIP-G
ADD (adversarial distillation), 1-4 steps
SD 3.0 (Medium)
Jun 2024
MM-DiT
~2B
1024x1024
CLIP-L + CLIP-G + T5-XXL
First production MM-DiT, rectified flow
SD 3.5 (Large)
Oct 2024
MM-DiT
~8B
1024x1024
CLIP-L + CLIP-G + T5-XXL
Scale-up of 3.0, MMDiT-X variants
FLUX.1 [dev]
Aug 2024
MM-DiT
~12B
up to 2048
T5-XXL + CLIP-L
Black Forest Labs, non-commercial license, SOTA quality
A few notes on the table. MM-DiT stands for Multimodal Diffusion Transformer — it replaces the U-Net with a transformer that handles image tokens and text tokens in a single shared attention stack rather than via cross-attention. SD 3, SD 3.5, and the entire FLUX family all use MM-DiT, and rectified flow (not DDPM) is the default training objective from SD 3 onward. The video models (Sora, Veo 2/3) are latent-diffusion in spirit but operate over spatio-temporal latents — patches of (T, H, W) feature volumes — instead of 2D image latents.
The trend is clear: larger models, multiple text encoders, higher resolutions, U-Net → transformer (DiT/MM-DiT), DDPM → rectified flow, and the same architecture lifted from images to video.
Diffusing in latent space provides 10-100x efficiency gains. By compressing images to a smaller latent representation via a VAE first, the U-Net operates on much smaller tensors without sacrificing perceptual quality
Stable Diffusion chains four components. A text encoder (CLIP) converts prompts to embeddings, a VAE encoder compresses images to latent space, a U-Net denoises in latent space, and a VAE decoder reconstructs the final image
Classifier-free guidance (CFG) controls the quality-diversity tradeoff. Higher guidance scale produces images that match the prompt more closely but with less diversity; lower guidance allows more creative variation
Cross-attention connects text conditioning to visual generation. At each U-Net layer, cross-attention lets the denoising process "look at" the text embedding, guiding generation toward the described content
Why does Stable Diffusion perform diffusion in latent space instead of pixel space?
You now understand the full Stable Diffusion architecture -- from text encoding through latent-space diffusion to pixel-space decoding. This architecture powers the vast majority of image generation tools used today. Next up: Flow Matching -- a new generation of generative models that offers faster, more elegant generation without the multi-step denoising dance.