Every image you generate with Stable Diffusion, FLUX, or Midjourney passes through a VAE. It's the unsung 8x compressor that turns a 512×512 image into a 64×64 latent — making diffusion 64× cheaper. Without VAEs, generating one image would burn a minute of GPU time instead of two seconds.
Learning Objectives
After this lesson, you will be able to:
Understand the encoder-bottleneck-decoder design and why compression forces the network to learn what matters
See where the VAE training objective (ELBO) comes from -- balancing reconstruction quality against KL regularization of the latent space
Explain the reparameterization trick and why it is necessary to train VAEs with gradient descent
Explore smooth latent spaces where nearby points produce similar outputs and perform meaningful interpolation
Autoencoders are one of the most elegant ideas in deep learning: force a network to compress data, and it discovers what matters on its own. No labels, no supervision, just the constraint that the output must match the input after passing through a narrow bottleneck. This is self-supervised learning at its purest.
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
The magic: nobody tells the network what to encode. It discovers on its own that face shape, hair color, expression, and lighting are the essential features. The compression forces understanding.
An autoencoder is a neural network trained to reconstruct its own input, with a bottleneck in the middle. It seems trivially easy -- just copy the input to the output. But the bottleneck makes this impossible by forcing the data through a much lower-dimensional representation. The network must learn what matters in order to survive this information bottleneck. The deep-learning track also covers GANs and VAEsGAN & VAE FoundationsGANs train a generator-discriminator pair adversarially; VAEs learn an encoder-decoder with a KL-regularized latent prior. The two pre-diffusion deep generative families and the architectures the generative-AI track builds on.Learn more → from the architectural angle — this lesson is the generative-AI track's deeper math view of the same family.
Try it! Describe a photo of a friend using only 5 words. Now try with 10 words. Now 50 words. Notice how more words let you capture more detail? That is exactly the tradeoff an autoencoder makes -- a wider bottleneck preserves more information but learns less efficient compression. The sweet spot is the smallest bottleneck that still gives good reconstruction.
If we made the bottleneck as wide as the input (same number of dimensions), what would the autoencoder learn?
Without a bottleneck, the easiest solution is to learn the identity function -- copy every pixel value through unchanged. No compression means no understanding. The bottleneck is not a limitation; it is the source of the autoencoder's power. By forcing information through a narrow channel, we compel the network to discover the most salient features of the data.
Try it: Build and Explore an AutoencoderInteractive
A full-resolution image enters the network -- for example, a 28x28 MNIST digit with 784 pixel values. Every pixel is a dimension. The autoencoder must learn to squeeze all 784 values through a much narrower bottleneck.
The encoder -- a stack of dense or convolutional layers -- progressively reduces dimensionality: 784 to 256 to 64 to 16. Each layer discards redundancy and keeps the features most useful for reconstruction. The network decides on its own what matters.
At the narrowest point, the data is represented as a compact latent vector z of just 16 dimensions. This is the "compressed description" of the input. For a vanilla autoencoder, z is a single deterministic point.
#Step 4: For VAE -- Sample z from N(mu, sigma squared)
In a Variational Autoencoder, the encoder outputs a mean mu and variance sigma squared instead of a single point. The latent code z is sampled from this Gaussian distribution using the reparameterization trick: z = mu + sigma * epsilon. This injects controlled randomness that makes the latent space smooth and generative.
The decoder mirrors the encoder in reverse: 16 to 64 to 256 to 784. It takes the latent vector z and expands it back to the original input dimensions, attempting to faithfully reconstruct every pixel of the input image.
#Step 6: Compare with Original (Reconstruction Loss)
The output x-hat is compared pixel-by-pixel with the original input x. The reconstruction loss (MSE or binary cross-entropy) measures how faithfully the decoder reproduced the input. For VAEs, a KL divergence term is added to keep the latent distributions close to a standard Gaussian.
After training, the decoder has learned to produce crisp reconstructions from compact latent codes. For generation, you can skip the encoder entirely -- sample z from N(0, I) and pass it through the decoder to create entirely new images that look like the training data.
Once trained, the encoder maps every training image to a point in the latent space. Similar images cluster together. The latent space becomes a compressed map of the data distribution.
For MNIST digits:
All "3"s cluster in one region
All "7"s cluster in another region
The boundary between 3 and 7 might contain ambiguous digits
Here is the critical limitation of vanilla autoencoders: they learn to compress and decompress, but they do not learn a proper probability distribution over the latent space. If you sample a random point in the latent space, the decoder might produce garbage -- because the encoder never mapped any real data to that region.
The latent space has "holes" -- regions that correspond to nothing meaningful. The autoencoder only learns to handle the specific latent codes produced by the encoder, not arbitrary points in the latent space.
The VAE fixes the autoencoder's generative limitations with one brilliant idea: instead of encoding each input to a single point in latent space, encode it to a probability distribution (a Gaussian). Then sample from that distribution to get the latent code.
Encoder outputs: μ(x),σ(x)z∼N(μ(x),σ2(x)⋅I)
This seemingly small change has profound consequences:
Continuity: Because the encoder outputs a Gaussian "cloud" rather than a single point, nearby latent codes must produce similar outputs. The decoder must handle a neighborhood, not a single point.
Coverage: The VAE loss encourages these Gaussian clouds to cover the latent space evenly, filling in the "holes" that vanilla autoencoders leave.
Generativity: Because the latent space is smooth and well-covered, you can sample any point and get a meaningful output.
The name ELBO stands for Evidence Lower BOund. To understand where it comes from: we want to maximize the log-likelihood of our data, log p(x). But this is intractable because computing p(x) requires integrating out the latent variable z over all possible values: p(x) = ∫ p(x|z) p(z) dz. The ELBO is a lower bound on this log-likelihood that we can compute:
Maximizing the ELBO is equivalent to simultaneously maximizing the expected log-likelihood of the data under the decoder (reconstruction quality) and minimizing how different the encoder's approximate posterior q(z|x) is from the prior p(z). Kingma & Welling (2013) showed that this bound becomes tight when q(z|x) matches the true posterior — meaning optimizing the ELBO is a principled way to do approximate maximum likelihood estimation in latent variable models.
The VAE loss (negative ELBO, which we minimize) has two terms that pull in complementary directions:
The reconstruction loss says: "The decoder must faithfully reproduce the input from the latent code." This pushes the encoder to preserve as much information as possible.
The KL divergence says: "The encoder's latent distribution should be close to a standard Gaussian N(0, I)." This regularizes the latent space, preventing the encoder from collapsing all information into tiny, isolated clusters. It forces the latent space to be smooth and continuous.
These two terms are in tension. The reconstruction loss wants to spread codes far apart for maximum information. The KL divergence wants to compress them all toward a standard Gaussian. The VAE finds a balance, and that balance creates a structured, generative latent space.
There is a subtle but critical problem: the sampling step z ~ N(mu, sigma^2) is not differentiable. You cannot backpropagate through random sampling. The VAE solves this with the reparameterization trick:
We want to sample z from N(mu, sigma^2), where mu and sigma come from the encoder. But sampling is a stochastic operation -- we cannot compute d(sample)/d(mu). Gradients cannot flow through the sampling step, so we cannot train the encoder.
The key insight: any sample from N(mu, sigma^2) can be written as mu + sigma * epsilon, where epsilon comes from a standard N(0,1). We split the operation into a deterministic transformation (mu + sigma * epsilon) and a fixed noise source (epsilon).
Now z = mu + sigma * epsilon is a deterministic, differentiable function of mu and sigma (epsilon is treated as a constant). Gradients flow cleanly: dz/d(mu) = 1, dz/d(sigma) = epsilon. The encoder can be trained with standard backpropagation.
During training, each forward pass samples a different epsilon, giving a different z. This stochasticity acts as regularization -- the decoder must handle a range of z values for each input, not just one specific code. Over many training steps, the gradients average out correctly.
Loading visualization...
The reparameterization trick is one of the most elegant ideas in modern ML. It appears everywhere -- in VAEs, in stochastic computation graphs, in policy gradient methods with continuous actions. The principle: when you need to differentiate through a sampling operation, reparameterize so the randomness is external to the parameters.
One of the most beautiful properties of VAEs is smooth latent interpolation. Given two inputs x1 and x2, you can:
Encode both: z1 = E(x1), z2 = E(x2)
Interpolate: z_alpha = (1 - alpha) * z1 + alpha * z2, for alpha in [0, 1]
Decode the interpolated points: D(z_alpha)
The result is a smooth morphing between the two inputs -- a face smoothly transforms into another face, a digit "3" gradually becomes a "7", a shoe morphs into a boot.
zα=(1−α)z1+αz2,α∈[0,1]
This works because the KL divergence term ensures the latent space is smooth -- there are no "dead zones" between encoded points. Every point in the latent space maps to a plausible output.
A variant called Beta-VAE increases the weight of the KL term (beta > 1), trading reconstruction quality for a more "disentangled" latent space where individual dimensions correspond to interpretable factors:
Higher beta means stronger regularization, more disentanglement, but blurrier reconstructions. It is a fundamental trade-off: you cannot have perfect reconstruction and perfect disentanglement simultaneously.
Tests · Verify that interpolation produces valid digits at all alpha values. Verify that alpha=0 returns the first digit and alpha=1 returns the second.
The original VAE had a key weakness: blurry outputs. The MSE reconstruction loss averages over possible outputs, producing a blurry mean. Several variants address this:
VQ-VAE (Vector Quantized VAE): Replaces the continuous latent space with a discrete codebook. Each latent vector is snapped to its nearest codebook entry. This eliminates blurriness and enables powerful autoregressive priors.
VQ-VAE-2: Hierarchical version with multiple codebook scales. Produced state-of-the-art image quality in 2019.
DALL-E (original): Used a VQ-VAE to tokenize images, then trained a transformer on those tokens alongside text tokens. The first viral text-to-image model.
Stable Diffusion's VAE: Every Stable Diffusion image passes through a VAE. The encoder compresses 512x512 images to a 64x64 latent space (8x spatial compression). Diffusion happens in this compressed latent space, then the VAE decoder upsamples back to pixel space. We will explore this in depth in Lesson 6.
The bottleneck forces useful representations. By compressing data through a narrow layer, autoencoders must learn the most important features, discarding noise and redundancy
VAEs add probabilistic structure to the latent space. Instead of encoding to a single point, VAEs encode to a distribution (mean + variance), which makes the latent space smooth and interpolatable
The reparameterization trick makes VAEs trainable. By expressing random sampling as z = mu + sigma * epsilon (where epsilon is noise), gradients can flow through the sampling step during backpropagation
Latent space interpolation generates meaningful transitions. Walking between two points in a well-structured latent space produces smooth, realistic transitions (e.g., gradually morphing one face into another)
What is the purpose of the bottleneck in an autoencoder?
You have now mastered autoencoders and VAEs -- the first deep generative architecture. The encoder-bottleneck-decoder pattern and the reparameterization trick are ideas you will see again and again. Next up: GANs -- a radically different approach where a generator and discriminator engage in an adversarial game that produces stunningly sharp outputs.