Variational Autoencoders

A variational autoencoder (VAE) is an autoencoder with a little randomness added. Instead of sending each picture to one exact point, it sends it to a small cloud of possible points. That one change makes the latent space tidy enough that you can pick a random point and decode a sensible new picture. A VAE sits inside Stable Diffusion and shrinks every image before the diffusion model touches it.

Recap of the last lesson. An autoencoder squeezes a picture through a narrow bottleneck into a short list of numbers (the latent) and rebuilds it, which forces the network to keep only what matters. Its weakness is that the latent space has holes, so decoding a random point gives garbage and a plain autoencoder cannot make new pictures. Read that lesson first if either idea is new.

Learning Objectives

After this lesson, you will be able to:

Say what a VAE changes: it stores a small cloud of possibilities, not one point, and why that closes the holes
Read the VAE training score (the ELBO) as two goals that pull against each other: rebuild well, and keep the latent space tidy
Explain the reparameterization trick: the random draw is moved outside the network so training can still pass through it
Walk smoothly between two latents and get sensible in-between pictures
Say what a bigger beta buys and what it costs, and name posterior collapse

Prerequisites

Before You Start

Review

#From Autoencoder to Generator

This lesson turns the squeezer into a generator. It uses the encoder, decoder, latent and bottleneck from the last lesson, and adds randomness. It covers the cloud-of-possibilities idea, the training score (the ELBO), the reparameterization trick, smooth walks between latents, beta-VAE, and the bridge to discrete latents.

Words you need7 terms

An autoencoder whose encoder outputs a small cloud of possible latents, not one point, so that decoding a random point gives a sensible picture.

The center of the cloud and how wide it is. The mean is written mu and the spread is written sigma.

A number for how different two bell curves are. Here: how far the encoder's cloud is from a plain bell curve centered at zero.

The VAE training score: how well the picture was rebuilt, minus a penalty for a messy latent space.

Write the random draw as mean plus spread times a fixed random number, so that training can pass gradients through it.

A VAE that multiplies the tidiness penalty by a number beta. A bigger beta gives a tidier but blurrier model.

A failure where the encoder ignores the input and the decoder ignores the latent, so the model learns nothing useful.

#The Variational Autoencoder (VAE)

The VAE fixes the holes with one idea. Instead of sending each input to a single point, the encoder sends it to a small cloud: a bell curve (Gaussian) described by a mean (the center) and a spread (how wide). The latent is then a random point picked from that cloud.

Picture first. A plain autoencoder puts the digit "3" at one exact spot. A VAE puts it in a fuzzy blob. The decoder has to rebuild a good "3" from any point in the blob, so nearby points must decode to similar pictures. Blobs of different digits spread out and touch, and the gaps close.

\text{Encoder outputs: } \mu(x), \; \sigma(x) \quad\quad z \sim \mathcal{N}(\mu(x), \, \sigma^2(x) \cdot I)

This small change has three effects:

Continuity: The encoder outputs a cloud, not a point, so nearby latents must give similar outputs. The decoder must handle a neighborhood, not one point.
Coverage: The VAE loss pushes the clouds to cover the latent space evenly, which fills the holes a plain autoencoder leaves.
Generativity: The latent space is smooth and well covered, so you can pick any point and get a meaningful output.

Here is the whole VAE path in one picture. The only new piece is the sample step. The random draw is made outside the network (reparameterization trick, below), so training can pass through it.

The sample is z = mean + spread x random draw. The mean and spread come from the network, and the random draw does not, so the network can still learn through that step.

#The ELBO: The VAE Loss Function

Picture first. The VAE is graded on two things at once. Did the decoder rebuild the picture well? And is the encoder's cloud close to a plain bell curve centered at zero? The ELBO is the first score minus a penalty for the second. A model that rebuilds perfectly but hides each picture in a private corner of the latent space gets a bad ELBO. So does a model that squeezes every cloud onto the center and rebuilds nothing.

A tiny worked example. Suppose the squared rebuilding error is 3.0 and the cloud is far from the plain bell curve, costing 2.0. The loss is 3.0 + 2.0 = 5.0. If a tweak lowers the error to 2.5 but raises the cost to 3.0, the loss is 5.5, so the tweak is rejected. Training keeps the tweaks that lower the total.

The name ELBO stands for Evidence Lower BOund. The "evidence" is p(x), how likely the model finds the real picture x. We would like to make p(x) large, but p(x) is intractable (too costly to compute), because it needs a sum over every possible latent z. The ELBO is a number that is always at most log p(x) and that we can compute, so raising it raises the thing we care about.

The VAE loss is the negative ELBO, which we minimize. It has two terms that pull in opposite directions:

\mathcal{L}_{\text{VAE}} = \underbrace{\mathbb{E}_{q(z|x)}\left[\|x - D(z)\|^2\right]}_{\text{Reconstruction loss}} + \underbrace{\beta \cdot D_{\text{KL}}\left(q(z|x) \;\|\; p(z)\right)}_{\text{KL divergence (regularizer)}}

The reconstruction loss says: "The decoder must rebuild the input from the latent." This pushes the encoder to keep as much information as possible.

The KL divergence says: "Each cloud should stay close to a plain bell curve centered at zero, N(0, I)." This keeps the encoder from hiding every picture in a tiny, isolated cluster, and it keeps the latent space smooth.

The two terms pull against each other. Reconstruction wants the clouds far apart, to keep every detail. KL wants them all pulled in toward the center. The VAE finds a balance, and that balance gives a tidy latent space you can sample from.

#The Reparameterization Trick

Picture first. Imagine the sample as a dice roll that happens in the middle of the network. Training works by asking "if I nudge this number, how does the final error change?" You cannot ask that of a dice roll, because the roll does not depend smoothly on anything. The trick is to roll the dice first, outside the network, and then build the sample from the result with plain arithmetic.

The problem is that the sampling step z ~ N(mu, sigma^2) is not differentiable. A step is differentiable if a tiny change in its inputs gives a tiny, predictable change in its output. Training needs this property to pass gradients (the "which way to nudge" signals) backward through the network. Random sampling breaks it. The reparameterization trick fixes it:

z = \mu + \sigma \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)

A tiny example. Say the encoder gives mean 2.0 and spread 0.5, and the dice roll gives epsilon = 1.2. Then z = 2.0 + 0.5 x 1.2 = 2.6. If the spread grows to 0.6, z becomes 2.72. The change in z is smooth and easy to follow, so training can learn the spread.

Try it below. The sliders set the mean and spread; the dot shows where a sample can land.

Loading visualization...

Do: Drag the point in the plane, set the mean and spread sliders, and press the Sad to Happy button.
Notice: A larger spread draws a wider circle of possible samples, and the shown face changes smoothly along the path.
Why: The sample is mean plus spread times a random draw, so each slider moves it smoothly. Drawn stand-in, not a trained model: the face is picked by distance to five hand-placed regions.

For a hands-on check of the trick itself, use the lab below. It lets you set the mean and spread of one latent value, draw a cloud of samples, and flip between rolling the dice inside the network and outside it.

Loading visualization...

Do: Leave the dice outside and press Sample twice. Then switch to "Random draw inside the network". Finally set the mean to 0 and the spread to 1.

Notice: With the dice outside, a red arrow carries the learning signal back to the encoder, and nudging the mean moves the sample by exactly 1 per unit. With the dice inside, the arrow is blocked. At mean 0 and spread 1 the cloud sits on the standard bell and KL reads 0.00.

Why: A random draw cannot be differentiated, so writing sample = mean + spread x draw puts the dice outside the network, so the mean and spread can be trained. Drawn stand-in, not a trained model: the cloud and KL number are computed from your sliders, with no real encoder or decoder.

#Step 1: The Problem

We want to sample z from N(mu, sigma^2), where mu and sigma come from the encoder. But sampling is a stochastic operation — we cannot compute d(sample)/d(mu). Gradients cannot flow through the sampling step, so we cannot train the encoder.

#Step 2: Separate the Randomness

The key insight: any sample from N(mu, sigma^2) can be written as mu + sigma * epsilon, where epsilon comes from a standard N(0,1). We split the operation into a deterministic transformation (mu + sigma * epsilon) and a fixed noise source (epsilon).

#Step 3: Gradients Flow Again

Now z = mu + sigma * epsilon is a deterministic, differentiable function of mu and sigma (epsilon is treated as a constant). Gradients flow cleanly: dz/d(mu) = 1, dz/d(sigma) = epsilon. The encoder can be trained with standard backpropagation.

#Step 4: Training Works

During training, each forward pass samples a different epsilon, giving a different z. This randomness acts as regularization — the decoder must handle a range of z values for each input, not just one specific latent. Over many training steps, the gradients average out correctly.

Loading visualization...

Do: Scroll through the four steps and watch where the random draw enters.
Notice: In step 3 the draw sits beside the network, not inside it, and the path from the loss back to the encoder is unbroken.
Why: Gradients can only pass through steps that depend smoothly on their inputs. Drawn stand-in, not a trained model.

Why mu plus sigma times epsilon has exactly the distribution N(mu, sigma squared) is derived in Multivariate Gaussian & Joint Distributions. The gradients are one line: the derivative of z with respect to mu is 1, and with respect to sigma it is epsilon.

The reparameterization trick appears far beyond VAEs: in stochastic computation graphs and in policy gradient methods with continuous actions. The principle is the same each time. When you must pass gradients through a sampling step, move the randomness outside the parameters.

#Latent Space Interpolation

Because the latent space is smooth, you can walk between two pictures. Given two inputs x1 and x2:

Encode both: z1 = E(x1), z2 = E(x2)
Interpolate: z_alpha = (1 - alpha) * z1 + alpha * z2, for alpha in [0, 1]
Decode the interpolated points: D(z_alpha)

The result is a smooth morph between the two inputs: one face turns into another, a "3" gradually becomes a "7", a shoe becomes a boot.

z_\alpha = (1 - \alpha)\, z_1 + \alpha\, z_2, \quad \alpha \in [0, 1]

This works because the KL term keeps the latent space smooth: there are no dead zones between encoded points, so every point maps to a plausible output.

Try it: Explore the Latent SpaceInteractive

Loading visualization...

Do: Click two points, then move the interpolation slider from one to the other.
Notice: The in-between point travels in a straight line and its class label changes where it crosses a cluster border.
Why: A smooth latent space keeps nearby points similar. Drawn stand-in, not a trained model: the points are placed by hand.

#Beta-VAE: Disentangled Representations

A variant called Beta-VAE raises the weight of the KL term (beta greater than 1). It trades reconstruction quality for a "disentangled" latent space, where each dimension tends to control one factor you can name:

Dimension 3 controls hair color
Dimension 7 controls smile intensity
Dimension 12 controls viewing angle

\mathcal{L}_{\beta\text{-VAE}} = \mathbb{E}_{q(z|x)}\left[\|x - D(z)\|^2\right] + \beta \cdot D_{\text{KL}}\left(q(z|x) \;\|\; p(z)\right), \quad \beta > 1

A higher beta means a stronger pull toward the plain bell curve, more disentanglement, and blurrier reconstructions. It is a real trade-off: you cannot have perfect reconstruction and perfect disentanglement at once.

The beta-VAE objective is derived as the ELBO with a weighted KL term in Autoencoders (Deep Learning track).

Watch Out

Common Mistake

Setting beta too high in a Beta-VAE, hoping for "better" representations. A larger beta does force more disentanglement, but it also hurts reconstruction. If beta is too large, the latent carries almost no information about the input (the KL term dominates), and the decoder learns to ignore z and output blurry averages whatever you encode. Watch reconstruction quality alongside disentanglement. They are in tension.

A second mistake is forgetting to ramp up the KL weight during training. Starting with a high KL penalty from step one makes the model collapse onto the prior at once. This is called posterior collapse: the encoder always outputs N(0, I) and the decoder ignores the latent completely. The standard fix is KL annealing: start with a KL weight of 0 and raise it linearly over the first several thousand steps, so the model learns to rebuild before the penalty kicks in.

#CodePlayground: VAE on MNIST

This is a toy, not a trained VAE. The ten digit centres are placed by hand, the encoder only adds a small random offset, and the decoder picks the nearest centre. It shows what walking through a 2-D latent space looks like, not what a real VAE learns.

pythonplayground.py · Pyodide

import numpy as np
import math

# Simulate VAE encoding and latent space interpolation
# Using simplified 2D latent space for visualization

means = {
  0: [-2.0, 1.5], 1: [2.5, 2.0], 2: [-1.0, -1.5],
  3: [0.5, -2.0], 4: [1.5, 0.0], 5: [-0.5, -0.5],
  6: [-2.5, -0.5], 7: [2.0, 1.0], 8: [0.0, 1.0], 9: [1.0, -1.0],
}

rng = np.random.default_rng(42)

def encode(digit):
  """Toy encoder: hand-placed 2D centres, not learned from MNIST."""
  mu = np.array(means[digit])
  # Toy jitter around a hand-placed centre. A real VAE would sample
  # z = mu + sigma * eps with Gaussian eps and learned mu, sigma.
  eps = rng.uniform(-0.15, 0.15, size=2)
  z = mu + eps
  return {"mu": mu, "z": z}

def interpolate(z1, z2, alpha):
  return (1 - alpha) * z1 + alpha * z2

def find_closest_digit(z):
  closest, min_dist = 0, float("inf")
  for d in range(10):
      m = np.array(means[d])
      dist = np.linalg.norm(z - m)
      if dist < min_dist:
          min_dist = dist
          closest = d
  return closest

# Encode two digits
enc3 = encode(3)
enc7 = encode(7)
print(f"Digit 3 -> latent: [{enc3['z'][0]:.2f}, {enc3['z'][1]:.2f}]")
print(f"Digit 7 -> latent: [{enc7['z'][0]:.2f}, {enc7['z'][1]:.2f}]")
print()

# Interpolate between them
print("Interpolation from 3 to 7:")
for a in np.arange(0, 1.01, 0.2):
  alpha = min(a, 1.0)
  z = interpolate(enc3["z"], enc7["z"], alpha)
  decoded = find_closest_digit(z)
  print(
      f"  alpha={alpha:.1f}"
      f" | z=[{z[0]:.2f}, {z[1]:.2f}]"
      f" | decoded digit: {decoded}"
  )

print()
print("CHALLENGE: What digit appears at alpha=0.5?")
print("Try encoding different digit pairs!")

Tests · Verify that interpolation produces valid digits at all alpha values. Verify that alpha=0 returns the first digit and alpha=1 returns the second.

#VAE Variants and Modern Descendants

Why are plain VAE outputs blurry? Squared error rewards the average. Under squared error, the best single output is the average of all the images that could belong to the same latent. The derivation shows it exactly.

The original VAE had one key weakness: blurry outputs. The squared-error loss averages over the possible outputs and produces a blurry mean. Several variants address this.

#Bridge: from continuous to discrete latents

So far the latent was a list of smooth numbers. Another way to build a generator is to make the latent a choice from a fixed list instead. This is the discrete-latent family, and the full story is in VQ-VAE & Discrete Latents.

VQ-VAE (Vector Quantized VAE): Replaces the continuous latent space with a discrete codebook, a fixed table of vectors. Each latent vector is snapped to its nearest codebook entry. Snapping avoids averaging over continuous values and can give sharper results, but it does not remove blur by itself: the decoder is still trained with a reconstruction loss. Discrete codes also let you train an autoregressive prior over them, a model that predicts one code after another like a language model.

VQ-VAE-2: A hierarchical version with codebooks at several scales. It produced state-of-the-art image quality in 2019.

DALL-E (original): Used a discrete VAE (a close cousin of VQ-VAE that picks codes with a Gumbel-softmax relaxation instead of a nearest-neighbour lookup) to turn images into tokens, then trained a transformer on those tokens alongside text tokens. It was an early, widely noticed text-to-image model.

#The continuous VAE inside Stable Diffusion

Stable Diffusion's VAE: Stable Diffusion works in the latent space of a VAE. It is a continuous, KL-regularised VAE (a small KL penalty, no codebook), not a VQ-VAE. The encoder compresses 512x512 images to a 64x64 latent space (8x spatial compression). Diffusion happens in this compressed latent space, then the VAE decoder upsamples back to pixel space. Latent Diffusion covers this in depth.

The two compression figures in this lesson, 64 times and 48 times, count different things. The derivation shows where each comes from.

Auto-Encoding Variational Bayes

Diederik Kingma, Max Welling (2013)

The foundational VAE paper. Introduced the ELBO objective, the reparameterization trick, and launched an entire subfield of deep generative modeling.

Neural Discrete Representation Learning

Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu (2017)

Introduced VQ-VAE: replacing continuous latent variables with discrete codebook entries. Discrete image tokens later influenced DALL-E 1's dVAE and many token-based image and audio models. Stable Diffusion's VAE is continuous (KL-regularised), not VQ.

#Key Takeaways

VAEs add structure to the latent space. Instead of one point, the encoder gives a cloud (mean and spread), which makes the latent space smooth and easy to walk through
The ELBO is the VAE training score: rebuild well, minus a penalty for a messy latent space
The reparameterization trick makes VAEs trainable. Writing the random sample as z = mu + sigma * epsilon (epsilon is a random draw) lets gradients pass through the sampling step
Walking between two points in a well-built latent space gives smooth, realistic in-between pictures
A bigger beta gives more readable axes and blurrier pictures. Too much, or a full KL weight from step one, and the latent carries nothing (posterior collapse)

#Quick Check

Quick Check1 / 4

What does a VAE encoder produce for one input picture?

Where to next. Go to GANs, a very different route to generating: two networks that compete, which trades the VAE's blur for sharp pictures. The way a VAE shrinks images before diffusion comes back in Latent Diffusion. The discrete-latent bridge continues, as optional deeper reading, in VQ-VAE & Discrete Latents.

Variational Autoencoders

Learning Objectives

After this lesson, you will be able to:

Say what a VAE changes: it stores a small cloud of possibilities, not one point, and why that closes the holes
Read the VAE training score (the ELBO) as two goals that pull against each other: rebuild well, and keep the latent space tidy
Explain the reparameterization trick: the random draw is moved outside the network so training can still pass through it
Walk smoothly between two latents and get sensible in-between pictures
Say what a bigger beta buys and what it costs, and name posterior collapse

Prerequisites

Before You Start

Review

#From Autoencoder to Generator

Words you need7 terms

An autoencoder whose encoder outputs a small cloud of possible latents, not one point, so that decoding a random point gives a sensible picture.

The center of the cloud and how wide it is. The mean is written mu and the spread is written sigma.

A number for how different two bell curves are. Here: how far the encoder's cloud is from a plain bell curve centered at zero.

The VAE training score: how well the picture was rebuilt, minus a penalty for a messy latent space.

Write the random draw as mean plus spread times a fixed random number, so that training can pass gradients through it.

A VAE that multiplies the tidiness penalty by a number beta. A bigger beta gives a tidier but blurrier model.

A failure where the encoder ignores the input and the decoder ignores the latent, so the model learns nothing useful.

#The Variational Autoencoder (VAE)

\text{Encoder outputs: } \mu(x), \; \sigma(x) \quad\quad z \sim \mathcal{N}(\mu(x), \, \sigma^2(x) \cdot I)

This small change has three effects:

Continuity: The encoder outputs a cloud, not a point, so nearby latents must give similar outputs. The decoder must handle a neighborhood, not one point.
Coverage: The VAE loss pushes the clouds to cover the latent space evenly, which fills the holes a plain autoencoder leaves.
Generativity: The latent space is smooth and well covered, so you can pick any point and get a meaningful output.

Here is the whole VAE path in one picture. The only new piece is the sample step. The random draw is made outside the network (reparameterization trick, below), so training can pass through it.

The VAE path

An architecture diagram: boxes are components, arrows show the direction data flows, and moving dots mark the live flow.

👤Picture

🧠Encoder

🗄️Mean

🗄️Spread

⚡Random draw

🔧Sample z

🧠Decoder

👤 user🧠 llm🗄️ database⚡ cache🔧 tool

The sample is z = mean + spread x random draw. The mean and spread come from the network, and the random draw does not, so the network can still learn through that step.

#The ELBO: The VAE Loss Function

The VAE loss is the negative ELBO, which we minimize. It has two terms that pull in opposite directions:

\mathcal{L}_{\text{VAE}} = \underbrace{\mathbb{E}_{q(z|x)}\left[\|x - D(z)\|^2\right]}_{\text{Reconstruction loss}} + \underbrace{\beta \cdot D_{\text{KL}}\left(q(z|x) \;\|\; p(z)\right)}_{\text{KL divergence (regularizer)}}

The reconstruction loss says: "The decoder must rebuild the input from the latent." This pushes the encoder to keep as much information as possible.

#The Reparameterization Trick

z = \mu + \sigma \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)

Try it below. The sliders set the mean and spread; the dot shows where a sample can land.

Loading visualization...

Do: Drag the point in the plane, set the mean and spread sliders, and press the Sad to Happy button.
Notice: A larger spread draws a wider circle of possible samples, and the shown face changes smoothly along the path.
Why: The sample is mean plus spread times a random draw, so each slider moves it smoothly. Drawn stand-in, not a trained model: the face is picked by distance to five hand-placed regions.

Loading visualization...

Do: Leave the dice outside and press Sample twice. Then switch to "Random draw inside the network". Finally set the mean to 0 and the spread to 1.

Notice: With the dice outside, a red arrow carries the learning signal back to the encoder, and nudging the mean moves the sample by exactly 1 per unit. With the dice inside, the arrow is blocked. At mean 0 and spread 1 the cloud sits on the standard bell and KL reads 0.00.

Why: A random draw cannot be differentiated, so writing sample = mean + spread x draw puts the dice outside the network, so the mean and spread can be trained. Drawn stand-in, not a trained model: the cloud and KL number are computed from your sliders, with no real encoder or decoder.

#Step 1: The Problem

#Step 2: Separate the Randomness

#Step 3: Gradients Flow Again

#Step 4: Training Works

Loading visualization...

Do: Scroll through the four steps and watch where the random draw enters.
Notice: In step 3 the draw sits beside the network, not inside it, and the path from the loss back to the encoder is unbroken.
Why: Gradients can only pass through steps that depend smoothly on their inputs. Drawn stand-in, not a trained model.

#Latent Space Interpolation

Because the latent space is smooth, you can walk between two pictures. Given two inputs x1 and x2:

Encode both: z1 = E(x1), z2 = E(x2)
Interpolate: z_alpha = (1 - alpha) * z1 + alpha * z2, for alpha in [0, 1]
Decode the interpolated points: D(z_alpha)

The result is a smooth morph between the two inputs: one face turns into another, a "3" gradually becomes a "7", a shoe becomes a boot.

z_\alpha = (1 - \alpha)\, z_1 + \alpha\, z_2, \quad \alpha \in [0, 1]

This works because the KL term keeps the latent space smooth: there are no dead zones between encoded points, so every point maps to a plausible output.

Try it: Explore the Latent SpaceInteractive

Loading visualization...

Do: Click two points, then move the interpolation slider from one to the other.
Notice: The in-between point travels in a straight line and its class label changes where it crosses a cluster border.
Why: A smooth latent space keeps nearby points similar. Drawn stand-in, not a trained model: the points are placed by hand.

#Beta-VAE: Disentangled Representations

Dimension 3 controls hair color
Dimension 7 controls smile intensity
Dimension 12 controls viewing angle

\mathcal{L}_{\beta\text{-VAE}} = \mathbb{E}_{q(z|x)}\left[\|x - D(z)\|^2\right] + \beta \cdot D_{\text{KL}}\left(q(z|x) \;\|\; p(z)\right), \quad \beta > 1

The beta-VAE objective is derived as the ELBO with a weighted KL term in Autoencoders (Deep Learning track).

Watch Out

Common Mistake

#CodePlayground: VAE on MNIST

pythonplayground.py · Pyodide

import numpy as np
import math

# Simulate VAE encoding and latent space interpolation
# Using simplified 2D latent space for visualization

means = {
  0: [-2.0, 1.5], 1: [2.5, 2.0], 2: [-1.0, -1.5],
  3: [0.5, -2.0], 4: [1.5, 0.0], 5: [-0.5, -0.5],
  6: [-2.5, -0.5], 7: [2.0, 1.0], 8: [0.0, 1.0], 9: [1.0, -1.0],
}

rng = np.random.default_rng(42)

def interpolate(z1, z2, alpha):
  return (1 - alpha) * z1 + alpha * z2

print()
print("CHALLENGE: What digit appears at alpha=0.5?")
print("Try encoding different digit pairs!")

Tests · Verify that interpolation produces valid digits at all alpha values. Verify that alpha=0 returns the first digit and alpha=1 returns the second.

#VAE Variants and Modern Descendants

The original VAE had one key weakness: blurry outputs. The squared-error loss averages over the possible outputs and produces a blurry mean. Several variants address this.

#Bridge: from continuous to discrete latents

VQ-VAE-2: A hierarchical version with codebooks at several scales. It produced state-of-the-art image quality in 2019.

#The continuous VAE inside Stable Diffusion

The two compression figures in this lesson, 64 times and 48 times, count different things. The derivation shows where each comes from.

Auto-Encoding Variational Bayes

Diederik Kingma, Max Welling (2013)

The foundational VAE paper. Introduced the ELBO objective, the reparameterization trick, and launched an entire subfield of deep generative modeling.

Neural Discrete Representation Learning

Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu (2017)

#Key Takeaways

VAEs add structure to the latent space. Instead of one point, the encoder gives a cloud (mean and spread), which makes the latent space smooth and easy to walk through
The ELBO is the VAE training score: rebuild well, minus a penalty for a messy latent space
The reparameterization trick makes VAEs trainable. Writing the random sample as z = mu + sigma * epsilon (epsilon is a random draw) lets gradients pass through the sampling step
Walking between two points in a well-built latent space gives smooth, realistic in-between pictures
A bigger beta gives more readable axes and blurrier pictures. Too much, or a full KL weight from step one, and the latent carries nothing (posterior collapse)

#Quick Check

Quick Check1 / 4

What does a VAE encoder produce for one input picture?

Variational Autoencoders

#From Autoencoder to Generator

#The Variational Autoencoder (VAE)

#The ELBO: The VAE Loss Function

Where the ELBO comes from (formula)

#The Reparameterization Trick

#Step 1: The Problem

#Step 2: Separate the Randomness

#Step 3: Gradients Flow Again

#Step 4: Training Works

#Latent Space Interpolation

#Beta-VAE: Disentangled Representations

#CodePlayground: VAE on MNIST

#VAE Variants and Modern Descendants

Why squared error produces a blurry average

#Bridge: from continuous to discrete latents

#The continuous VAE inside Stable Diffusion

Why 8 times smaller on each side means 64 times fewer positions and 48 times fewer numbers

Deep Dive

#Key Takeaways

#Quick Check

Related Lessons

Your Reflection

Continue Learning

Discussion

Variational Autoencoders

#From Autoencoder to Generator

#The Variational Autoencoder (VAE)

The VAE path

#The ELBO: The VAE Loss Function

Where the ELBO comes from (formula)

#The Reparameterization Trick

#Step 1: The Problem

#Step 2: Separate the Randomness

#Step 3: Gradients Flow Again

#Step 4: Training Works

#Latent Space Interpolation

#Beta-VAE: Disentangled Representations

#CodePlayground: VAE on MNIST

#VAE Variants and Modern Descendants

Why squared error produces a blurry average

#Bridge: from continuous to discrete latents

#The continuous VAE inside Stable Diffusion

Why 8 times smaller on each side means 64 times fewer positions and 48 times fewer numbers

Deep Dive

#Key Takeaways

#Quick Check

Related Lessons

Your Reflection

Continue Learning

Discussion

The VAE path