Generative Expert Lab

Reading a formula and writing it are two different skills. This lab has nine exercises. Each is a short function you write yourself in numpy, then check with tests that run in your browser. Each one starts with a plain-words setup and a trace by hand, so you know the right number before you type a line.

Recap of earlier lessons. Diffusion Models gives the forward and reverse steps and guidance. Score-Based Models gives the score. Flow Matching gives the straight-line target. Autoregressive and Discrete Generation gives the unmasking idea. Evaluating Generative Models: The Basics gives FID. This lab does not teach those ideas again. It makes you build them.

Learning Objectives

After this lesson, you will be able to:

Write the forward noising jump, the DDPM reverse mean, and the DDIM step with its eta dial, and say which quantities each one uses
Combine two noise guesses with classifier-free guidance in the track convention, and translate the Ho and Salimans form into it
Convert between noise, clean image and velocity predictions using only the signal fraction
Write the denoising score matching loss and check a true score against a slope found by finite differences
Write the flow matching target and an Euler sampler, and say which way time runs in this track and in SD3 and FLUX
Write a confidence-based unmasking step and a Frechet distance, and know the three traps that break most implementations: sign, time direction, and alpha versus alpha bar

Prerequisites

Before You Start

Review

#How This Lab Works

Every exercise has the same shape. First comes a short setup in plain words. Then a hand trace: one small calculation done on paper, with the answer your code should give. Then a card with the goal, a ladder of hints that appear one at a time, and a code editor. The editor has a starter with TODO marks, a Run button, a Check my work button, and a Show solution button.

Check my work runs your code, then runs a list of tests, one at a time. It tells you how many passed and shows the message of the first one that failed. The starter is built to fail on purpose, so a fresh cell always says some checks failed. When you finish the TODO lines, all of them pass.

Three things about the tests are worth knowing before you start.

They are tolerance based. A result close enough to the right answer passes, so a harmless rounding difference never fails you.
They are deterministic. Any random numbers come from a fixed seed, so a test that fails today fails the same way tomorrow.
They check more than the one example in the hand trace. Some compare your answer with a second route to the same number, such as a slope found by finite differences or a closed form for a simple case. If your code only matches the trace, it will still fail those.

The cells use numpy only. The browser has no torch, so none of this needs a graphics card, and each cell runs in well under a second.

Words you need8 terms

The share of the original picture left at step t. It starts near 1 and falls towards 0 as noise is added.

The noise that was mixed in, a draw from a bell curve with mean 0 and spread 1. The network is often trained to guess it.

The network's noise guess with no prompt, and with the prompt. Guidance uses the gap between them.

The direction in which the probability of the data rises fastest, at a given point and noise level.

How fast, and in which direction, a point should move at a given time. Flow matching trains a network to predict it.

The simplest way to follow a velocity: move by velocity times a small time step, then look again.

Replacing a hidden placeholder in a token list with a real token, a few positions at a time.

A gap between two bell curves that counts both the gap in centre and the gap in spread. FID is this distance between fitted curves for real and generated image features.

#The Conventions Used Here

Most mistakes in this area are not about maths. They are about which of two conventions a sentence uses. Fix them before writing code. This table is the single source for the whole lab, and the tests assume it.

Quantity	Meaning in this lab
Step index t	An array index from 0 to 999. t = 0 is the first, lightest noising step. t = 999 is almost pure noise.
beta_t	1000 values from 0.0001 to 0.02, evenly spaced (the linear schedule).
alpha_t	1 minus beta_t.
alpha bar_t	The running product of alpha_0 up to alpha_t. The signal fraction.
Guidance	eps_guided = eps_u + s * (eps_c - eps_u). s = 1 is the plain conditional model, s = 0 the unconditional one. The `guidance_scale` of the `diffusers` library is this s.
Flow time	t = 0 is pure noise and t = 1 is data. The point is x_t = (1 - t) x0 + t x1, with x0 the noise and x1 the data. The target velocity is x1 - x0.
SD3 and FLUX time	The other way round: t = 0 is data and t = 1 is noise, so their target is noise minus data.
Masked token	The number -1 in a list of token ids.

Two notes. First, the diffusion lesson writes steps from 1 to T in places, and its code indexes arrays from 0. Here the code indexes from 0 throughout, so "t = 499" means the 500th step. Second, the flow matching lesson and the SD3 paper disagree about which end is t = 0. Neither is wrong. Exercise 7 makes you write both.

#Exercise 1: Jump to Any Noise Level

In plain words. Adding noise one step at a time, 1000 times, is slow. But every step adds a little noise to a signal that was already scaled down a little, and the sum of many bell-curve draws is again a bell-curve draw. So you can jump straight from the clean picture x0 to the noisy picture at step t. The result is the clean picture scaled by the square root of the signal fraction, plus fresh noise scaled by the square root of what is left.

You write two functions. One builds the signal fraction alpha bar for all 1000 steps. The other makes the jump: x_t = sqrt(alpha bar_t) * x0 + sqrt(1 - alpha bar_t) * eps.

Hand trace. At t = 499 the signal fraction is 0.0786. Its square root is 0.2803, and the square root of 1 - 0.0786 is 0.9599. With x0 = 1.3 and eps = -0.4: x_t = 0.2803 x 1.3 + 0.9599 x (-0.4) = 0.3644 - 0.3840 = -0.0195. At t = 0 the signal fraction is 0.9999, and at t = 999 it is about 0.00004.

Mini project

Jump to any noise level

~7 min

Write make_alpha_bar, which builds the signal fraction for all 1000 steps, and q_sample, which jumps straight from clean x0 to the noisy x_t. Press Check my work to run the hidden tests.

Your codeEdits save in this browser

pythonrunnable cell

The test that most often catches a wrong answer averages many draws. The mean of x_t must be sqrt(alpha bar) times x0, and the variance must be 1 - alpha bar. If you used the plain alpha instead of the running product, the mean stays near x0 even at the middle of the chain, which is wrong by a factor of more than three.

#Exercise 2: One Reverse Step (DDPM)

In plain words. The network guesses the noise in x_t. DDPM (denoising diffusion probabilistic model) turns that guess into a mean for the slightly cleaner picture x_: take x_t, subtract a small multiple of the noise guess, and rescale. Then it adds a little fresh noise, so that the walk is random and not a straight line. On the very last step there is nothing left to blur, so no noise is added.

The mean is (x_t - beta_t / sqrt(1 - alpha bar_t) * eps_pred) / sqrt(alpha_t). The added noise is sqrt(beta_t) times a standard bell-curve draw z. The caller passes z in, which keeps the function deterministic and testable.

Hand trace. At t = 499: beta = 0.01004, alpha = 0.98996, so sqrt(alpha) = 0.99497. And sqrt(1 - alpha bar) = 0.9599. With x_t = 0.7 and eps_pred = 0.3: the factor is 0.01004 / 0.9599 = 0.01046, times 0.3 gives 0.00314. Then (0.7 - 0.00314) / 0.99497 = 0.7004. With z = 1 the noise adds sqrt(0.01004) = 0.1002, giving 0.8006. At t = 0 the answer is the mean alone, 0.6970 here.

Mini project

One reverse step (DDPM)

~7 min

Write ddpm_mean, the mean of the slightly cleaner picture x_{t-1} given a noise guess, and ddpm_step, which adds fresh noise on every step except the last. Press Check my work.

Your codeEdits save in this browser

pythonrunnable cell

A strong test feeds in the true noise. With the true eps, your mean must equal the exact posterior mean, which the test computes by a different formula that uses x0 directly. The two formulas agree only if you divided by the right square root.

#Exercise 3: A DDIM Step

In plain words. DDIM (denoising diffusion implicit model) reads the same noise guess differently. First it uses the guess to estimate the clean picture, x0_hat. Then it builds the less noisy picture from that estimate plus a pointer back towards the noise. It can jump between any two noise levels, not just neighbours, which is why a 50-step sampler works. A dial called eta sets how much fresh noise to add: eta = 0 adds none, so the sampler is deterministic, and eta = 1 gives back DDPM-like randomness.

Write the estimate first: x0_hat = (x_t - sqrt(1 - ab_t) * eps) / sqrt(ab_t). Then sigma = eta * sqrt((1 - ab_prev) / (1 - ab_t)) * sqrt(1 - ab_t / ab_prev). Then x_prev = sqrt(ab_prev) * x0_hat + sqrt(1 - ab_prev - sigma^2) * eps + sigma * z. Here ab_prev is the signal fraction you are going to, and it is larger than ab_t.

Hand trace. Go from t = 499 to t = 479 with x_t = 0.7 and eps = 0.3. The signal fractions are 0.07859 and 0.09579. The estimate is x0_hat = (0.7 - 0.9599 x 0.3) / 0.2803 = 1.4698. With eta = 0: 0.3095 x 1.4698 + 0.9509 x 0.3 = 0.4549 + 0.2853 = 0.7402. With eta = 1 and z = 1, sigma = 0.4198, the pointer term shrinks to 0.2560, and the answer is 0.4549 + 0.2560 + 0.4198 = 1.1307.

Mini project

A DDIM step

~8 min

Write predict_x0, the clean picture implied by a noise guess, and ddim_step, which jumps from noise level ab_t to a cleaner level ab_prev with a dial eta for fresh noise. Press Check my work.

Your codeEdits save in this browser

pythonrunnable cell

Two tests deserve a note. With eta = 0 and the true noise, DDIM must land exactly on the forward formula at the earlier level, because the same noise is reused. With eta = 1 and neighbouring steps, the part that has no randomness must equal the DDPM mean from exercise 2. That ties the two exercises together.

#Exercise 4: Classifier-Free Guidance

In plain words. To make a picture follow the prompt more closely, run the network twice: once with no prompt, once with the prompt. The gap between the two noise guesses points in the direction "more like the prompt". Guidance starts from the no-prompt guess and moves along that gap by a factor s. If s = 1 you get the plain prompt guess. If s = 0 you get the no-prompt guess. Values above 1, such as 7.5, push further than the model alone would.

In this track the formula is eps_guided = eps_u + s * (eps_c - eps_u). The original paper by Ho and Salimans writes (1 + w) * eps_c - w * eps_u. They are the same thing with s = 1 + w. You write the combination, the conversion, and a two-pass wrapper that calls a model with and without a prompt.

Hand trace. With eps_u = 0.2 and eps_c = 0.5 the gap is 0.3. At s = 3 the answer is 0.2 + 3 x 0.3 = 1.1. At s = 1 it is 0.5 and at s = 0 it is 0.2. For w = 6.5 in the paper's form, s = 7.5.

Mini project

Classifier-free guidance

~6 min

Write cfg_combine, which pushes the unconditional noise guess towards the conditional one by a factor s, s_from_ho_salimans, which converts the paper's w into s, and two_pass_guided, which calls the model with and without a prompt. Press Check my work.

Your codeEdits save in this browser

pythonrunnable cell

The tests check the two edge values, an order swap (the gap flips sign if you swap the two guesses), that the inputs are not changed in place, and that the wrapper calls the model exactly twice.

#Exercise 5: Velocity Prediction

In plain words. A network can be trained to guess three different things that carry the same information once x_t and the signal fraction are known: the noise eps, the clean picture x0, or a mix called the velocity v. The velocity is defined as v = sqrt(ab) * eps - sqrt(1 - ab) * x0. Why use it? At very high noise, guessing x0 is hard and guessing eps is easy. At very low noise it is the other way round. The velocity stays well behaved at both ends.

Once you have a v guess you can recover the others: x0 = sqrt(ab) * x_t - sqrt(1 - ab) * v and eps = sqrt(1 - ab) * x_t + sqrt(ab) * v. Notice that the two roles swap: one formula is the mirror of the other.

Hand trace. With ab = 0.0786, x0 = 1.3, eps = -0.4: x_t = -0.0195 (as in exercise 1). The velocity is 0.2803 x (-0.4) - 0.9599 x 1.3 = -0.1121 - 1.2479 = -1.3600. Recover x0: 0.2803 x (-0.0195) - 0.9599 x (-1.3600) = -0.0055 + 1.3055 = 1.3000. Recover eps: 0.9599 x (-0.0195) + 0.2803 x (-1.3600) = -0.0187 - 0.3812 = -0.4000.

Mini project

Velocity prediction

~6 min

Write v_target, which builds the velocity from x0 and eps, and x0_from_v and eps_from_v, which recover the clean picture and the noise from x_t and v. Press Check my work.

Your codeEdits save in this browser

pythonrunnable cell

The round trip tests use two extreme levels, ab = 0.99 and ab = 0.01. A formula with a wrong sign can still pass at one level by luck, but not at both.

#Exercise 6: Denoising Score Matching

In plain words. The score of a distribution at a point is the slope of its log probability: the direction in which the data looks more likely. We cannot compute the score of real image data. But if we add noise of size sigma to a data point, the score of that one noisy copy, taken relative to the original point, is known exactly: it is -eps / sigma, because the noisy point sits eps times sigma away from the clean one. So we train a network s(x_t) to match that target. After weighting by sigma squared, the loss is the average of (sigma * s(x_t) + eps)^2.

To test it we need a case where the true score is known. Take two bumps at -2 and +2, each with spread 0.5. After adding noise of size sigma, each bump has variance 0.25 + sigma^2, and the score is a weighted pull towards the bumps: each bump pulls with strength -(x - mu) / variance, weighted by how much that bump owns the point. You write that score, then the loss.

Hand trace. Take sigma = 0.8, so the variance is 0.25 + 0.64 = 0.89, and the point x = 1. The distances to the two bumps are 3 (from -2) and -1 (from +2). The ownership weights are about 0.011 and 0.989. The pulls are -3 / 0.89 = -3.371 and +1 / 0.89 = +1.124. The score is 0.011 x (-3.371) + 0.989 x 1.124 = 1.074. It is positive: at x = 1 the nearer bump is at +2, so the score points right. A network that always says 0 gets a loss of about 1.0 (the mean of eps squared). The true score gets about 0.46.

Mini project

Denoising score matching

~9 min

Write mixture_score, the exact score of two noisy bumps, and dsm_loss, the sigma-weighted denoising score matching loss. Press Check my work.

Your codeEdits save in this browser

pythonrunnable cell

import numpy as np

MU = np.array([-2.0, 2.0])      # two bumps, equal weight, each with spread 0.5
S0 = 0.5

def sample_mixture(n, rng):
  which = rng.integers(0, 2, n)
  return MU[which] + S0 * rng.standard_normal(n)

def mixture_score(x, sigma):
  """True score d/dx log p_sigma(x) of the mixture after adding noise of size sigma."""
  x = np.asarray(x, dtype=float)
  var = S0 ** 2 + sigma ** 2                     # each bump widens: 0.25 + sigma^2
  diff = x[:, None] - MU[None, :]                # (n, 2)
  # TODO: r[i, k] = share of point i owned by bump k = softmax over k of  -diff**2 / (2 * var)
  # TODO: score[i] = sum over k of r[i, k] * (-diff[i, k] / var)
  return np.zeros(len(x))

def dsm_loss(score_fn, x0, eps, sigma):
  """Denoising score matching, weighted by sigma^2."""
  x_t = x0 + sigma * eps
  # TODO: mean of (sigma * score_fn(x_t) + eps) ** 2     (the target of the score is -eps / sigma)
  return 0.0

rng = np.random.default_rng(0)
x0 = sample_mixture(20000, rng)
eps = rng.standard_normal(20000)
sigma = 0.8
print("loss with score = 0 :", round(dsm_loss(lambda x: 0 * x, x0, eps, sigma), 4))
print("loss with true score:", round(dsm_loss(lambda x: mixture_score(x, sigma), x0, eps, sigma), 4))
print("score at x = 1.0:", round(float(mixture_score(np.array([1.0]), sigma)[0]), 4))

Look at how the tests avoid trusting your formula. They find the slope of log p by a finite difference, which is a completely separate route, and require your score to match it. They also check that scaling the true score by 0.8 or 1.2 makes the loss worse, since the true score is the best possible guess, and that a flipped sign is worse than predicting zero.

#Exercise 7: Flow Matching and Euler

In plain words. Flow matching draws a straight line from a noise point x0 to a data point x1. At time t the point is (1 - t) * x0 + t * x1. Along that line the velocity never changes: it is x1 - x0, data minus noise. A network is trained to predict that velocity from x_t and t. To sample, you start at noise and follow the predicted velocity from t = 0 to t = 1 with small Euler steps: x becomes x + dt * v(x, t).

In this track t = 0 is noise and t = 1 is data. SD3 and FLUX flip it: t = 0 is data, t = 1 is noise, and so their target is noise minus data. You write the interpolation, the target in this track's convention, the SD3 target, and the sampler.

Hand trace. Take noise x0 = -1 and data x1 = 3. At t = 0.25 the point is 0.75 x (-1) + 0.25 x 3 = 0. The target is 3 - (-1) = 4. The SD3 target is -4. For sampling, imagine a perfect network that always points straight at the data: v(x, t) = (3 - x) / (1 - t). With one Euler step from x = -1 at t = 0, the move is 1 x 4 = 4, landing on 3. A straight path needs only one step if the network is perfect.

Mini project

Flow matching and Euler

~8 min

Write fm_interpolate, fm_target, sd3_target and euler_sample for flow matching in this track's convention (t = 0 is noise, t = 1 is data). Press Check my work.

Your codeEdits save in this browser

pythonrunnable cell

import numpy as np

# Track convention (flow-matching lesson): t = 0 is pure NOISE, t = 1 is DATA.
#   x_t = (1 - t) * x0 + t * x1     with x0 = noise and x1 = data
#   target velocity = x1 - x0

def fm_interpolate(x0, x1, t):
  """The point at time t on the straight line from noise x0 to data x1."""
  # TODO: (1 - t) * x0 + t * x1
  return x0

def fm_target(x0, x1):
  """The velocity the network should predict on that line."""
  # TODO: data minus noise
  return x0

def sd3_target(noise, data):
  """SD3 / FLUX write time the other way (t = 0 is data), so their target points from data to noise."""
  # TODO: noise minus data
  return data

def euler_sample(v_fn, x0, n_steps):
  """Walk from t = 0 (noise) to t = 1 (data) in n_steps equal Euler steps."""
  x = x0
  dt = 1.0 / n_steps
  for i in range(n_steps):
      t = 0.0
      # TODO: the time of step i is i * dt (starting at 0, never reaching 1)
      # TODO: move x by dt * v_fn(x, t)
  return x

x1 = 3.0                                       # one data point
oracle = lambda x, t: (x1 - x) / (1.0 - t)     # velocity that points straight at it
print(fm_interpolate(-1.0, 3.0, 0.25), fm_target(-1.0, 3.0), sd3_target(-1.0, 3.0))
print([round(float(euler_sample(oracle, -1.0, n)), 6) for n in (1, 2, 4)])

The sharpest test records the times at which your sampler calls the network. With 4 steps they must be 0, 0.25, 0.5 and 0.75: step i is taken at time i times dt, and the network is never asked about t = 1. Another test uses a curved velocity, 1 + 2t, where Euler really is inexact, and expects the known four-step answer of 1.75 against the exact 2.

#Exercise 8: Unmasking Tokens

In plain words. Masked generation starts with every position hidden and fills them in over several rounds. In each round the model gives a probability for every possible token at every position. The sampler takes the positions it is surest about, which means the highest probability after the softmax, and fills only those with their best token. Positions it is unsure about stay hidden for a later round, when more context is known.

You write two functions. One splits the positions over the rounds as evenly as possible, giving the first rounds the leftovers. The other does one round: pick the k most confident hidden positions, fill them, and leave everything already filled alone.

Hand trace. Ten positions over four rounds is 3, 3, 2, 2. For a round with logits [2, 0, 0], the probabilities are e^2 / (e^2 + 2) = 0.787 for the first token. Take five positions, where position 3 is already filled. The best-token confidences are 0.787 at position 0, 0.367 at position 1, 0.909 at position 2, and 0.690 at position 4. With k = 2 the sampler fills position 2 (token 1) and position 0 (token 0), and the list becomes [0, -1, 1, 2, -1].

Mini project

Unmasking tokens

~8 min

Write tokens_per_step, which splits positions over rounds as evenly as possible, and unmask_step, which fills the k most confident hidden positions. Press Check my work.

Your codeEdits save in this browser

pythonrunnable cell

import numpy as np

MASK = -1      # a position holding -1 is still hidden

def softmax(logits):
  e = np.exp(logits - logits.max(axis=-1, keepdims=True))
  return e / e.sum(axis=-1, keepdims=True)

def tokens_per_step(n_positions, n_steps):
  """Split n_positions over n_steps as evenly as possible; the first steps take the extras."""
  # TODO: use divmod; the first "extra" steps take base + 1, the rest take base
  return [0] * n_steps

def unmask_step(logits, tokens, k):
  """Fill in the k most confident masked positions. Never touch a position that is already filled."""
  tokens = np.array(tokens)                  # a copy
  probs = softmax(logits)                    # (L, V)
  best = probs.argmax(axis=-1)               # most likely token at each position
  conf = probs.max(axis=-1)                  # how sure the model is about it
  masked = np.flatnonzero(tokens == MASK)
  # TODO: sort the MASKED positions by conf, most confident first (ties go to the lower index)
  # TODO: write best[pos] into the first k of them
  return tokens

logits = np.array([[2.0, 0.0, 0.0], [0.1, 0.2, 0.0], [0.0, 3.0, 0.0], [1.0, 1.0, 1.0], [0.0, 0.0, 1.5]])
tokens = np.array([MASK, MASK, MASK, 2, MASK])
print(tokens_per_step(10, 4))
print(unmask_step(logits, tokens, 2))
print(unmask_step(logits, tokens, 9))

One test is built so that the largest raw logit and the largest probability point to different places. A position with logits [9, 9] has a huge logit but only a 50 percent best choice. Ranking by raw logits picks it, which is wrong.

#Exercise 9: FID from Features

In plain words. FID compares two clouds of feature vectors, one from real images and one from generated images. It fits a bell curve to each cloud: a centre (the mean) and a spread (the covariance matrix). The Frechet distance between the two curves is the squared distance between centres, plus a term for the spread: tr(S1 + S2 - 2 * (S1 S2)^(1/2)). The hard part is the matrix square root. For matrices like these, numpy can do it with an eigen-decomposition: split the matrix into directions and strengths with np.linalg.eigh, take the square root of each strength, and put it back together.

The product S1 S2 is not symmetric, so its root is awkward. A neat fix: with R the square root of S1, the matrix R S2 R is symmetric and has the same eigenvalues as S1 S2. So tr((S1 S2)^(1/2)) equals tr((R S2 R)^(1/2)), and you can use your symmetric square root twice.

Hand trace. Take two 2-dimensional curves with the same centre and diagonal spreads S1 = diag(1, 4) and S2 = diag(9, 1). The traces add to 5 + 10 = 15. The matrix S1 S2 = diag(9, 4) has square root diag(3, 2) with trace 5. So the distance is 15 - 2 x 5 = 5. That equals (1 - 3)^2 + (2 - 1)^2, the sum of squared gaps in standard deviation on each axis. With identity spreads and centres 3 and 4 apart on the two axes, the distance is 3 squared plus 4 squared, which is 25.

Mini project

FID from features

~10 min

Write psd_sqrt, a symmetric matrix square root, frechet_distance between two Gaussians, and fid_from_features, which fits the Gaussians to two feature sets. Press Check my work.

Your codeEdits save in this browser

pythonrunnable cell

Real FID uses 2048 features and tens of thousands of images, so the covariance is large and the square root is slower, which is why libraries use dedicated routines. The formula you wrote is the same. The dimension is only a size.

#Where the Traps Are

Every exercise has one common wrong answer that looks right. They fall into three families.

Sign. The velocity has a minus sign on the x0 term. The score points uphill, so for a bump it is minus the distance. The guidance gap is conditional minus unconditional, not the reverse. The flow target is data minus noise here and noise minus data in SD3. A wrong sign gives a smooth, plausible-looking number, so only a test with a known answer catches it.

Time direction. Diffusion code sweeps from high noise to low: t counts down, and the signal fraction rises towards 1 as you go. DDIM goes to a larger ab_prev. Flow code in this track sweeps t up from 0 to 1. SD3 and FLUX sweep from 1 down to 0. When you read a sampler, first find which end is data. And an Euler loop over n steps visits times 0 to (n-1)/n, never 1.

Alpha versus alpha bar. There are two lookalikes. alpha_t = 1 - beta_t is the scale of one single step. alpha bar_t is the running product, the scale of the whole chain from the clean picture. The jump to x_t uses alpha bar. The reverse mean uses both: beta_t over the square root of one minus alpha bar, then dividing by the square root of the plain alpha. Here is the size of the mistake. At t = 499 the plain alpha is 0.98996, with square root 0.995, while alpha bar is 0.0786, with square root 0.280. Using the first in the jump leaves 99.5 percent of the picture where 28 percent should be.

Symptom	Likely cause
Sampled pictures are washed-out, or all noise, or burned in	The guidance gap was reversed, or s was set far too high, or alpha was used where alpha bar belongs
A sampler runs without error but ends as pure noise	Time direction reversed: it walked from clean to noisy
Training loss never moves from about 1	The target sign is wrong, so the best the network can do is predict zero
A deterministic sampler is not deterministic	A noise draw is still being added when eta is 0, or the same z is reused in the wrong place
Last step looks grainy	Noise was added at t = 0, the final step
Masked sampler overwrites a token it already chose	The update did not skip already-filled positions

#Capstone Checklist

Use this list to check that you are ready to read and write real diffusion and flow code. Tick each item only if you can do it without looking.

I can write the forward jump and say which of alpha and alpha bar it uses, and why
I can write the DDPM mean from a noise guess, and say when no noise is added
I can write a DDIM step, set eta to get a deterministic or a random sampler, and say what happens at ab_prev = 1
I can write classifier-free guidance, state what s = 0 and s = 1 give, and turn a Ho and Salimans w into s
I can convert between noise, clean image and velocity predictions in both directions
I can write the score matching loss and name the target the network is matched to
I can write a flow matching target and an Euler sampler, and say which end is data in this track and in SD3
I can write a confidence-based unmasking round that never overwrites a chosen token
I can write the Frechet distance with an eigen-decomposition square root
Given a sampler I have never seen, I can find the sign, the time direction and the alpha-or-alpha-bar choice in under a minute

If you can tick every line, you can read the sampler code of a library and spot a convention mismatch. If you cannot, go back to the exercise for that line, open the solution, and write the function again without it.

Denoising Diffusion Probabilistic Models

Jonathan Ho, Ajay Jain, Pieter Abbeel (2020)

The forward jump and the reverse mean from exercises 1 and 2.

Denoising Diffusion Implicit Models

Jiaming Song, Chenlin Meng, Stefano Ermon (2021)

The deterministic step with the eta dial from exercise 3.

Progressive Distillation for Fast Sampling of Diffusion Models

Tim Salimans, Jonathan Ho (2022)

Introduces the velocity prediction of exercise 5.

Flow Matching for Generative Modeling

Yaron Lipman and others (2023)

The straight-line target and Euler sampling of exercise 7.

#Key Takeaways

The jump to any noise level uses alpha bar: x_t = sqrt(alpha bar) x0 + sqrt(1 - alpha bar) eps. The single-step alpha is a different number
The DDPM mean is (x_t - beta / sqrt(1 - alpha bar) * eps) / sqrt(alpha), with noise added on every step except the last. DDIM goes through an x0 estimate and an eta dial, and with eta = 1 it matches DDPM
Guidance in this track is eps_u + s * (eps_c - eps_u). Ho and Salimans' w gives s = 1 + w
Velocity v = sqrt(ab) eps - sqrt(1 - ab) x0 lets you recover x0 and eps with two mirror formulas
The denoising score matching loss is the mean of (sigma s + eps) squared, and a true score beats every scaled copy of itself
Flow matching here runs from noise at t = 0 to data at t = 1 with target x1 - x0. SD3 and FLUX run the other way. A straight path needs one Euler step
Unmasking ranks by softmax probability, never by raw logit, and leaves chosen tokens alone. FID is a Frechet distance whose matrix square root comes from an eigen-decomposition

#Quick Check

Quick Check1 / 4

In the forward jump, which number scales the clean picture x0?

Where to next. Revisit Diffusion Models and read its sampler code with the trap table in mind. For how the same ideas run in a large system, see the Modern GenAI Landscape. To build a working app on top of them, continue to Building a Diffusion App.

Generative Expert Lab

Learning Objectives

After this lesson, you will be able to:

Write the forward noising jump, the DDPM reverse mean, and the DDIM step with its eta dial, and say which quantities each one uses
Combine two noise guesses with classifier-free guidance in the track convention, and translate the Ho and Salimans form into it
Convert between noise, clean image and velocity predictions using only the signal fraction
Write the denoising score matching loss and check a true score against a slope found by finite differences
Write the flow matching target and an Euler sampler, and say which way time runs in this track and in SD3 and FLUX
Write a confidence-based unmasking step and a Frechet distance, and know the three traps that break most implementations: sign, time direction, and alpha versus alpha bar

Prerequisites

Before You Start

Review

#How This Lab Works

Three things about the tests are worth knowing before you start.

They are tolerance based. A result close enough to the right answer passes, so a harmless rounding difference never fails you.
They are deterministic. Any random numbers come from a fixed seed, so a test that fails today fails the same way tomorrow.
They check more than the one example in the hand trace. Some compare your answer with a second route to the same number, such as a slope found by finite differences or a closed form for a simple case. If your code only matches the trace, it will still fail those.