Sora made a 60-second cinematic video. Veo 3 (Google DeepMind) generates 8K at 60fps with native synchronized audio. Runway Gen-3 ships in production for filmmakers. HunyuanVideo and Wan 2.2 are 13B+ open-weights models you can fine-tune on two H100s. Video generation is where image gen was in 2022 — about to explode into every workflow.
Learning Objectives
After this lesson, you will be able to:
Explain why video generation is fundamentally diffusion in space PLUS time, and what each new dimension costs in compute and memory
Compare the four production architectures (Sora's diffusion transformer, Runway Gen-3, Veo, open-weights HunyuanVideo) and pick which one fits a given use case
Describe how a latent video VAE compresses 60 frames of pixels into ~8 frames of latents, and why this single trick made multi-second video generation tractable
Diagnose the three classic video-gen failure modes — flicker, character drift, and physics breakdown — and know which architectural choice mitigates each
Don't worry if "diffusion in 4D" sounds intimidating — once you see how the latent VAE compresses time into a few frames-worth of latents, the architecture clicks fast. Video generation is image generation with an extra axis.
Sora, Veo 2, FLUX, SD3, and most modern video diffusion models share a common backbone: the Diffusion Transformer or DiT (Peebles & Xie 2023). It replaces the U-Net that powered Stable Diffusion 1 and 2.
DiT block: x←x+MHSA(LN(x),c)x←x+MLP(LN(x),c)where c=embed(t,y) injects timestep t and conditioning y via AdaLN
The breakthrough Sora demonstrated: the DiT scales just like LLMs do. Train a bigger DiT on more video data, and quality keeps climbing — there's no obvious wall yet.
Sora's technical report describes inputs as "spacetime patches." A 60-frame, 256x256 video gets divided into a grid of 3D cubes (e.g., 4 frames x 16 pixels x 16 pixels per patch), and each cube becomes a single token in the transformer.
Number of tokens for a T×H×W video: N=ptT⋅phH⋅pwW
Self-attention over thousands of spacetime tokens is expensive. Most models factorize it.
Full spacetime attention: O(N2)=O((THW)2/p6)Factorized: O((T)2⋅(HW)/p4)+O((HW)2⋅T/p4)
Sora reportedly uses full spacetime attention; Veo 2 uses some form of factorization; HunyuanVideo factorizes. The trade-off: full attention is more expressive but ~10-100x more expensive at typical resolutions.
The single trick that makes multi-second video generation feasible is the latent video VAE. Like the Stable Diffusion image VAE, it has an encoder that compresses pixels to latents and a decoder that reconstructs pixels from latents — but it compresses BOTH spatial and temporal dimensions.
Building a good video VAE is hard. The decoder must produce temporally coherent reconstructions — small errors in latent space can manifest as flicker. HunyuanVideo's open-weights VAE is the current open standard; Sora's and Veo's VAEs are proprietary but presumably similar in spirit.
If you had to bet which training objective Sora uses — DDPM noise prediction, score matching, or flow matching — which would you pick based on what 2024+ frontier labs have converged on?
Modern frontier video models (FLUX, SD3, and likely Sora and Veo) train with flow matching rather than the original DDPM noise-prediction objective. Why?
Flow matching admits straighter probability paths from noise to data, which empirically converges faster and produces better samples per training FLOP.
The probability flow ODE view makes deterministic, fast samplers natural — useful for video where each step is expensive.
Conditional flow matching lets you condition on text, image, or other video frames cleanly via the velocity field.
(Full coverage of flow matching is in the dedicated Flow MatchingFlow MatchingFlow matching learns a vector field that transports a simple distribution (noise) to a complex one (data) along straight paths, enabling fast generation.Learn more → lesson.)
#Try It: Image-to-Video with Stable Video Diffusion
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
Tests · Run the script. You should produce 14 frames as an MP4. Inspect that the first frame matches the input image and that subsequent frames have plausible motion (camera pan, slight subject movement). At motion_bucket_id 200, look for character drift or texture morphing.
Video gen is diffusion in space + time. Add a temporal dimension to image diffusion and you get video diffusion. The math is the same; the cost is brutal.
Latent video VAE is the make-or-break trick. Compressing both spatial and temporal dimensions before diffusion gives a ~50-90x reduction in compute. Without it, multi-second video generation would be intractable.
DiT is the dominant backbone. Sora, Veo 2, FLUX, SD3 all use diffusion transformers. The architecture scales with parameters and data the way LLMs do.
Three failure modes recur. Flicker (insufficient temporal attention), character drift (weak long-range coherence), physics breakdown (insufficient world-model training). Each architectural choice mitigates a different one.
Open-weights closed the gap fast. HunyuanVideo, Mochi-1, and CogVideoX reached Runway Gen-3 territory in 2024-2025. Frontier still leads on long-form, but the cliff is smaller every quarter.
Why does almost every modern video diffusion model use a latent VAE that compresses time as well as space?
Next up: 3D Generation — DreamFusion, NeRF, and 3D Gaussian Splatting. Where video adds time to images, 3D adds depth, and the techniques to lift 2D priors into 3D structure.