Sora generates 60-second cinematic videos. Midjourney v7 ships production-grade images in 30 seconds. ElevenLabs clones voices in 10 seconds. FLUX.1 Pro renders 4K photoreal stills locally on a single GPU. Generative AI is the most economically disruptive technology of the decade — and the math is simpler than most engineers think.
Learning Objectives
After this lesson, you will be able to:
Know the difference between discriminative models (classifiers) and generative models (creators)
Understand latent spaces -- the hidden coordinate system where generative models do their work
Identify the main generative model families (VAEs, GANs, diffusion, flows) and when to use each
Explain the manifold hypothesis and why it makes learning p(x) over high-dimensional data tractable
Welcome to generative AI -- the field that powers everything from AI art to ChatGPT to AI music. This is where machine learning goes from "analyzing things" to "creating things." It is a genuine paradigm shift, and understanding these foundations will make every subsequent lesson click.
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
A discriminative model learns the boundary between classes. A generative model learns the entire data distribution. The generative task is fundamentally harder -- and fundamentally more powerful.
Every supervised ML model you have seen so far has been discriminative. Given an input x, predict a label y. A cat classifier does not understand what a cat looks like in its entirety -- it only understands what makes a cat different from a dog. It learns the decision boundary, not the data distribution.
Try it: Explore the Latent SpaceInteractive
Loading visualization...
Try it! Open any AI image generator (Midjourney, DALL-E, or even a free one like Playground AI). Type the same prompt three times. Notice how you get three different but plausible images each time. That is sampling from a learned distribution -- the model is not retrieving stored images, it is creating new ones from its understanding of what images look like.
A generative model learns the probability distribution of the data itself. Once you have learned p(x), you can:
Generate new samples by drawing from p(x)
Evaluate how likely a given sample is
Interpolate between samples by walking through the distribution
Condition generation on prompts, labels, or other signals
This is the difference between a critic who can say "this is a good painting" and an artist who can actually paint.
The discriminative model learns p(y|x) -- "given this image, what class is it?" The generative model learns p(x) -- "what does the space of all possible images look like?"
By Bayes' theorem, if you know p(x) and p(y|x) you can derive anything. But learning p(x) is exponentially harder. An image with 256x256 pixels and 3 color channels has 256^(2562563) possible values. The generative model must learn which of these astronomical possibilities are "real" images and which are noise.
What Do You Think?
Why can't we just explicitly model p(x) with a lookup table or histogram for images?
The answer is the curse of dimensionality. A 256x256 RGB image lives in a space with ~200,000 dimensions. The number of possible pixel configurations is larger than the number of atoms in the observable universe. Yet real images -- faces, landscapes, cats -- occupy a vanishingly thin manifold within this space. A random pixel configuration is almost certainly noise. The generative model must find and parameterize this thin manifold.
The manifold hypothesis is not just a philosophical idea -- it is the reason generative models work at all. If images truly used all 200,000 dimensions independently, we could never learn p(x) from a finite dataset. But because the data lives on a low-dimensional manifold, a generative model can parameterize that manifold with a manageable number of latent variables.
Every generative model works with a latent space -- a lower-dimensional space where each point corresponds to a possible output. The model learns a mapping from this simple latent space to the complex data space.
z∼p(z),x=G(z)where z∈Rd,x∈RD
The magic of latent spaces is that they are structured. Nearby points in latent space produce similar outputs.
Figure
A 2D latent space where each point maps to a generated image -- nearby points produce similar faces, distant points produce different faces.
You can:
Interpolate: Walk smoothly between two faces by interpolating their latent vectors
Disentangle: Find directions in latent space that correspond to individual attributes (hair color, age, expression)
This structure emerges automatically from training. The model discovers that organizing the latent space this way makes reconstruction or generation easier.
The field has converged on four major families of generative models, each with a different strategy for learning p(x). The deep-learning track covers the architectural side of GANs and VAEsGAN & VAE FoundationsGANs train a generator-discriminator pair adversarially; VAEs learn an encoder-decoder with a KL-regularized latent prior. The two pre-diffusion deep generative families and the architectures the generative-AI track builds on.Learn more → — the pre-diffusion deep generative families this lesson builds on.
How do you evaluate a generative model? You cannot just compute accuracy like a classifier. The field uses several metrics:
Frechet Inception Distance (FID):
Figure
Two overlapping bell curves: one for real image features, one for generated -- FID measures the gap between them.
Compares statistics of generated images to real images in a feature space. Lower is better. FID captures both quality (are individual images realistic?) and diversity (does the model generate varied outputs?).
CLIP Score: For text-conditioned generation, measures how well generated images match their text prompts using the CLIP model.
Precision & Recall (Kynkäänniemi 2019): Modern complement to FID that decomposes "quality" (precision = fraction of generated samples on the data manifold) from "coverage" (recall = fraction of the data manifold covered). Catches mode collapse that FID alone can miss.
DINOv2-FID / FD-DINOv2: A drop-in replacement for FID that swaps Inception-v3 features for DINOv2 features. DINOv2's self-supervised features correlate far better with human preference and are the 2024+ default for SOTA papers. (Note: classic Inception Score is deprecated for rigorous evaluation — use FID + DINOv2-FID + precision/recall instead.)
2023: Consistency models, flow matching -- faster generation without quality loss
2024: Sora (60s video), Veo, Gen-3 — text-to-video crosses the quality threshold
2025: FLUX.1 Pro, SD 3.5, Midjourney v7, Veo 3 (8K@60fps), open-weight video models go mainstream; flow matching and consistency distillation become standard for production inference
#Why Generative AI Changed Everything in 2022-2025
The years between 2022 and 2025 represent the most transformative period in AI history. What changed was not just technical capability -- it was the public's relationship with artificial intelligence. For the first time, non-technical people could interact with AI systems that felt creative, useful, and sometimes unsettlingly human.
June 2020 -- GPT-3 (OpenAI): The 175-billion-parameter language model that demonstrated emergent abilities -- few-shot learning, code generation, creative writing -- without being explicitly trained for these tasks. GPT-3 was the "proof of concept" that scale alone could unlock capabilities. Access was limited to an API, but the demos shocked the research community. This was the moment the scaling hypothesis became consensus.
January 2021 -- DALL-E (OpenAI): The first model to convincingly generate images from arbitrary text prompts. The images were low-resolution (256x256) and often distorted, but the concept was electrifying: type "an armchair shaped like an avocado" and a machine creates it. DALL-E proved that the text-to-image bridge was crossable. The architecture (VQ-VAE + autoregressive transformer) would soon be superseded by diffusion, but the idea was permanently planted in the public imagination.
April 2022 -- DALL-E 2 (OpenAI): The jump from DALL-E to DALL-E 2 was staggering. Photorealistic images at 1024x1024, inpainting, outpainting, and variations. The secret: replacing the autoregressive approach with a CLIP-guided diffusion model. DALL-E 2 showed that diffusion models could be conditioned on text through CLIP embeddings, a pattern that would define the next two years of image generation.
August 2022 -- Stable Diffusion (Stability AI): The democratization moment. Stable Diffusion was open-source, ran on consumer GPUs (a single 8GB GPU), and produced images rivaling DALL-E 2. Within weeks, thousands of tools, plugins, and fine-tuned models appeared. The key innovation was latent diffusion -- performing the diffusion process in a compressed latent space (64x64) rather than pixel space (512x512), reducing compute requirements by 10-100x. This single architectural decision turned image generation from a cloud-only capability into something anyone could run at home.
November 2022 -- ChatGPT (OpenAI): The application that changed the world's understanding of AI overnight. Built on GPT-3.5 with RLHF (reinforcement learning from human feedback), ChatGPT reached 100 million users in two months -- the fastest-growing consumer application in history. The key was not the model itself but the conversational interface that made AI accessible to everyone. ChatGPT proved that the generative AI revolution was not just about images -- language generation was the killer application.
March 2023 -- GPT-4 (OpenAI): A massive leap in reasoning, instruction-following, and multimodal understanding (text + images). GPT-4 passed the bar exam (90th percentile), solved complex coding problems, and demonstrated genuine cross-domain expertise. The model showed that scaling continued to produce qualitative improvements, not just quantitative ones. GPT-4 was also the first commercially deployed large multimodal model.
March 2023 -- Claude (Anthropic): Anthropic's Claude introduced a different philosophy to AI development: Constitutional AI (CAI), where the model is trained to be helpful, harmless, and honest using a set of principles rather than pure RLHF. Claude demonstrated that safety and capability were not necessarily in tension. By 2024-2025, Claude evolved into one of the most capable AI systems available, with Claude 3.5 Sonnet and Claude Opus 4 excelling at coding, analysis, and extended reasoning tasks.
February 2024 -- Sora (OpenAI): Text-to-video generation that stunned the world. Sora generated 60-second, 1080p videos with temporal coherence, realistic physics, and cinematic quality. The underlying architecture -- a spatiotemporal diffusion transformer operating on compressed video patches -- showed that the same principles powering image generation could extend to the temporal domain. While not publicly released initially, Sora signaled that video generation had crossed the quality threshold.
2024-2025 -- The Open-Source Explosion: Llama 3 (Meta), Mistral (Mistral AI), FLUX (Black Forest Labs), Stable Diffusion 3, Qwen (Alibaba), and dozens of other open models reached or exceeded the quality of proprietary systems from just 12 months earlier. The "open vs. closed" debate intensified. Fine-tuning with LoRA and QLoRA made it practical to customize billion-parameter models on a single GPU. The ecosystem of tools -- LangChain, LlamaIndex, vLLM, Ollama, Hugging Face -- matured into a complete open-source AI stack.
This 2022-2025 window was the "internet moment" for AI -- the transition from research curiosity to foundational technology. Just as the web went from academic tool (1993) to mainstream infrastructure (2000) in seven years, generative AI went from "interesting demo" to "embedded in every product" in three years. The speed of adoption was unprecedented because the distribution channels (cloud APIs, app stores, browser-based tools) already existed. Understanding this timeline is not just history -- it is context for every technical and product decision you will make in AI.
Generative models learn the data distribution itself. Unlike discriminative models that draw boundaries between classes, generative models learn P(data) and can create entirely new samples from the distribution
The manifold hypothesis explains why generation works. High-dimensional data (images, text) actually lives on low-dimensional surfaces; generative models learn to navigate these manifolds
Latent spaces compress data into meaningful representations. By encoding data into compact latent vectors, generative models capture the essential factors of variation (pose, lighting, expression) as navigable dimensions
Each generative family has distinct strengths. VAEs provide structured latent spaces, GANs produce sharp samples, diffusion models offer the best quality-diversity tradeoff, and flows provide exact likelihood computation
What is the fundamental difference between discriminative and generative models?
You now understand the fundamental distinction between discriminative and generative models, and why latent spaces and the manifold hypothesis make generation possible. Next up: Autoencoders and VAEs -- your first hands-on generative architecture that compresses data into a structured latent space and generates new samples from it.