Apple Vision Pro and Meta Quest 3 fill their displays with photoreal scenes captured by Gaussian Splatting from a phone video. Luma AI, Polycam, and Scaniverse turned photogrammetry into a one-tap consumer feature. Text-to-3D models (DreamFusion, Magic3D, Shap-E, Meshy) let you type "a fox in a wizard hat" and get a riggable mesh in 60 seconds. The trick: piggyback on a 2D diffusion model and lift its supervision into 3D with score distillation.
Learning Objectives
After this lesson, you will be able to:
Explain how a Neural Radiance Field (NeRF) represents a 3D scene as a tiny neural network — and why volumetric ray marching is what turns that network into a rendered image
Describe 3D Gaussian Splatting as the explicit, fast cousin of NeRF — a scene stored as a cloud of fuzzy ellipsoids that you alpha-blend in screen space
Use Score Distillation Sampling (SDS) to turn any text-to-image diffusion model into a text-to-3D generator — even though no 3D ground truth exists
Pick the right 3D representation — NeRF, Gaussian Splatting, mesh — for a given task, weighing render quality, training time, editability, and downstream tooling
Don't worry if "3D generation" sounds like a different field from image generation — it actually rides on the same diffusion machinery underneath. The key trick is just learning a 3D representation that diffusion can guide.
Generative 3D divides cleanly along two axes: what representation (implicit NeRF / explicit Gaussian / mesh / point cloud) and how to learn it (per-scene optimization from photos / 2D-diffusion-as-prior via SDS / native 3D diffusion trained on 3D datasets).
A NeRF is a function F_θ : (x, y, z, θ_view, φ_view) → (r, g, b, σ) — given a point in space and a viewing direction, return color and density. This function is a small MLP (typically 8 layers, 256 hidden units).
Volumetric rendering integral. To render a pixel, shoot a ray from the camera through that pixel. Sample N points along the ray. Evaluate the MLP at each point. The pixel's color is a weighted integral of the per-sample colors, where the weights come from accumulated transmittance and per-sample density.
Training signal. Photometric loss: render the same pixels you have photographs of, take MSE between rendered pixel and ground-truth pixel, backpropagate through the rendering integral and into the MLP weights. With 30-100 photos taken from different angles, the MLP converges to a coherent 3D scene.
Why it took off: photographs are everywhere; 3D scans are rare. NeRF only needs photos + camera poses (which COLMAP can recover automatically). It produces view-consistent renders with proper occlusion and lighting "for free."
Why pure NeRF is no longer the production choice: training is slow (1-2 hours even with Instant-NGP speedups), rendering is slow (1-30 seconds per frame), editing is hard (the scene is locked inside MLP weights — try moving one chair in the scene).
Take ~1M tiny 3D Gaussians, each parameterized by:
Position μ ∈ ℝ³
Anisotropic covariance Σ (encoded via rotation quaternion + scale vector for stability)
Color c ∈ ℝ³ (sometimes spherical harmonics for view-dependent color)
Opacity α ∈ [0, 1]
To render: project each 3D Gaussian onto the 2D screen (becomes a 2D Gaussian via the projection of Σ), sort by depth, alpha-blend front-to-back.
C(p)=i=1∑NciαiGi(p)j=1∏i−1(1−αjGj(p))
Adaptive density control. The clever training trick: every few hundred steps, look at the gradient on each Gaussian. If a Gaussian has high gradient (the loss says it should be doing more work), clone or split it. If a Gaussian's opacity drops below a threshold, prune it. Start with a sparse point cloud from COLMAP and end with ~1-5M Gaussians that fully describe the scene.
Why it won production. Trains in 7-30 minutes. Renders at 30-100 FPS on a laptop GPU. Easy to ship — the scene is just a list of Gaussians, viewable in a Three.js scene, AR Quick Look, or Unity plugin. Edit-friendly — you can literally grab a subset of Gaussians and move them.
The big problem with native 3D generation: 3D training data is scarce. Objaverse has ~10M assets; the internet has billions of images. Could we leverage 2D image generation models (which are amazing) to generate 3D?
Score Distillation Sampling (SDS) is the answer. Take a 3D representation (NeRF or Gaussian Splatting) parameterized by θ. Render a random view to get image x. Pretend x is a sample from a diffusion model and ask: "what is the score (gradient of log p) at this image, conditioned on the text prompt y?" The pretrained diffusion model gives you that gradient — the same score function from the stochastic-calculus / SDEStochastic CalculusStochastic calculus extends derivatives to random processes — Brownian motion, Itô integrals, and stochastic differential equations. The math diffusion models, flow matching, and score-based generative models all live in.Learn more → view of diffusion. Backpropagate the gradient through the renderer into θ.
Variants worth knowing.Magic3D (NVIDIA 2022) — coarse SDS into mesh, then high-res refinement. Fantasia3D (Chen 2023) — disentangles geometry from appearance. ProlificDreamer (2023) — variational score distillation (VSD), fixes the over-smoothed Janus-face problem of vanilla SDS.
Native 3D diffusion is the 2024+ alternative. Shap-E (OpenAI 2023) trains a diffusion model directly on 3D representations. LRM (Large Reconstruction Model), Instant3D, One-2-3-45, Zero123, SyncDreamer — feed-forward image-to-3D models that don't require per-asset SDS optimization. Genie 3 (DeepMind 2024) generates explorable interactive 3D worlds from a prompt.
What Do You Think?
You're building a tool that lets a furniture retailer scan a chair from 30 phone photos and serve a 3D viewer in the browser. Which representation do you use?
Gaussian Splatting wins this one — it trains in minutes, runs in real time, and the Three.js / Babylon.js communities both have mature in-browser viewers. NeRF would render too slowly. DreamFusion ignores your actual photos. Classical mesh-based MVS struggles with reflective or thin surfaces (chair frames!). The retailer can ship the splat directly to customers.
Implicit (NeRF) vs explicit (Gaussian Splatting) is the central tradeoff. NeRF stores the scene in MLP weights and queries it per-ray; Gaussian Splatting stores millions of explicit blobs and rasterizes them. Implicit wins on storage and photorealism on translucent scenes; explicit wins on training and rendering speed.
Volumetric rendering is the bridge from differentiable graphics to deep learning. NeRF's contribution wasn't a new neural architecture, it was making the volume-rendering integral differentiable so gradient descent could fit it from photographs.
Gaussian Splatting's adaptive density control is the secret sauce. Clone-and-split where loss is high, prune where opacity is low. This turns gradient descent on millions of parameters into a stable training loop.
Score Distillation Sampling lifts 2D diffusion to 3D. DreamFusion proved you don't need a 3D-trained diffusion model to do text-to-3D. Just borrow gradients from any pretrained 2D model and chain-rule them through your renderer.
The 2024-2025 wave shifted from per-scene optimization to feed-forward 3D diffusion. LRM, Instant3D, Shap-E, Genie 3. Slow SDS is for bespoke hero assets; fast feed-forward is for everything else.
What is the key difference between NeRF and 3D Gaussian Splatting?
3D bridges generative AI from screens into worlds. Audio bridges it from screens into ears — the next lesson covers TTS, music, and voice cloning, and how neural audio codecs turned sound into a sequence of tokens that LLMs can speak.