This lesson shifts from "learn by doing" to "learn by thinking." Model-based RL is how AlphaGo imagined millions of possible games before each move, how MuZero plans without even being told the rules, and how Waymo's planner runs counterfactuals on every intersection. It's the closest AI comes to thinking before acting — and it's one of the most sample-efficient ways to learn.
Learning Objectives
After this lesson, you will be able to:
Understand the model-free vs model-based distinction: model-free agents (Q-learning, PPO, SAC) learn entirely from real interaction; model-based agents learn a transition model P̂(s'|s,a) and reward model R̂(s,a), then use them for planning or generating synthetic experience
See how the Dyna-Q architecture blends both approaches: each real step triggers k simulated Q-learning updates from the learned world model, dramatically accelerating convergence
Understand how MCTS + neural networks power AlphaGo and AlphaZero: the policy network focuses search on promising moves; the value network replaces random rollouts; the UCB formula balances exploration vs exploitation across the tree
Know the tradeoffs of model-based methods: higher sample efficiency but risk of model exploitation -- policies that look great inside an inaccurate model but fail in the real world
Every RL algorithm we have studied so far is model-free: the agent interacts with the environment, collects rewards, and adjusts its behavior. It never builds an internal representation of how the environment works. It learns what to do without understanding why.
Model-based RL takes a different approach: learn a model of the environment, then use it to plan.
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
How AlphaGo imagined millions of future moves before making one -- when AlphaGo defeated the world Go champion, it was not just reacting to the board; it had a mental model of Go and used Monte Carlo Tree Search to simulate thousands of future games in its head for every single move, picking the path most likely to win; this "think before you act" approach is model-based RL, and AlphaZero later used it to learn chess in just 4 hours and become the strongest player in history
How self-driving cars practice dangerous scenarios without anyone getting hurt -- Waymo and Tesla cannot wait for a real child to run into the road to learn how to brake; instead they build a learned model of the driving world and simulate millions of scary scenarios (ice patches, jaywalkers, sudden lane changes) inside a computer, training the car's brain in a safe virtual world before it ever touches a real street
Think of it like planning your route before leaving the house -- model-free RL is like exploring a new city by wandering randomly until you find the restaurant; model-based RL is like checking Google Maps first, simulating different routes in your head, and picking the best one before you even step outside; the planning takes extra brainpower, but you arrive way faster
Build this --> Build a planning agent: create a simple grid game where the agent first learns a "world model" (predicting what happens for each action), then uses that model to mentally simulate 5 steps ahead before making a real move -- compare its performance against a model-free agent that learns only from trial and error
Transition model + reward model, then derives policy
Planning
No internal planning; reactive
Simulates future states to plan ahead
Sample efficiency
Low (needs millions of interactions)
High (learns from real + imagined experience)
Computation
Low per step (just forward pass)
High per step (must simulate/plan)
Model bias
None (learns from real data)
Can fail if the learned model is inaccurate
Examples
Q-learning, PPO, SAC
AlphaGo, MuZero, Dreamer
What Do You Think?
A robot needs to learn to stack blocks. It can try about 100 real-world attempts per day. Which approach is better: model-free or model-based?
With only 100 real attempts per day, model-free RL would take months to years. Model-based RL can learn a physics model from those 100 attempts, simulate thousands of additional attempts internally, and converge much faster. This is why model-based approaches dominate real-world robotics.
Try it! Next time you plan a route somewhere, notice that you are doing model-based planning in your head: "If I take Highway 1, there might be traffic. If I take the side streets, there are stop signs but no congestion." You are simulating futures without actually driving them. That is exactly what model-based RL does.
Dyna (Sutton, 1991) elegantly combines model-free and model-based learning:
Take a real action, observe result, update Q-values (model-free)
Use the result to update the world model (model learning)
Use the world model to generate k simulated experiences (planning)
Update Q-values on simulated experiences too (model-based)
For each real step: k simulated updates via Q(s,a)←Q(s,a)+α[R^+γa′maxQ(s^′,a′)−Q(s,a)]
Try it: Model-Based Planning in a Grid WorldInteractive
Watch how a model-based agent learns faster than a model-free one. The agent builds an internal model of the grid world's transitions and rewards, then uses that model to simulate additional experiences (Dyna-style planning). Compare how quickly Q-values converge with and without model-based planning updates.
Loading visualization...
Figure
A tree diagram showing MCTS in action -- the root node (current state) branches into possible moves, each explored to different depths based on promise, with the most-visited branch highlighted as the chosen action.
MCTS is a planning algorithm that uses the world model (or a simulator) to search for the best action by building a search tree through simulated rollouts.
Starting from the root (current game state), traverse the tree by selecting child nodes using UCB (the same exploration-exploitation formula from bandits):
From the new leaf, simulate a random game to completion (or use a neural network to estimate the value). This gives an estimate of how good this position is.
Update the visit counts and value estimates for every node on the path from root to leaf, propagating the simulation result back up the tree.
After many iterations (typically 1,000-100,000), the root's most-visited child is chosen as the action. MCTS converges to the minimax-optimal action given enough iterations.
In 2016, AlphaGo defeated Lee Sedol, the world Go champion. Go has ~10^170 possible positions (compared to ~10^47 for chess), making brute-force search impossibly complex. AlphaGo combined MCTS with deep neural networks:
Policy network: Trained on millions of human games to predict likely next moves. Used in MCTS selection to focus the search on promising moves (instead of trying all legal moves).
Value network: Trained to predict the winner from any board position. Used in MCTS evaluation to replace random rollouts with accurate position assessment.
MuZero matched AlphaZero's performance on Go, Chess, and Shogi while ALSO mastering Atari games -- all without being given the rules of any game. The model learns whatever internal representation is useful for planning, not a pixel-level simulation.
Dreamer (Hafner et al., 2020-2023) takes world models in a different direction: instead of MCTS-based planning, it trains a policy entirely inside the learned model's "imagination":
Learn a world model from real environment data (RSSM: Recurrent State-Space Model)
Dream: Generate long imagined trajectories using the world model
Train actor-critic on imagined trajectories (cheap -- no real environment needed)
Act: Use the learned policy in the real environment to collect more data
Repeat
Dreamer achieves strong performance on continuous control tasks and Atari with far fewer real environment interactions than model-free methods.
What Do You Think?
What is the biggest risk of training a policy purely on imagined trajectories from a learned world model?
The policy can learn to "exploit" inaccuracies in the world model -- finding trajectories that look high-reward in the model but are unrealistic. This is similar to reward hacking but worse because the "environment" itself is wrong. Solutions include: short planning horizons (less time for model errors to accumulate), model ensembles (disagree on unrealistic trajectories), and periodic reality-checking with real environment data.
Model-based RL learns a world model to plan ahead. By predicting what will happen before taking actions, model-based methods can be dramatically more sample-efficient than model-free approaches
MCTS combines tree search with neural network evaluation. Monte Carlo Tree Search explores promising sequences of actions by simulating future states, and AlphaGo/AlphaZero showed this can master complex games at superhuman levels
MuZero learns to plan without knowing the rules. Unlike AlphaGo which used known game rules, MuZero learns its own internal model of the environment, making it applicable to any domain
World models enable imagination-based training. Architectures like Dreamer train policies entirely within a learned world model, generating thousands of imagined trajectories without interacting with the real environment
Interactive Lab
Step through Monte Carlo Tree Search one iteration at a time — see selection (UCB1), expansion, simulation, and backup grow a search tree from a single root node and watch visit counts crystallize into the agent's move.
What is the primary advantage of model-based RL over model-free RL?
Model-based RL shows us the power of learning a world model and planning ahead. But the most impactful application of RL in recent years is not about games or robots -- it is about aligning language models with human values. Next up: the bridge from RL to RLHF, where reinforcement learning meets large language models to create ChatGPT.