Search + Learning: The Idea Behind AlphaGo, AlphaZero, MuZero
This is the recipe that beat Lee Sedol, crushed Stockfish in 4 hours of training, and reached IMO silver-medal level on Olympiad math problems in 2024. MCTS guided by a neural network — the network gives the search a smart prior, the search gives the network a stronger target. They bootstrap each other into superhuman play. And in 2025 the same recipe shows up in o1, o3, and DeepSeek-R1's reasoning. Search + learning may be the most general AI idea we have.
Scope of this lesson — the IDEA. This page teaches MCTS (the general algorithm) and tells the AlphaGo → AlphaZero → MuZero story at a conceptual level: why search and learning compose, where each step in the progression removed an assumption, and how the recipe escapes board games. For the math — the PUCT derivation, the term-by-term AlphaZero loss, the MuZero K-step loss, and a full AlphaZero (not just MCTS) tic-tac-toe trainer — read the companion lesson AlphaZero & MuZero: Search + Learning in One Loop after this one.
Learning Objectives
After this lesson, you will be able to:
- Walk through MCTS step by step — selection (UCB1), expansion, simulation, backup — and understand why it became the planning algorithm of choice for combinatorial games
- Trace the AlphaGo → AlphaGo Zero → AlphaZero → MuZero progression: each step removes more human knowledge while keeping the same core recipe — deep network + search + self-play
- See how MCTS acts as a policy improvement operator at training time: the search produces a stronger policy than the network alone, and the network learns to imitate that stronger policy
- Recognize the recipe in modern systems beyond board games — AlphaProof's IMO-medal performance and o1/o3 reasoning models are the same idea applied to math and language
Prerequisites