Track 07 · Reinforcement Learning · 13 min
The AI that learns by playing.
Atari, AlphaGo, AlphaStar, AlphaProof, ChatGPT's RLHF training, DeepSeek-R1's reasoning — the same framework powers all of them. Five interactive demos cover the algorithms that taught machines to win at games, fly drones, and align with human preferences.
“Reinforcement learning is the closest thing we have to a theory of intelligence.”
#The hook
This is the field manual for the algorithm family that taught machines to win.
#Why this matters in 2026 — the receipts
RL by the numbers
From games to language models
100%
Frontier LLMs trained with RLHF or successors
Open/closed AI labs
4-1
AlphaGo's defeat of Lee Sedol
March 2016
10M+
Games AlphaStar self-played
DeepMind 2019
30+
RL papers per day on arXiv
arXiv 2025
#The four core ideas
The framework
State → action → reward → policy
1. The Markov Decision Process
the formalismAgent in state S takes action A, receives reward R, transitions to new state S'.
- Every RL problem reduces to: define state, action space, reward function, transition dynamics.
- Hard part is usually defining state and reward — not the algorithm.
- Real-world MDPs are 'partially observable' — you see only part of the true state.
2. Value functions and Q-learning
1989Estimate 'how good is each state?' or 'how good is each action in this state?'
- Q-learning: learn Q(s, a) = expected total reward if you take action a in state s.
- Bellman equation lets you bootstrap learning from future estimates.
- Deep Q-Network (DQN, 2013) used a CNN to learn Atari pixel-to-Q mappings.
3. Policy gradients
1992 (REINFORCE)Directly optimize the policy — the function that maps state to action.
- REINFORCE → A2C → PPO → GRPO. Same family.
- PPO (Proximal Policy Optimization) is the workhorse — stable, simple, dominant.
- Used by OpenAI Five, AlphaStar, and (with critic) RLHF for LLMs.
4. RLHF / preference learning
2017 → ChatGPTUse human comparisons (A vs B) instead of explicit reward functions.
- Train a 'reward model' on human preference pairs.
- Then RL the LLM against that reward model — usually with PPO.
- DPO (Direct Preference Optimization) eliminates the explicit reward model. Now standard.
4-1
AlphaGo's victory over Lee Sedol — March 2016
The week that ended a 2,500-year-old game's resistance to machine play. AlphaGo combined deep policy/value networks with Monte Carlo tree search, trained from millions of human games and then refined via self-play. Within a year, AlphaGo Zero learned the entire game from scratch — no human data — and beat the original 100-0.
DeepMind, March 2016
Vocabulary
Six RL terms you'll see in every paper
Concept
Reward
Scalar feedback the agent tries to maximize.
Like: A score that tells you 'good' or 'bad'.
e.g. +100 for goal, -1 per step
Concept
Policy
The agent's strategy — a function from state to action.
Like: The decision rule a chess grandmaster has internalized.
e.g. PPO trains policies directly
Concept
Value
Expected total future reward from a state.
Like: Google Maps' ETA — expected outcome from here.
e.g. V(s) for state value, Q(s,a) for action value
Concept
Exploration
Try unfamiliar actions to learn what works.
Like: Trying a new restaurant instead of your usual.
e.g. epsilon-greedy, UCB, Thompson sampling
Concept
RLHF
RL from human feedback. Aligns LLMs with preferences.
Like: A child shaped by parental approval.
e.g. OpenAI used PPO; Anthropic uses DPO/CAI
Concept
GRPO
Group Relative Policy Optimization. DeepSeek's PPO simplification.
Like: Grading on a curve within each batch.
e.g. Used to train DeepSeek-R1
#Idea 1 — Q-learning, animated
The classic RL setup: an agent navigating a grid world. Each cell is a state, four actions per state. The agent learns Q-values — the expected future reward from each (state, action) pair.
After enough episodes, the Q-table converges to optimal:
#Idea 2 — The explore-exploit dilemma
#Idea 3 — Policy gradients, the LLM-training engine
The "advantage" trick — the most important variance reduction in policy gradients — subtracts a baseline from the reward to reduce noise:
#What's been built with RL
RL in production
What this framework has actually shipped
Game-playing
AlphaGo / AlphaZero
4-1
Defeated Lee Sedol
Self-play + Monte Carlo tree search + deep value/policy nets. Ended a 2,500-year-old game.
Self-play RL
StarCraft II
AlphaStar
99%
vs human players
Beat top StarCraft pros at the GrandMaster level. Massively more complex than Go (real-time, partial observability).
Multi-agent RL
LLM alignment
ChatGPT (RLHF)
1B+
Users impacted
RLHF turned the bare GPT-3.5 into something usable. The single most important applied RL system in history.
RLHF / PPO
Reasoning RL
DeepSeek R1
90%+
MATH benchmark
Open-weight reasoning model trained with GRPO. Showed reasoning RL can be replicated outside frontier labs.
GRPO
Math olympiad
AlphaProof
4/6
IMO problems solved
Silver-medal performance at IMO 2024. RL on synthetic theorem proofs.
RL + search
Robotics control
Boston Dynamics Spot
0
Falls / hour (typically)
Locomotion controllers trained with massive simulated RL, then transferred to hardware.
Sim-to-real RL
#The 2026 frontier
#Where to go next
- Reinforcement Learning track — 16 lessons: MDPs, Q-learning, policy gradients, AlphaGo, RLHF, GRPO.
- Deep Learning — function approximation prerequisites.
- NLP & Transformers — RLHF requires you to know how the LLM works.
- AI Agents — the natural application of RL is building agents.
#Key takeaways
Key Takeaways
- RL framework: agent in state S takes action A, gets reward R, transitions to S'.
- Q-learning estimates 'how good is this (state, action)?' Bootstrap from future estimates.
- Policy gradients (REINFORCE → PPO → GRPO) optimize the policy directly. The LLM-training engine.
- RLHF aligned ChatGPT and every modern LLM. Train a reward model from human preferences, then PPO/DPO.
- Explore-vs-exploit is fundamental — you can't just always pick the best-so-far action.
- 2026 frontier: reasoning RL, Constitutional AI, GRPO open-weight reasoning, offline RL.
#References & further reading
- Sutton & Barto — Reinforcement Learning: An Introduction (free PDF). Field bible.
- Mnih et al. — Playing Atari with Deep Reinforcement Learning (NIPS 2013). DQN's birth.
- Silver et al. — Mastering the Game of Go without Human Knowledge (Nature 2017). AlphaGo Zero.
- Ouyang et al. — Training Language Models to Follow Instructions with Human Feedback (NeurIPS 2022). The InstructGPT/RLHF paper.
- DeepSeek — DeepSeek-R1: Reasoning via Reinforcement Learning (2025). The open-weight reasoning playbook.
- OpenAI Spinning Up (spinningup.openai.com). Free, practical, code-first RL course.