From MDPs and Q-learning to AlphaGo/MuZero, multi-agent self-play, offline RL, hierarchical RL/options, and RLHF/GRPO for LLMs. 18 lessons covering classical foundations, deep RL, search-and-learning, and the 2024-2026 frontier (DeepSeek-R1, AlphaProof, AlphaStar).
Agent, environment, reward, MDPs, Bellman equations. Learning by trial and error.
Exact-solution methods when the MDP is fully known. Policy evaluation, policy improvement, value iteration, generalized policy iteration.
First model-free methods. MC vs TD bias-variance tradeoff, n-step returns, eligibility traces, on-policy vs off-policy.
Tabular control. SARSA (on-policy) vs Q-learning (off-policy). Build a table of state-action values; watch an agent learn a maze.
Replace Q-table with neural network. Vanilla DQN, Double DQN, Dueling, Prioritized Replay, Rainbow.
REINFORCE with baselines. Learn the policy directly via gradient ascent on expected return.
Combine policy and value learning. A2C, A3C, GAE, IMPALA — the lineage to PPO.
TRPO and PPO. The workhorse of modern deep RL — stable, efficient, widely deployed in production.
Maximum entropy RL for robotics and locomotion. Twin critics, Gaussian policies, off-policy efficiency.
Learn from expert demos. BC, DAgger (covariate shift fix), GAIL (adversarial), AIRL, Max-Ent IRL.
ε-greedy, UCB, Thompson sampling, intrinsic motivation, curiosity-driven exploration (RND, ICM).
Learn a world model. Plan in imagination. Dyna, Dreamer, world models, MuZero teaser.
MCTS + neural networks. The most spectacular successes of RL — game playing from Go to AlphaProof.
Self-play, CFR, MADDPG, QMIX, MAPPO. AlphaStar, Pluribus, CICERO — multi-agent coordination and competition.
Train from logged data. CQL, IQL, BCQ, Decision Transformer. Tesla FSD's recipe.
From game-playing to language model alignment. PPO/GRPO/DPO + outcome-based RL with verifiable rewards (o1 style).
*Why is this called "semi-Markov"?*
*AlphaZero = train a policy network to imitate MCTS; use MCTS to play by querying the policy network.
10 interactive labs — hands-on exercises for this track
Train an agent to navigate a maze — it learns from rewards and penalties
Peek inside the agent's brain — a table showing how valuable each action is in each state
5 slot machines, hidden rewards, limited pulls — find the best arm without wasting too many pulls on bad ones.
Two agents solve the same 5×5 grid side-by-side — one with explicit arrows (policy), one with a value landscape. Inject a bad arrow and watch the value agent still succeed.
Drag ε and flip advantage sign to see PPO's three curves: unclipped, clipped step function, and PPO = min(unclipped, clipped).
Run a 5-state random walk with three estimators — MC, TD(0), and TD(λ). Drag λ from 0 (TD) to 1 (MC) to see the bias-variance dial.
Actor + Critic neural networks learning together. Step through TD error flowing back into both networks on a 3-arm contextual bandit.
Two Q-learning agents on the same 8×8 grid. Sparse reward (+1 only at goal) vs dense reward (distance-based). Dense wins — but beware reward hacking.
Tic-tac-toe MCTS with the full 4-phase loop: Selection (UCB1) → Expansion → Simulation (rollout) → Backpropagation.
Drag γ from 0 to 0.999 on a 1D chain with a cliff. Watch the optimal policy flip from myopic (γ≈0) to far-sighted (γ≈0.99).
450 questions across 18 modules — check how well you understood this track.