Multi-Agent RL

AlphaGo beat Lee Sedol via self-play. Pluribus beat the top human poker pros via CFR. AlphaStar reached StarCraft Grandmaster via a league of agents. Cicero negotiated alliances in Diplomacy at human level. Every superhuman game-playing AI is multi-agent, and the moment you have two learners in the same room, the convergence proofs you grew up with stop working. This is where RL gets strange and beautiful.

Learning Objectives

After this lesson, you will be able to:

Distinguish cooperative, competitive, and mixed multi-agent settings, and pick the right algorithm family for each
Use self-play to train two-player zero-sum agents that converge to Nash equilibria — the recipe behind AlphaGo, AlphaZero, AlphaStar
Apply Counterfactual Regret Minimization (CFR) to imperfect-information games like poker — the math that powers Pluribus and Cepheus
Pick between centralized-critic (MADDPG / QMIX / MAPPO) and independent learners, and know why naive PPO often surprisingly works

Prerequisites

Before You Start

Review

Why

Why This Matters

How Pluribus beat the world's top poker pros — Brown & Sandholm 2019 used CFR-based search + neural net evaluation to win 6-player No-Limit Hold'em against five human pros; the first time AI beat humans at a game with imperfect information AND more than two players
How AlphaStar reached StarCraft II Grandmaster — DeepMind's 2019 Nature paper combined imitation learning bootstrap + population-based self-play league + asynchronous actor-critic; the first AI to reach top 0.2% in a real-time strategy game
How CICERO played Diplomacy at human level with natural-language negotiation — Meta 2022 combined an RL agent for moves with an LLM for negotiation, achieving more than double the average human score; a sneak preview of multi-agent LLMs
Build this → Train two PPO agents to play rock-paper-scissors against each other via self-play; watch convergence to the 1/3-1/3-1/3 Nash equilibrium; then try a simple cooperative grid-world where two agents must coordinate to reach a goal

Don't worry if "multi-agent" sounds intimidating — once you have one PPO agent working, multi-agent is mostly variations on "let multiple PPO agents play each other and see what falls out". The hard parts are the math of equilibria and the tricks for stability.

System	Year	Domain	Algorithm
TD-Gammon	1995	Backgammon	Self-play TD-λ
Cepheus	2015	Heads-up Limit Hold'em	CFR+
AlphaGo	2016	Go	Sup. + RL self-play + MCTS
AlphaZero	2018	Chess/Shogi/Go	Pure self-play + MCTS
Pluribus	2019	6-player No-Limit Hold'em	MCCFR + depth-limited search
AlphaStar	2019	StarCraft II	Population-based + IL bootstrap
OpenAI Five	2019	Dota 2	Self-play PPO at massive scale
CICERO	2022	Diplomacy	RL + LLM negotiation

Multi-Agent RL

After this lesson, you will be able to:

Before You Start

Your Reflection

Continue Learning

Discussion

#Three Multi-Agent Settings

#The Non-Stationarity Problem

#Self-Play: The Workhorse for Two-Player Zero-Sum

The recipe

#CFR: Imperfect-Information Games

#Independent vs Centralized Critic

#Real-World Multi-Agent Wins

#Hands-On

#Key Takeaways

#Quick Check

Deep Dive