Tesla's FSD, Waymo's planner, every Vision-Language-Action robot from Physical Intelligence and Google DeepMind — they're mostly trained by imitation, not by reward. When exploration is dangerous (driving), expensive (surgery), or slow (real robots), cloning the experts gets you 99% of the way there. The hard part isn't the algorithm; it's the covariate shift trap — which is why DAgger and GAIL were invented.
Learning Objectives
After this lesson, you will be able to:
Train a policy from expert demonstrations using Behavioral Cloning — supervised learning on (state, action) pairs — and recognize the covariate-shift trap
Apply DAgger to fix BC's compounding errors by aggregating expert corrections during agent rollouts, and explain why one extra training round per iteration restores performance
Recover a reward function from expert behavior with Inverse RL (Maximum Entropy IRL, GAIL, AIRL), and decide when reward inference beats reward design
Pick imitation vs RL based on whether you have abundant expert data, dangerous exploration, or the goal of exceeding the expert
Don't worry if "imitation learning" sounds like cheating after the elegance of policy gradients — it is the most-used technique in production RL today, and the math is much friendlier than computing advantage estimates.
Ross 2011's Dataset Aggregation (DAgger) algorithm fixes covariate shift with a deceptively simple trick: roll out the current policy, then ask the expert what they would have done at each visited state, and add those (state, expert action) pairs back to the training set.
DAgger's drawback is the obvious one: you need an interactive expert during training. For a human teleoperator, this is expensive — every iteration requires hours of new annotations. For a scripted/MPC expert, it's free; for a foundation-model-as-expert (e.g., asking GPT-4 what to do), it's cheap.
A different philosophy: instead of mimicking actions, infer why the expert acted that way. Recover a reward function r(s,a) such that the expert is (approximately) optimal under r. Then run standard RL on the inferred reward.
The classic formulation is Maximum Entropy IRL (Ziebart 2008): the expert is assumed to act stochastically with probability proportional to exp(reward of trajectory):
The chicken-and-egg cost is real: every reward update requires solving an RL problem. For tabular MDPs this is fine; for continuous-control deep RL, it's a research problem. Modern variants amortize this with replay buffers and approximate planning.
Generative Adversarial Imitation Learning (Ho & Ermon 2016) skips the explicit reward and trains a GAN-like discriminator to distinguish expert from agent state-action visits. The agent's reward becomes "fool the discriminator":
AIRL (Fu 2017) extends GAIL by structuring the discriminator so that the recovered reward is invariant to the policy distribution — a cleaner reward to transfer to new dynamics.
What Do You Think?
You have 1,000 expert demonstration trajectories from a teleoperator. The teleoperator is no longer available. You want to train a robotic-manipulation policy. Which approach?
The answer is (c) GAIL (or BC if data covers the visited distribution well). DAgger is great but requires an interactive expert — when the teleoperator is gone, you can't query them. GAIL only needs the static demonstration set, plus environment interaction. BC also works without the expert, but distribution shift will bite on long horizons.
Tests · Verify the BC policy trains successfully (loss decreases). Verify mean evaluation return is reasonable (typically 200-500 for a well-fitted CartPole BC policy). The DAgger extension should improve robustness on perturbed initial states.
Behavioral Cloning is supervised learning on (state, action) pairs. Cheap, simple, and works when the test distribution matches the expert's; fails predictably under covariate shift due to compounding errors O(T²ε)
DAgger fixes BC's covariate shift by aggregating expert relabels of policy-visited states. Improves the error bound to O(Tε), but requires an interactive expert during training
Inverse RL recovers the expert's reward function instead of mimicking actions. More interpretable and transferable than BC, but each iteration solves a forward-RL problem, making it expensive at scale
GAIL trains an adversarial discriminator to distinguish expert vs agent visitations. The agent maximizes "fool the discriminator" as a reward; converges in distribution to the expert without explicit reward design
Production AI runs on imitation + RL fine-tune. Tesla FSD, RT-2/Pi-0 robotics, LLM SFT+RLHF, o1 reasoning RL — every modern frontier system bootstraps from demonstrations and adds RL polish
A pure Behavioral Cloning policy has perfect test accuracy on held-out expert data, yet crashes within 30 steps in the real environment. What is happening?
Imitation learning shows you the easiest first step in real-world RL: skip the reward fight, learn from demonstrations. Next up: Exploration — the hardest open problem in RL, and what you'll need when no expert exists.