PPO is the most-used RL algorithm in 2026. It trains GPT-class models via RLHF, dominates robotic locomotion, won at Dota 2 and StarCraft, and is the default baseline in every RL benchmark. Why? It's almost always good enough, almost never breaks. Stability beats genius — and if you learn one RL algorithm deeply, make it this one.
Learning Objectives
After this lesson, you will be able to:
Understand why unconstrained policy gradient updates are catastrophically dangerous: data collected under the old policy becomes invalid if the new policy diverges too far, causing the entire learning signal to be misleading
Follow the PPO clipped surrogate objective L^CLIP(θ) = E[min(r_t A_t, clip(r_t, 1-ε, 1+ε) A_t)] and explain mechanically why the min-of-two-terms structure prevents the policy from moving beyond the trust region
Understand why multiple gradient epochs per data batch are safe in PPO (but not in vanilla policy gradients) and how the entropy bonus prevents premature policy collapse
Know why PPO dominates in practice -- its combination of on-policy simplicity, robustness to hyperparameters, and suitability for RLHF makes it the default choice for LLM alignment
Policy gradient methods from the previous lesson have a fundamental instability: the step size is hard to control. A single bad gradient update can push the policy far from the region where it collected data, causing performance to collapse catastrophically.
Why is this a problem technically? The policy gradient theorem gives us the direction to update, but not how far to go. If the learning rate is too high, the policy changes dramatically. The new policy visits completely different states, encounters different rewards, and the gradient computed from old data becomes misleading. This can cascade into catastrophic collapse.
The insight behind TRPO and PPO is the concept of a trust region: only update the policy within a region where the old data is still trustworthy. Formally, limit how much the policy distribution can change between updates.
How do we measure how much the policy changed? Using the probability ratio:
rt(θ)=πθold(At∣St)πθ(At∣St)
TRPO constrains the KL divergence between old and new policies to be below a threshold. PPO achieves a similar effect much more simply: it clips the probability ratio.
A graph showing the PPO objective as a function of the probability ratio r: a straight line within the clipping range [0.8, 1.2] but flat outside it, preventing the gradient from pushing the policy ratio beyond the trust region.
PPO clips the update to prevent too-large changes
Let us unpack how the clipping works in different scenarios:
The gradient wants to increase the probability of this action (increase r). Without clipping, r could shoot up to 10x or 100x. With clipping at 1+epsilon (e.g., 1.2), the objective is flat beyond r=1.2. There is no gradient incentive to push r further. The policy change is capped.
The gradient wants to decrease the probability (decrease r). Without clipping, r could drop to near 0, essentially eliminating the action. With clipping at 1-epsilon (e.g., 0.8), the objective is flat below r=0.8. The policy cannot reduce this action's probability by more than 20% in one update.
Try it: PPO's Clipped Policy UpdatesInteractive
Observe how PPO constrains policy updates compared to vanilla policy gradients. The clipping mechanism prevents the policy distribution from shifting too far in a single update. Toggle the visualization to compare unconstrained updates (which can be catastrophically large) with PPO's conservative, clipped updates.
Loading visualization...
What Do You Think?
If PPO's epsilon is set to 0.0 (no clipping range), what happens?
Try it! Think of PPO's clipping like a volume knob with a max setting. Without the cap, a really good experience might make you crank the volume to 100 -- but that blows out the speakers (destroys the policy). PPO's clip says "the knob only goes to 1.2x." You can turn it up, but not so far that it breaks. Try imagining what happens with no cap at all (vanilla PG) vs. a tight cap (PPO with small epsilon).
With epsilon = 0, the clipped range is [1.0, 1.0], meaning r is always clipped to exactly 1.0. The clipped objective becomes a constant, and the minimum with the unclipped objective means no useful gradient. The policy freezes. In practice, epsilon = 0.1 to 0.3 works well, with 0.2 being the standard default.
So far we have described PPO's clipped objective and argued informally that it keeps the policy close to where it collected data. To make this rigorous, we need to derive PPO from its theoretical parent -- TRPO -- and show exactly where the min-of-two-terms structure comes from and why it implements a trust region.
The starting point is a classical result by Kakade and Langford (2002), tightened by Schulman et al. (2015) for TRPO. For any two policies π_old and π_new, the expected return of the new policy is bounded below by a tractable surrogate plus a divergence penalty:
The proof sketch: imagine sampling trajectories under π_new and tracking, at each step, the probability they could have been sampled from π_old. As long as the two policies agree, the rewards are interchangeable; once they disagree, an error term accumulates that is bounded by the per-state KL divergence. The full coupling argument is in Schulman et al. (2015) §4 and Kakade & Langford (2002) Lemma 6.1.
The crucial consequence: if we maximize the right-hand side, we are guaranteed to improve J -- monotonic improvement. This is the holy grail of policy optimization. The catch: the bound is only tight when π_new stays close to π_old, because the surrogate L uses the old state distribution d^π_old while the true return uses the new state distribution d^π_new.
#Step 2: TRPO -- maximize the surrogate subject to a hard KL constraint
TRPO converts the bound into a constrained optimization problem. Rather than subtract the KL penalty, it imposes it as a hard limit:
The Lagrangian of this constrained problem yields a natural-gradient step: descend in the direction F^{-1} g, where g is the surrogate gradient and F is the Fisher information matrix (the local quadratic approximation of KL). A conjugate-gradient solver computes F^{-1} g without materializing F, then a backtracking line search shrinks the step until the constraint holds. It works -- but it is complex, expensive, and brittle. Hence the demand for something simpler.
PPO replaces the KL constraint with a much cheaper proxy: clip the importance ratio directly. Define r_t(θ) = π_θ(a_t | s_t) / π_old(a_t | s_t), then optimize:
#Step 4: why min-of-two is a valid lower bound on L
The case analysis is short but worth doing carefully. There are four regions defined by the sign of A_t and whether r_t lies inside or outside [1-ε, 1+ε].
Case A: A_t > 0 (the action was good; gradient pushes r_t upward).
If r_t ≤ 1+ε: the clip is inactive, surr_clipped = r_t · A_t = surr_unclipped, and the min is just r_t · A_t. Normal gradient flows.
If r_t > 1+ε: the clip activates, surr_clipped = (1+ε) · A_t < r_t · A_t = surr_unclipped. The min selects the clipped version (1+ε) · A_t, which is flat in r_t. Gradient w.r.t. θ is zero -- the policy gains nothing by pushing r_t higher.
Case B: A_t < 0 (the action was bad; gradient pushes r_t downward).
If r_t ≥ 1-ε: the clip is inactive, the min picks r_t · A_t. Normal gradient flows toward smaller r_t.
If r_t < 1-ε: the clip activates, surr_clipped = (1-ε) · A_t. Now A_t < 0 so the clipped value (1-ε) · A_t > r_t · A_t (less negative). The min selects the unclippedr_t · A_t (the more negative one). The gradient flows freely -- there is no cap on pushing r_t further down.
This last sub-case is the famous asymmetry: PPO's clipping is one-sided per advantage sign. The min chooses pessimism, which sometimes means the unclipped term and sometimes the clipped term. The result is a lower bound on the true surrogate L over the trusted region r_t ∈ [1-ε, 1+ε], with the bound becoming tight at r_t = 1.
What Do You Think?
What happens during a PPO update step when r_t > 1+ε AND A_t < 0 simultaneously? (The action was bad, but the new policy has somehow become more likely to take it -- e.g. an earlier mini-batch overshot.)
#Step 5: PPO as a first-order approximation of TRPO's trust region
Why does ratio clipping approximate a KL constraint? The link is a Taylor expansion of KL near r = 1. For any two distributions p, q with ratio r = q/p:
DKL[p∥q]=Ep[−logr]=Ep[21(r−1)2+O((r−1)3)]
So if PPO holds |r_t - 1| ≤ ε (which clipping encourages, though does not strictly enforce -- gradients can still push r_t outside in the unclipped regions), then the average KL is bounded by roughly ε² / 2. For ε = 0.2 this gives a KL budget of 0.02 per step, comparable to TRPO's typical δ = 0.01. PPO is a first-order, sample-based, gradient-friendly approximation of TRPO's second-order trust region.
#Step 6: the KL-penalty variant (and why RLHF uses it)
The original PPO paper also proposed a second variant that subtracts a KL penalty rather than clipping the ratio:
The two extremes are instructive: as β → 0 the penalty vanishes and we recover pure importance-sampled policy gradient (no trust region, dangerous); as β → ∞ the policy cannot move at all and we approach a true trust-region method. Modern RLHF for large language models almost always uses the KL-penalty variant, not the clipped variant -- with β (the "KL coefficient") typically in the range 0.01-0.1. The penalty anchors the RL-tuned policy to a frozen SFT reference, preventing the model from "gaming" the reward model by producing distributions wildly different from natural text.
#Step 7: the empirical caveat -- implementation matters
A landmark paper by Engstrom et al. (2020) -- "Implementation Matters in Deep Policy Gradients" -- delivered an uncomfortable finding: most of PPO's empirical success is not due to the clipped objective.
This does not mean PPO's clipped surrogate is wrong -- the TRPO-PPO derivation above is genuinely correct, and the clip provides essential protection against ratio blow-up. It means that deployed PPO is the clip plus a careful engineering recipe, and any implementation that omits the recipe will underperform. When debugging a "PPO is unstable" problem in your own code, start by auditing those nine choices before suspecting the math.
#PPO implementation tricks: the difference between paper and production
The Engstrom result deserves its own enumeration because in 2026 it is the difference between PPO that trains and PPO that silently underperforms. The list below is the canonical "PPO implementation details" checklist drawn from Engstrom et al. (2020), Huang et al. (2022) — "The 37 Implementation Details of PPO" — and Costa et al.'s CleanRL reference implementation. If you fork a PPO from a paper-faithful source, walk through this list before debugging your math.
Orthogonal weight initialization (orthogonal_init_): initialize hidden layers with orthogonal matrices scaled by sqrt(2), and the policy output layer scaled by 0.01 (much smaller) — keeps the initial policy near-uniform, preventing premature action collapse.
Advantage normalization per mini-batch: subtract mean and divide by std of advantages within each mini-batch (not across the whole batch); this stabilizes the policy-gradient scale across heterogeneous environments.
Value function loss clipping: apply the same (1 - ε, 1 + ε) clip idea to the value loss — V_new is clipped relative to V_old + ε — so a bad value-target sample can't blow up the critic in one update.
Gradient clipping: clip the global gradient norm to 0.5 (sometimes 1.0) after backprop; a single outlier batch can otherwise produce a parameter step that destroys the policy.
Reward scaling / reward clipping: divide rewards by a running standard deviation and clip to [-10, 10] (especially for Atari, where rewards have huge ranges); makes the value-function regression target numerically stable.
The slogan (Engstrom 2020): "Code-level tricks dwarf algorithmic improvements." On the standard MuJoCo suite, applying the 10 tricks above to a vanilla policy-gradient implementation (no clipping, no trust region) recovers most of the gap to "PPO" — and PPO's clipping on top of these tricks adds the final 10-15%, not the headline 100%. The clipping is necessary, but it is not the algorithm; the algorithm is clip + 10 tricks.
For a canonical, single-file modern implementation, study CleanRL's ppo.py (Costa et al. 2022) — it implements all ten tricks plus correct logging in ~200 lines of PyTorch, audited against OpenAI's reference numbers. The pedagogical companion is Huang's "37 Implementation Details" blog post, which expands the list above with deeper environment-specific tricks (frame-stacking, action-space-specific gradient handling, reward-clipping variants per game).
Quick check
You're porting PPO to a new framework. Ablation studies show your implementation is 30% below CleanRL's reported scores on MuJoCo. Which trick is most likely the single biggest contributor to the gap?
Run the current policy across M parallel environments for N steps each. At each step, record the state, action, reward, value estimate V(s), and the log-probability of the action under the current policy. This creates a batch of M x N transitions.
The log-probabilities are saved because PPO needs them later to compute the probability ratio between the old and updated policies. These are the "old policy" probabilities.
Using Generalized Advantage Estimation (GAE), compute the advantage at each time step. The advantage A_t measures how much better the actual outcome was compared to what the value function predicted.
GAE with lambda=0.95 provides a smooth balance: it uses the TD error (one-step advantage) exponentially weighted over multiple steps. Positive advantage means "this action was better than expected." Negative means "worse than expected."
For each state-action pair in the batch, compute the probability ratio: the new policy's probability of the action divided by the old policy's probability (saved from Step 1). If r = 1, the policies agree. If r = 1.5, the new policy is 50% more likely to take this action.
This ratio is computed as exp(new_log_prob - old_log_prob) for numerical stability. It is the key quantity that PPO monitors and controls.
PPO won the RL algorithm popularity contest for several compelling reasons:
Simplicity. TRPO requires computing the Fisher information matrix and solving a constrained optimization problem. PPO achieves similar results with a simple clipped objective that can be optimized with standard gradient descent. The entire implementation fits in ~100 lines of PyTorch.
Robustness. PPO is remarkably insensitive to hyperparameters compared to other RL algorithms. The defaults (epsilon=0.2, GAE lambda=0.95, 3-10 optimization epochs) work well across a wide range of problems. This is invaluable in practice -- hyperparameter tuning is the biggest time sink in RL.
Sample efficiency. By performing multiple gradient steps on each batch of data, PPO extracts more learning signal per environment interaction than one-step methods like A2C. This matters when environment interactions are expensive (robotics, complex simulations).
Scalability. PPO parallelizes naturally across many environments. OpenAI trained their Dota 2 agent using PPO on thousands of concurrent game instances. The algorithm's on-policy nature means you just need more parallel environments for more data.
You need to train an RL agent with a continuous action space (e.g., robot joint angles). Which algorithm should you NOT use?
DQN requires computing max over Q-values for all actions, which is only possible with a finite, discrete action set. For continuous actions, you need policy gradient methods (PPO, SAC, TD3) that directly parameterize the action distribution.
Unconstrained policy updates can be catastrophically large. A single bad gradient step can push the policy far from the data distribution, causing performance to collapse and never recover
PPO clips the objective to create a trust region. By limiting the policy ratio to the range [1-epsilon, 1+epsilon], PPO prevents destructive updates while still allowing meaningful learning steps
PPO balances simplicity and performance. Unlike TRPO which requires expensive second-order optimization, PPO achieves similar trust region behavior with a simple clipped objective that is easy to implement and tune
PPO is the algorithm behind RLHF for LLM alignment. Its stability and reliability made it the standard choice for fine-tuning language models with human feedback, powering ChatGPT, Claude, and other aligned models
Interactive Lab
Drag the probability ratio across the clipping boundary and watch the objective flatten — the single most important diagram for understanding why PPO is the workhorse of modern RL.
PPO gives us stable, efficient policy optimization. But so far, all our RL algorithms learn purely from interaction -- they have no internal model of how the world works. Next up: Model-Based RL, where the agent learns a world model and uses it to plan ahead, as AlphaGo did to conquer the game of Go.
the min is what lets PPO recover from overshoots; without it, PPO would have one-way ratchets in the wrong direction.
This is not hypothetical -- in early PPO implementations the bug existed in the wild, and training curves silently diverged on hard environments. Always use torch.min(surr1, surr2), never just surr_clipped.
Learning rate annealing: linearly decay the Adam LR from its initial value to 0 over training; without this PPO drifts in late training and final-policy quality degrades.
Generalized Advantage Estimation (GAE) with λ ≈ 0.95: not λ = 1.0 (Monte Carlo) and not λ = 0 (TD); the 0.95 sweet spot is empirically near-universal across MuJoCo, Atari, and procgen.
Mini-batch shuffling: shuffle the indices into the rollout buffer every epoch, not just once at the start; without per-epoch shuffling the policy overfits to the order of mini-batches within an epoch.
Entropy bonus on the policy: add +0.01 * H[π] to the loss (so the optimizer maximizes entropy); without this the policy collapses to deterministic too quickly and exploration dies.
Number of epochs per rollout: K = 4 (sometimes up to 10); Engstrom's ablation showed this is the most sensitive hyperparameter — K = 1 is too few (vanilla policy gradient), K = 20 overshoots the trust region despite clipping.
Clamp the ratio to the range [1-epsilon, 1+epsilon], typically [0.8, 1.2]. This prevents the policy from changing too drastically in any single update. If the ratio wants to go to 3.0 (triple the probability), clipping caps it at 1.2 (20% increase).
The clipping creates a "trust region" -- the new policy must stay close to the old policy that collected the data. Beyond the clip boundary, there is no gradient incentive to push further.
Compute both the unclipped objective (r * A) and the clipped objective (clip(r) * A). Take the minimum of the two. This creates a pessimistic lower bound: for good actions, the objective cannot improve beyond the clipped region. For bad actions, the penalty cannot be relaxed beyond the clipped region.
The min operation is what makes PPO conservative -- it prevents the policy from "exploiting" the advantage estimate by moving too aggressively.
Unlike REINFORCE (which uses each batch once), PPO performs K epochs (typically 3-10) of gradient updates on the same batch data. The clipping mechanism makes this safe -- even after multiple passes, the policy cannot move far from where the data was collected.
Each epoch samples random mini-batches from the full batch, computing the clipped loss and updating via Adam optimizer. This extracts maximum learning signal from each batch of environment interactions.
Simultaneously with the policy, update the value network (critic) to minimize the squared error between its predictions V(s) and the actual returns. A well-trained value function produces better advantage estimates, which produce better policy updates.
The value loss is added to the total objective with a coefficient (typically 0.5). After updates complete, the old batch is discarded, fresh trajectories are collected with the new policy, and the cycle repeats from Step 1.