RLHF is RL with humans as the reward function. ChatGPT was the moment the world realized RL had broken into the mainstream — and it works because chess-playing optimization methods turned out to be perfect for fine-tuning LLMs. The agent is the LLM. The action is the next token. The reward is a human preference. Same math, applied to the most important problem in AI: making models do what we actually want.
Learning Objectives
After this lesson, you will be able to:
Map every RL concept onto LLM alignment: the LLM is the agent/policy, the prompt is the state, each generated token is an action, and the reward model provides the scalar reward at the end of each response
Walk through all three stages of the full RLHF pipeline: supervised fine-tuning (SFT) on demonstrations, reward model training on Bradley-Terry pairwise comparisons, and PPO optimization with a KL divergence penalty against the SFT reference model
Understand reward hacking in RLHF -- why optimizing a proxy reward model diverges from true human preferences (Goodhart's Law) -- and the mitigations: KL penalty, conservative optimization, and reward model ensembles
Know the newer alternatives to full RLHF: DPO eliminates the separate reward model; GRPO eliminates the value network; Constitutional AI uses the model itself to critique responses -- and understand what each gives up vs standard RLHF
This is the bridge lesson -- the one that connects everything you have learned about reinforcement learning to the technology behind ChatGPT, Claude, and every aligned language model.
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
A pretrained language model (like GPT-3 base) is an extraordinary next-token predictor. But next-token prediction is not the same as being helpful:
It mimics training data, including toxic, biased, and false content
It does not know what humans want -- it predicts the most likely continuation, which might be anything
It can be inconsistent -- the same question may get contradictory answers depending on prompt framing
It does not refuse harmful requests -- it just generates the most probable text
The gap between "predicts text well" and "is genuinely helpful and harmless" is the alignment gap. RLHF bridges it.
Try it! Ask ChatGPT or Claude something slightly sensitive, like "How do I pick a lock?" Notice how it gives a thoughtful, nuanced response instead of just dumping instructions? That nuance -- knowing when to help, when to refuse, when to add caveats -- is RLHF in action. The raw model would have just predicted the most likely internet text.
Start with a large language model pretrained on internet-scale text data. This model is an extraordinary next-token predictor -- it has learned grammar, facts, reasoning patterns, and coding ability from trillions of tokens. But it is unaligned: it will happily generate toxic content, follow harmful instructions, or produce confident misinformation.
The pretrained model is the raw material. It has capability but no alignment. It does not know what humans want -- it only knows what text is likely.
#Step 2: Supervised Fine-Tuning (SFT) on Demonstrations
Fine-tune the pretrained model on thousands of high-quality (prompt, response) pairs written by human demonstrators. The model learns the format of helpful responses: how to follow instructions, structure answers, and maintain a helpful tone.
SFT is standard supervised learning -- maximize the probability of the human-written tokens. This creates the SFT model, which can follow instructions but is limited to imitating the demonstrations. It cannot generalize beyond what was explicitly shown.
Generate multiple responses to the same prompt using the SFT model. Show pairs of responses to human labelers and ask: "Which response is better?" Labelers choose based on helpfulness, accuracy, safety, and overall quality.
This creates a dataset of preference pairs: (prompt, chosen_response, rejected_response). Tens of thousands of these comparisons capture nuanced human values that would be impossible to encode as explicit rules.
Train a neural network (typically the same architecture as the LLM, with a scalar output head) on the preference data. The reward model learns to assign higher scores to responses that humans preferred. It uses the Bradley-Terry pairwise ranking loss: maximize the probability that the preferred response scores higher.
The reward model becomes an automated proxy for human judgment -- it can evaluate millions of responses at machine speed, enabling the RL training loop.
Apply PPO to fine-tune the SFT model to maximize the reward model's score, with a KL divergence penalty that prevents the policy from straying too far from the SFT model. The LLM is the agent, each generated token is an action, and the reward comes from the reward model at the end of the response.
The KL penalty is the safety net: without it, the model would exploit reward model weaknesses, generating gibberish that scores high. With it, the model stays grounded while shifting toward responses humans prefer.
After PPO training, the model consistently produces responses that align with human preferences: helpful, accurate, safe, and well-structured. It has internalized the patterns of "good" responses beyond what the original demonstrations covered.
The aligned model is deployed to users. Ongoing monitoring, red-teaming, and iterative RLHF cycles continue to improve alignment. The pipeline is not a one-time process -- it is a continuous loop of collecting feedback, training reward models, and refining the policy.
Loading visualization...
Figure
Four training stages build on one another. First, pretraining on internet-scale text produces raw capability. Second, supervised fine-tuning on curated demonstrations teaches the model to follow instructions. Third, a reward model is trained on human comparisons between candidate responses. Fourth, PPO fine-tuning optimises against that reward model, with a KL penalty anchoring the policy to the fine-tuned starting point so it improves without drifting into degenerate text.
The RLHF pipeline: pretrain → SFT → reward model → PPO
Start with the pretrained base model and fine-tune it on a dataset of high-quality (prompt, response) pairs written by human demonstrators. This gives the model a starting point: it learns the format of helpful responses, how to follow instructions, and basic quality standards.
LSFT=−t=1∑TlogPθ(yt∣x,y<t)
SFT alone improves the model significantly, but it has limits: the model can only be as good as the demonstrations, and writing perfect demonstrations for every possible query is prohibitively expensive. What if the model could learn from preferences instead of demonstrations?
Collect comparison data: show human labelers two (or more) model responses to the same prompt and ask them to rank which is better. Train a reward model to predict these human preferences.
LRM=−E(x,yw,yl)[logσ(Rϕ(x,yw)−Rϕ(x,yl))]
The reward model takes a (prompt, response) pair and outputs a scalar score. Higher score = better response according to human preferences. This model becomes the automated judge that replaces human raters during PPO training.
What Do You Think?
Why train a reward model instead of just using human raters directly during PPO training?
PPO requires evaluating hundreds of thousands to millions of responses during training. Human labelers can rate perhaps hundreds per day. The reward model is a scalable proxy for human judgment -- it is not perfect, but it can evaluate responses instantly, enabling the RL training loop to run at machine speed.
Now apply PPO (from Lesson 6) to fine-tune the language model to maximize the reward model's score, with a crucial constraint: do not stray too far from the SFT model.
The KL divergence penalty is critical. Without it, the model would learn to exploit weaknesses in the reward model -- generating gibberish that happens to score high, or repeating phrases the reward model likes regardless of the question. The KL penalty says: "maximize reward, but stay close to a known-good model."
Try it: RL Fine-Tuning the PolicyInteractive
Visualize how PPO shifts the language model's policy distribution during RLHF. The policy starts close to the SFT model (reference), and PPO gradually adjusts action probabilities to maximize the reward model score. Watch how the KL constraint keeps the policy from drifting too far -- this is the same mechanism that prevents ChatGPT from generating degenerate outputs.
This is Goodhart's Law applied to AI: "When a measure becomes a target, it ceases to be a good measure." The stronger you optimize the reward model, the more you exploit its imperfections rather than improve actual quality.
DPO (Rafailov et al., 2023) eliminates the reward model entirely. It directly optimizes the language model on preference data, showing that the RLHF objective can be reparameterized into a supervised loss:
DPO is simpler (no reward model, no PPO), more stable (supervised loss), and often achieves comparable results. It has become the default for many alignment applications.
Deriving DPO from the RLHF objective
The DPO loss above isn't a heuristic — it's the exact solution to the RLHF objective, recast as supervised learning. The derivation is short but illuminating, and it explains why DPO works without ever instantiating a reward model. The argument follows Rafailov et al. (2023), building on the closed-form analysis of KL-constrained policies in Korbak et al. (2022) and the original RLHF framing in Christiano et al. (2017).
Step 1: The RLHF objective. The PPO-RLHF objective from earlier in this lesson, written cleanly, is:
Step 2: Closed-form optimal policy. This isn't an objective you have to solve numerically — it has an analytical solution. Treat the optimization as a per-prompt problem (since the objective decomposes across prompts) and use a Lagrange multiplier λ(x) to enforce that π(·|x) is a probability distribution (sums to 1 over y). The Lagrangian, with the KL expanded as E_π[log(π/π_ref)], is:
Setting ∂L / ∂π(y|x) = 0 yields r(x, y) - β log(π(y|x)/π_ref(y|x)) - β - λ(x) = 0, which rearranges to log(π(y|x)/π_ref(y|x)) = r(x, y)/β - 1 - λ(x)/β. Exponentiating and absorbing the y-independent constants into a normalizer Z(x) (chosen so probabilities sum to 1) gives the closed-form optimal policy:
This is the same Boltzmann-policy form that shows up in maximum-entropy RL (Soft Actor-Critic) — KL-constrained reward maximization always produces an exponentially-tilted reference distribution. The problem with using this formula directly: Z(x) is a sum over all possible responsesy, which for a language model is the entire space of token sequences. Intractable.
Step 3: Invert to get reward in terms of policy. The key DPO move. Take logarithms of the closed-form solution and solve for r(x, y):
r(x,y)=βlogπref(y∣x)π∗(y∣x)+βlogZ(x)
The first term — β log(π(y|x)/π_ref(y|x)) — is just a log-ratio of two probabilities the policy can compute on the fly. The second term, β log Z(x), is still intractable. But it depends only on the prompt x, not on the response y. That's the lever DPO pulls next.
Step 4: Substitute into Bradley-Terry — and log Z(x) cancels. The reward model is trained on preference pairs (y_w, y_l) (winner, loser) under the Bradley-Terry preference model:
P(yw≻yl∣x)=σ(r(x,yw)−r(x,yl))
Substitute the implicit-reward expression from Step 3 into the Bradley-Terry difference r(x, y_w) - r(x, y_l):
This is the algebraic magic of DPO: the intractable normalizer Z(x) cancels because both responses in a preference pair share the same prompt.
Step 5: The DPO loss. Plug the cancellation into Bradley-Terry, then take the negative log-likelihood over the preference dataset, parameterizing the policy as π_θ:
This is the same expression we wrote down in the DPO section above — but now you've derived it from first principles. Every term has a meaning: π_θ/π_ref is the implicit reward (up to scaling by β), β is the inverse temperature controlling how aggressively the policy deviates from π_ref, and σ plus -log is the standard Bradley-Terry NLL. The reward model never had to exist; it's been re-parameterized into the policy itself.
What β controls. Smaller β → larger allowed divergence from π_ref → more aggressive optimization, but more risk of exploiting any flaw in the preference dataset (the DPO analog of reward hacking). Larger β → policy stays close to π_ref → safer, but slower preference learning. This is exactly the role β played in the PPO-RLHF KL penalty — same hyperparameter, same trade-off, just expressed through the algebra differently. In practice, β for DPO is usually in the range 0.1–0.5.
What Do You Think?
In the DPO loss, what happens as β → 0?
Why DPO works. DPO solves the RLHF problem implicitly via supervised learning on the closed-form optimal policy. There's no reward model to train, no PPO rollouts to generate, no value head to fit — drastically simpler engineering, fewer hyperparameters, and a stable supervised loss curve instead of the notoriously finicky PPO training dynamics. Under the Bradley-Terry preference assumption, DPO and full RLHF have the same global optimum: the closed-form Boltzmann-weighted policy from Step 2.
The trade-off. DPO is offline and off-policy by construction — it trains on a fixed preference dataset and never samples from the current policy during training. This means: (a) you can't easily do online exploration the way PPO can (you'd have to repeatedly collect new preferences and re-train, which is iterative DPO), (b) the policy is biased toward the distribution of y_w and y_l in the training data — if those responses are very far from what the policy would naturally produce, the loss surface gets weird, and (c) multi-step credit assignment (per-token credit for long responses) is harder than in PPO, which has a value function for exactly this purpose. DPO can also degrade when the preference data is noisy or when y_w and y_l are too similar (the Bradley-Terry log-likelihood bound becomes loose); these failure modes motivated the entire DPO family — IPO, KTO, ORPO, SimPO, CPO — covered in the next section. Each one patches a specific weakness in vanilla DPO: IPO fixes over-optimization on near-deterministic preferences, KTO removes the pairwise data requirement, ORPO and SimPO eliminate the frozen reference model, CPO drops the explicit KL term and replaces it with an SFT regularizer. None of them change the core idea — they all train directly on preference data using a closed-form parameterization of the policy.
GRPO (DeepSeek, 2024) simplifies PPO by eliminating the value network entirely. Instead of estimating advantages with a learned critic, it generates multiple responses to the same prompt and uses the relative reward statistics within the group as the advantage. This subsection derives GRPO from first principles, shows why it's a valid baseline, and traces its connection to RLOO and the DeepSeek-R1 reasoning-RL recipe.
The PPO problem with reasoning
Standard PPO-RLHF needs a critic — a value network V(s) that predicts the expected return from state s. In conversational alignment that's already painful (the reward signal is the whole response, so V has to integrate over a giant action space). For reasoning — math proofs, code that has to compile, multi-step logic — the critic is much harder still. The "value" of a partially-completed math derivation is not a smooth function of the tokens written so far; it's effectively binary (this proof will work / this proof will fail), with a huge variance regime in between. Training a critic to estimate this reliably eats compute, requires its own hyperparameters, and is a leading source of PPO training instabilities in reasoning RLHF. GRPO's question: can we kill the critic entirely?
GRPO's trick: the group itself replaces V(s)
For each prompt x, sample G completions y_1, ..., y_G from the current policy. Score each with the reward model (or a rule-based reward): r_1, ..., r_G. Now use the within-group mean and standard deviation as a state-dependent baseline:
The group statistics replace both the critic and the per-token advantage estimator. There is no value head, no GAE, no lambda to tune. The only new hyperparameter is G (group size), typically 4-64 in production runs.
Why this is a valid baseline
A core result from the policy-gradients lesson: subtracting any function b(s) of the state from the advantage does not bias the policy gradient. Formally,
∇θJ(θ)=Es,a∼πθ[∇θlogπθ(a∣s)(Q(s,a)−b(s))]
In GRPO, the "state" is the prompt x. The baseline b(x) = mean_i(r_i) is the empirical mean of rewards over the G samples drawn for this prompt — a noisy estimator of the trueE_{y ~ π}[r(x, y)], which is exactly V^π(x) for the prompt-conditioned MDP. The group mean is a Monte Carlo estimate of the critic's target. So GRPO is provably unbiased: the group mean is a valid baseline because (a) it depends only on the prompt, not on the specific completion within the group, and (b) it converges to the true V^π(x) as G → ∞.
Variance reduction via std normalization
The mean-subtraction handles the bias. The standard deviation normalization handles a second problem: prompts vary wildly in difficulty, and therefore in reward scale. An easy prompt might give rewards in [0.8, 1.0] (all completions are decent); a hard prompt might give [-3.0, +5.0] (some completions are great, some are terrible). Without normalization, gradient updates from the hard prompt dominate gradient updates from the easy prompt purely because the magnitudes differ. Dividing by σ_r makes the advantage scale-invariant across prompts — both prompts contribute comparable-magnitude advantages, and the optimizer treats them equally. This is the same trick advantage-normalization-per-mini-batch performs in standard PPO, just applied at the group level instead of the batch level.
Var[A^iGRPO]=Var[σrri−r]≈1(for all prompts, by construction)
Connection to RLOO (Leave-One-Out)
GRPO is not the first algorithm to use within-group rewards as a critic substitute. RLOO (REINFORCE with Leave-One-Out, Cohere/Ahmadian et al. 2024) uses a closely related trick: for each completion y_i, the baseline is the mean reward of the otherG-1 completions, with no std normalization:
A^iRLOO=ri−G−11j=i∑rj
The Cohere paper reports that RLOO matches or exceeds PPO on alignment benchmarks while being simpler — same headline as GRPO. In practice the two algorithms are very close: both eliminate the critic, both use within-group statistics as the baseline, and the choice between them is a hyperparameter decision rather than a conceptual one. Empirically, GRPO's std-normalization helps more on reasoning tasks (where reward variance differs sharply across prompts of varying difficulty); RLOO's leave-one-out is cleaner on dialogue alignment (where rewards are roughly equally noisy per prompt).
DeepSeek-R1: the no-critic, no-reward-model reasoning recipe
GRPO's biggest impact has been in the DeepSeek-R1 training recipe (DeepSeek-AI 2025), which strips out every learned component except the policy itself:
No critic. GRPO's group mean replaces V(s).
No reward model. Rewards come directly from rule-based verifiers — math problems are scored by checking whether the final boxed answer equals the ground truth; code problems are scored by running the code and checking unit tests; format rewards check that the response contains <think>...</think> reasoning sections. No human preference data, no Bradley-Terry, no reward-model training run.
No PPO state-action ratios beyond the policy ratio. GRPO uses PPO's clipped surrogate min(r_t A_t, clip(r_t, 1-ε, 1+ε) A_t) over the policy ratio r_t = π_θ(y_t|x, y_{<t}) / π_{old}(y_t|x, y_{<t}), but there's no value-function clipping (no critic), no GAE, no separate value-loss coefficient.
The result is reasoning RL with only three components: the policy, a verifier, and the GRPO update. DeepSeek-R1-Zero (the variant trained from scratch with this recipe, no SFT) demonstrated that pure RL on verifiable rewards is sufficient to elicit chain-of-thought reasoning capability — the model spontaneously learns to write long <think> traces, self-correct, and verify its own work, all from the rule-based reward signal alone. This was a major surprise to the alignment field: prior to DeepSeek-R1, the conventional wisdom was that strong reasoning required either (a) lots of human-curated reasoning demonstrations or (b) a powerful preference-based reward model. GRPO + rule-based rewards skipped both.
Anthropic's approach: instead of relying solely on human raters, use the AI model itself to identify harmful outputs. The model critiques its own responses according to a set of principles (the "constitution"), revises them, and the revised responses become training data.
Replace human raters entirely with AI judges. Use a powerful model (e.g., GPT-4) to evaluate responses from a smaller model being trained. This dramatically reduces the cost of collecting preference data but introduces the risk of "distilling" the judge model's biases.
Since DPO landed in 2023, the alignment community has produced a cluster of follow-on losses that all share DPO's structural idea — train the policy directly on preference data with a closed-form objective — while patching specific failure modes. The high-level taxonomy below covers the ones you'll see in 2026 production stacks.
Method
Year
Core idea
Data format
Reference model needed?
Trade-off
DPO
2023
Reparameterize RLHF as supervised pairwise loss on (chosen, rejected)
Pairwise preferences
Yes (frozen SFT)
Simple, but overfits to preference dataset; no online exploration
IPO (Azar 2023)
2023
Replace DPO's log-sigmoid with a squared-loss "identity" preference objective
Pairwise preferences
Yes
Bounded loss prevents the over-optimization DPO suffers when preferences are deterministic
KTO (Ethayarajh 2024)
2024
Drop pairwise labels — use unpaired "this response was thumbs-up/down" labels via a prospect-theoretic Kahneman-Tversky utility
Binary thumbs-up / thumbs-down (NOT pairs)
Yes
Cheaper data collection (no need to compare pairs), but slightly weaker signal than DPO when paired data is available
ORPO (Hong 2024)
2024
Combine SFT + preference learning in one stage by adding an odds-ratio penalty to the standard SFT loss
Pairwise preferences
No (reference-free)
Now the default in TRL — eliminates the frozen reference model, halving memory; works in a single training stage instead of SFT-then-align
SimPO (Meng 2024)
2024
Reference-free DPO using length-normalized average log-probability as the implicit reward
Pairwise preferences
No
Reduces memory and length bias; often matches or beats DPO on Arena-Hard, AlpacaEval
CPO (Contrastive Preference Optimization)
2024
Drop the reference KL term and add an explicit SFT term to keep the policy grounded
Pairwise preferences
No
Faster, smaller memory footprint; first deployed for machine translation by Haoran Xu et al.
Trade-off summary. DPO is the conceptual baseline but has two practical pain points: it requires keeping a frozen reference model in memory (doubling parameter count for the loss) and it's known to over-optimize when preference labels are near-deterministic, which IPO fixed by using a bounded loss. KTO loosens the data requirement — instead of forcing labelers to compare two responses, they can just rate one at a time, which is how most production telemetry already looks (thumbs up/down on individual chat responses). ORPO and SimPO both attack the reference-model overhead from different angles: ORPO folds preference learning into the SFT stage itself, eliminating the separate alignment phase; SimPO normalizes by response length so it doesn't need the reference model as an anchor. CPO sits closest to a pragmatic "DPO without the KL term, plus an SFT regularizer" recipe and is now standard in translation and other narrow-domain fine-tuning. In 2026, ORPO is the most common default for new alignment runs in TRL because it has the lowest operational complexity (one stage, no reference model), but SimPO and KTO each win on specific benchmarks — the choice has become a hyperparameter selection problem rather than an algorithmic one. None of these methods solve the underlying issues with preference learning itself (label noise, distributional shift, reward hacking at the dataset level), but each removes one engineering friction.
What Do You Think?
What is the main advantage of DPO over the full RLHF pipeline?
DPO's main advantage is simplicity: it collapses the reward model + PPO stages into a single supervised training step. There is no reward model to train, no PPO hyperparameters to tune, no value function to estimate. The tradeoff is that DPO cannot adapt the reward signal during training (it is fixed by the preference dataset), while RLHF with PPO can dynamically explore and exploit the reward landscape.
RLHF is not just a training technique -- it is the first practical approach to the AI alignment problem. The question "how do we make powerful AI systems do what humans want?" has been debated for decades. RLHF provides a concrete (if imperfect) answer:
Specify preferences through human comparisons (not through hand-coded rules)
Learn a reward model that captures these preferences (not perfect, but scalable)
Optimize the model to satisfy learned preferences while staying grounded (KL penalty)
This is not a solved problem. Reward hacking, sycophancy, inconsistent human preferences, scalable oversight of superhuman systems -- these remain open challenges. But RLHF turned alignment from a philosophical question into an engineering one, and that is profound.
Tests · Verify the reward model assigns a higher score to 'helpful great' than to 'sorry long response'. Verify the length weight is negative (shorter is preferred).
RLHF maps RL concepts onto LLM alignment. The LLM is the agent/policy, the prompt is the state, the generated text is the action, and human preference is the reward signal
The RLHF pipeline has three stages. Supervised fine-tuning (SFT) teaches format, reward model training captures human preferences, and PPO optimization maximizes the reward while staying close to the base model via KL constraints
KL divergence constraints prevent reward hacking. Without the KL penalty, the model finds degenerate outputs that score high on the reward model but are meaningless to humans; the constraint keeps the model close to its pretrained behavior
DPO and GRPO are emerging alternatives to PPO. Direct Preference Optimization eliminates the separate reward model by optimizing preferences directly, simplifying the pipeline while achieving comparable alignment quality
In the RLHF pipeline, what does the reward model learn from?
From the agent-environment loop to RLHF, you now understand the algorithms that taught machines to play Go, walk, and converse like humans. The next two lessons -- hierarchical RL with options, and the AlphaZero / MuZero search-and-learning recipe -- close the RL track by showing the same Bellman machinery scaled up: temporally-abstract policies on one side, and a learned latent world model with MCTS on the other.