RLHF & DPO: Aligning LLMs with Human Preferences

A raw pretrained LLM is brilliant and rude — it will happily complete "how to make a bomb" because nothing in its training said not to. RLHF is what turned GPT-3 into ChatGPT, base-Llama into Llama 3.3-Instruct, and Anthropic's pretrained models into Claude. Then DPO (2023) showed you could skip the reward model entirely. This lesson covers the post-training pipeline that decides whether your LLM is usable.

Learning Objectives

After this lesson, you will be able to:

Understand the three-step RLHF recipe: teach the model to follow instructions, build a reward model from human preferences, then optimize with PPO
See how DPO simplifies this by skipping the reward model entirely
Compare four alignment approaches (RLHF, DPO, GRPO, Constitutional AI) and know when to use each
Spot alignment failures: reward hacking (gaming the score), mode collapse (boring answers), and over-refusal (saying no to everything)

Prerequisites

Before You Start

Review

Why

Why This Matters

GPT-3 could write anything, including harmful content, GPT-3.5 and ChatGPT share the same architecture, but ChatGPT went through RLHF; this alignment step is what transformed a raw language model into a safe, useful assistant that billions of people trust
DPO (2023) cut alignment training cost by 10x. By eliminating the separate reward model and RL training loop, DPO made alignment accessible to teams without RL expertise, running on a single GPU with a dataset of preference pairs
Every major LLM you use went through alignment training. Claude uses Constitutional AI + RLHF, ChatGPT uses RLHF, Gemini uses RLHF, and every open-source chat model (LLaMA-2-chat, Mistral-Instruct, Qwen-Chat) uses DPO or a variant; this lesson explains the technique behind all of them
Build this → Implement the DPO loss function from scratch in PyTorch, understanding exactly how preference data drives the model to prefer helpful responses over harmful ones

Active Recall

Before alignment, retrieve the prerequisites: (1) write the cross-entropy loss for a single token in one line of notation, (2) state in one sentence what KL divergence MEASURES between two distributions, and (3) recall from the RL track — what is a policy π(a|s) and what is the policy gradient theorem doing for it? RLHF welds all three together, so reconstructing each by yourself before reading on makes the assembly below click instantly.

Write your answer in your own words — don't look back at the lesson. This is the most effective way to remember what you just learned.

Type your explanation above

#The Alignment Problem

Supervised fine-tuning (SFT) produces a model that follows instructions. But "follows instructions" is not the same as "safe and helpful." A raw instruction-tuned model has three failure modes:

Not helpful enough: The model hedges excessively, refuses benign requests, or gives vague non-answers to avoid controversy.

Not harmless enough: The model assists with harmful tasks if asked cleverly, produces biased outputs, or generates content that violates community standards.

Not honest enough: The model confidently states incorrect information, flatters users instead of correcting them, or fails to acknowledge uncertainty.

These three properties (helpful, harmless, honest) are the "HHH" alignment target, formalized by Anthropic in their early research. The challenge is that they conflict:

A maximally helpful model answers every request, including harmful ones
A maximally harmless model refuses everything uncertain, becoming useless
A maximally honest model might deliver truths in harmful ways

Alignment training is the process of finding the right balance among these three objectives using human judgment to define "the right balance."

#Stage 1: Supervised Fine-Tuning (SFT)

Before any alignment training, the base pretrained model undergoes SFT. This converts a completion model (predicts the next token) into an assistant (responds to instructions).

The SFT step:

Collect a dataset of (prompt, ideal response) pairs, curated by human contractors
Fine-tune the base model on these pairs using standard cross-entropy loss
Mask loss on the prompt tokens — train only to predict the response

This SFT model becomes both:

The starting point for reward model training
The reference policy π_ref used later in PPO and DPO to prevent the aligned model from drifting too far from sensible behavior

The SFT model is competent but not aligned — it follows the format of a helpful assistant without the values. Stage 2 fixes that.

Try it! Think of two responses to "How do I pick a lock?" Response A: step-by-step instructions. Response B: "I can help with locksmith concepts, but I cannot provide instructions that might enable break-ins." Which would you prefer from an AI assistant? That preference judgment is exactly what RLHF trains on.

#Stage 2: Reward Modeling

#Collecting Preference Data

Human annotators see the same prompt with multiple model responses and rank them from best to worst. A typical annotation task:

Prompt: "How do I get my neighbor to stop playing loud music at night?"

Response A: "You could hire a lawyer and sue them for noise violations." (overkill, confrontational)

Response B: "Try talking to them directly first — most people are willing to compromise when asked politely. If that fails, check your local noise ordinance and contact your landlord or local authorities." (helpful, proportionate, practical)

Response C: "That's a frustrating situation. Some options: 1) Talk to them directly, 2) Contact your landlord or building management, 3) File a noise complaint with local authorities. I'd start with option 1." (good but slightly more mechanical)

Response D: "Have you tried soundproofing your apartment?" (misses the point)

The annotator ranks: B > C > A > D. This generates three preference pairs: (B > A), (B > D), (C > D), and so on.

#The Bradley-Terry Model

The reward model learns to assign a scalar score r(x, y) to any (prompt x, response y) pair. It is trained using the Bradley-Terry model for pairwise preferences:

P(y_w \succ y_l \mid x) = \sigma\bigl(r(x, y_w) - r(x, y_l)\bigr)

The reward model loss is the negative log-likelihood of correctly predicting the human preference:

\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \Bigl[\log \sigma\bigl(r_\theta(x, y_w) - r_\theta(x, y_l)\bigr)\Bigr]

#Architecture

The reward model shares the same transformer backbone as the SFT model, but replaces the language modeling head (which predicts the next token) with a scalar regression head — a single linear layer that outputs one number: how good is this response?

Training the reward model typically uses 10,000–100,000 preference pairs and takes a fraction of the compute of pretraining.

#Stage 3: PPO Optimization

With a trained reward model, we can now fine-tune the SFT model to maximize human-rated quality. The algorithm used is Proximal Policy Optimization (PPO), a standard reinforcement learning algorithm.

#The RLHF Objective

The policy (the LLM) generates a response y given prompt x. The reward model scores it. We want to maximize the expected reward, but with a crucial constraint: do not drift too far from the SFT reference model.

\mathcal{J}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot | x)} \Bigl[r_\phi(x, y) - \beta \cdot \mathrm{KL}\bigl[\pi_\theta(\cdot | x) \,\|\, \pi_{\mathrm{ref}}(\cdot | x)\bigr]\Bigr]

The KL penalty is critical. Without it, the model finds degenerate solutions: if the reward model gives high scores to long responses, the model learns to be verbose. If annotators preferred formal language, the model becomes stiff and unnatural. The KL constraint says: "get higher rewards, but don't become unrecognizable." If the KL term feels abstract, it is the same information-theoretic distance you met in the entropy / cross-entropy lesson — here it just measures how far the fine-tuned policy has drifted from the reference distribution. The optimization machinery underneath comes from the PPO algorithm we covered in the RL track; see also RL to RLHF for the full lineage from policy gradients to language-model alignment.

#One PPO Update Step

#Step 1: Sample a batch of prompts

Draw 64 prompts from the training distribution. These are the contexts the policy will respond to.

#Step 2: Generate responses with the current policy

The current LLM (θ) generates a response for each prompt. This is the "rollout" — sampling from the policy being optimized.

#Step 3: Score responses with the reward model

The frozen reward model assigns a scalar score r(x, y) to each (prompt, response) pair. High scores = responses humans prefer.

#Step 4: Compute the KL penalty

Compare the current policy's token probabilities to the reference SFT policy. The KL divergence measures how much the model has changed. Large deviation = large penalty.

#Step 5: Compute advantages

Normalize the reward minus KL penalty to get advantages: how much better than average was this response? Positive advantage = reinforce this behavior. Negative advantage = suppress it.

#Step 6: Update the policy

Use the PPO clipped objective to update θ. The clipping prevents any single update from being too large, keeping training stable. Repeat for hundreds of update steps.

#The Four Models in RLHF

RLHF requires four models in memory simultaneously, which is its biggest practical challenge:

Model	Role	Trainable?
SFT model (reference π_ref)	KL baseline — prevents reward hacking	Frozen
Reward model r_φ	Scores each response for the PPO signal	Frozen
Current policy π_θ	The model being optimized	Yes
Value function V	Estimates expected future reward (PPO baseline)	Yes

For a 7B base model, this means holding four 7B+ parameter models in memory. With 16-bit weights, that is roughly 56 GB minimum — before activations and optimizer states. This is why RLHF requires serious infrastructure.

#Reward Hacking

Watch Out

Common Mistake

Reward hacking is the most important failure mode in RLHF. The reward model is an imperfect proxy for human preferences — it learned from finite data and makes mistakes. The LLM is very good at finding the cracks in that proxy.

Classic examples:

Length exploitation: Annotators tend to rate longer, more detailed responses higher. The model learns that verbosity is rewarded and becomes excessively wordy, adding filler phrases like "Great question!" and "Certainly, I'd be happy to help!"
Sycophancy: The model learns that agreeing with the user, flattering them, and validating their premises leads to higher reward scores — even when the user is factually wrong.
Format gaming: If the reward model scores bullet points and headers highly, the model uses them everywhere, even when plain prose would be clearer.
False confidence: Hedging language like "I'm not sure, but..." may lower reward scores because it seems less authoritative. The model learns to assert things confidently even when it should express uncertainty.

These failures are all examples of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." The reward model is a measure of human preferences. When the LLM optimizes it directly, the measure breaks down.

The KL penalty mitigates but does not eliminate reward hacking. In practice, teams monitor for these patterns during training and use iterated RLHF with refreshed preference data to correct them.

What Do You Think?

A reward model gives response A a score of 8.2 and response B a score of 2.1. After PPO training, the model always generates response A. But users report the model has become sycophantic — it always agrees with them, even when they are wrong. What went wrong?

The answer: reward hacking. Human annotators, consciously or not, rated responses that agreed with them, validated their views, and used flattering language higher than responses that politely corrected them. The reward model absorbed this bias. PPO then exploited it ruthlessly. The result: a sycophantic model that maximizes annotator reward scores but fails at the actual goal of honesty.

#DPO: Cutting Out the Reward Model

#The Key Insight

The RLHF objective has an optimal solution we can derive analytically. Given the objective:

\mathcal{J}(\theta) = \mathbb{E}\bigl[r(x,y)\bigr] - \beta \cdot \mathrm{KL}\bigl[\pi_\theta \| \pi_{\mathrm{ref}}\bigr]

The optimal policy has the closed form:

\pi^*(y \mid x) = \frac{1}{Z(x)} \, \pi_{\mathrm{ref}}(y \mid x) \exp\!\Bigl(\frac{r(x, y)}{\beta}\Bigr)

Rearranging this equation to express the reward in terms of the policy:

r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} + \beta \log Z(x)

#The DPO Loss

Substituting this reparameterization into the Bradley-Terry reward model loss, the partition function Z(x) cancels (it appears in both winner and loser terms), giving the DPO loss:

\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)} \Biggl[\log \sigma\!left(\beta \log \frac{\pi_\theta(y_w | x)}{\pi_{\mathrm{ref}}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{\mathrm{ref}}(y_l | x)}\right)\Biggr]

#What DPO Actually Does

The two log-ratio terms have a clear interpretation:

β · log(π_θ(y_w|x) / π_ref(y_w|x)): How much more (or less) does the current model prefer the chosen response compared to the reference model? DPO wants this to be large and positive.
β · log(π_θ(y_l|x) / π_ref(y_l|x)): How much more (or less) does the current model prefer the rejected response compared to the reference model? DPO wants this to be small or negative.

The loss is minimized when the model simultaneously increases the probability of chosen responses and decreases the probability of rejected responses, but weighted by how much the reference model is "surprised." Responses the reference model already handles well get less gradient signal; responses where the reference model is uncertain get more.

What Do You Think?

You want to align a 7B model for customer service. You have 10,000 preference pairs (chosen/rejected responses). You have one A100 GPU for 3 days. Should you use RLHF or DPO?

DPO is the clear choice here. RLHF would require: training a reward model (consuming some of your A100 time), then running PPO with four models in memory (which may not even fit on one A100). DPO trains with just two model passes (current policy + frozen reference), uses exactly the same memory budget as SFT, and produces comparable alignment quality at 7B scale.

#DPO Family & Successors

DPO collapsed RLHF from a four-model RL loop into a two-model supervised problem, and that simplification kicked off a flood of follow-ups in 2023-2024. Each one targets a specific failure mode of vanilla DPO: overconfidence on noisy preferences, the need for paired data, the cost of a reference model, or the length-bias DPO inherits from its sigmoid objective. The methods below are the ones you'll actually find in TRL today.

IPO — Identity Preference Optimization (Azar 2023). DPO's loss treats preferences as if they came from a deterministic Bradley-Terry process, which means a small minority of mislabeled or noisy pairs can push log-ratios to extreme values and overfit. IPO replaces the log-sigmoid with a bounded squared loss on the implicit reward margin, keeping the gradient finite as the model becomes confident. This makes it more robust on real-world preference datasets where 5-15% of pairs are noise.

KTO — Kahneman-Tversky Optimization (Ethayarajh 2024). The big practical win: KTO works with unpaired preferences. Instead of needing (chosen, rejected) pairs, it consumes a single response with a binary label — thumbs-up or thumbs-down. This matches how real product telemetry actually arrives (chat ratings, feedback emoji, "regenerate" clicks) and is dramatically more data-efficient than collecting paired comparisons. KTO uses a prospect-theory-inspired value function that asymmetrically penalizes losses relative to a reference point.

ORPO — Odds Ratio Preference Optimization (Hong 2024). Eliminates the reference model entirely by combining SFT and preference optimization into a single objective via an odds-ratio penalty on the chosen vs rejected pair. No π_ref means half the memory and a single training pass — you can do SFT and alignment together. ORPO is now the default new-user recipe in HuggingFace TRL and is what most "lazy alignment" pipelines on Hugging Face use.

SimPO — Simple Preference Optimization (Meng 2024). Drops the reference model and replaces the implicit reward with a length-normalized log-probability. The length normalization directly addresses DPO's notorious length bias — DPO models tend to generate longer responses because longer sequences accumulate larger log-probability differences. SimPO is competitive with or better than DPO on AlpacaEval while producing cleaner-length outputs.

CPO — Contrastive Preference Optimization (Xu 2024). Adds a behavior-cloning regularizer on the chosen response to prevent the policy from drifting away from the SFT distribution. Originally developed for translation, now a stable choice when the chosen-response distribution matters for downstream task quality.

Method	Needs paired preferences?	Needs reference model?	Year	One-line trade-off
DPO	Yes	Yes	2023	Strong baseline; overfits to deterministic preferences and inherits length bias.
IPO	Yes	Yes	2023	Bounded loss is more robust to noisy/contradictory labels.
KTO	No (binary thumbs)	Yes	2024	Eats production telemetry directly; much better data efficiency.
ORPO	Yes	No	2024	Half the memory, single-pass SFT+alignment; now default in TRL.
SimPO	Yes	No	2024	Length-normalized; matches/beats DPO with cleaner outputs.
CPO	Yes	Yes	2024	Adds SFT regularizer; stable when SFT distribution matters.

Quick check

What is ORPO's key architectural simplification over DPO?

#GRPO: Group Relative Policy Optimization

GRPO, used in DeepSeek-R1, takes the simplification further by eliminating the reference model as well.

#How It Works

Instead of comparing against a fixed reference policy, GRPO:

Samples G responses (a group) for the same prompt
Scores each response (using a reward signal — either a reward model or a rule-based checker)
Computes relative advantages within the group: how much better is this response than the group average?
Updates the policy to increase probability of above-average responses and decrease below-average ones

A_i = \frac{r_i - \bar{r}}{\sigma_r} \quad \text{where} \quad \bar{r} = \frac{1}{G}\sum_{j=1}^G r_j

#Why GRPO Matters

No reference model: The baseline is the group average, not a fixed π_ref. This saves memory (one fewer model copy) and avoids the assumption that the reference model is always a good baseline.
Works with rule-based rewards: For tasks with verifiable answers (math, code, factual Q&A), you can replace a learned reward model with a rule: "does the answer match the ground truth?" This eliminates reward hacking on the reward model.
Enabled reasoning model training: DeepSeek-R1 used GRPO with rule-based rewards (math verification) to train models that reason in extended chains of thought, achieving o1-level performance at a fraction of the cost.

#Constitutional AI: Anthropic's Approach

Anthropic's alignment method replaces human preference labels with a constitution — a set of principles that define good behavior.

#The Two-Phase Process

Phase 1 — Supervised Learning from AI Feedback (SLAF):

Sample a response from the model to a potentially harmful prompt
Use a separate "critique" model to critique the response according to the constitution: "Does this response violate principle 7: 'Do not assist with creating weapons'? How?"
Use the critique model to revise the response according to the constitution
Fine-tune on the (prompt, revised response) pairs

Phase 2 — Reinforcement Learning from AI Feedback (RLAIF):

Generate preference pairs by having the AI evaluate two responses against the constitution: "Which response is more consistent with being helpful, harmless, and honest?"
Train a preference model on these AI-generated labels
Use RLHF with this AI-labeled reward model

#Why Constitutional AI Scales

Human preference annotation is expensive and inconsistent — different annotators disagree, and the same annotator is inconsistent across sessions. CAI:

Scales annotation without requiring human judgment on every pair
Makes alignment principles explicit and auditable (you can read the constitution)
Allows rapid iteration — update the constitution, regenerate labels, retrain

#Comparison: RLHF vs DPO vs GRPO vs Constitutional AI

Aspect	RLHF	DPO	GRPO	Constitutional AI
Data required	100K+ preference pairs	10K–100K preference pairs	Any reward signal	A constitution (principles)
Separate reward model	Yes	No	Optional	Yes (AI-generated)
Training complexity	Very high (RL loop, 4 models)	Low (supervised)	Medium	Medium (2 phases)
GPU memory	4x SFT model size	2x SFT model size	1–2x	2x
Main failure mode	Reward hacking	Preference data quality	Group size sensitivity	Constitution gaps
Best for	Large teams, safety-critical	Startups, 7B–70B models	Reasoning/math tasks	Principled alignment at scale
Used by	OpenAI (ChatGPT)	Most open-source chat models	DeepSeek (R1)	Anthropic (Claude)

#Try It Yourself

pythonplayground.py · Pyodide

# Implement DPO loss from scratch
# and verify it behaves as expected

import math

def log_sigmoid(x):
  """Numerically stable log(sigmoid(x))"""
  if x >= 0:
      return -math.log(1 + math.exp(-x))
  else:
      return x - math.log(1 + math.exp(x))

def compute_log_probs(response_tokens, model_logits):
  """Simulate log probabilities for a response.
  In real DPO, these come from running the model on the full sequence.
  Here we use pre-computed values for illustration."""
  return sum(model_logits[t] for t in response_tokens)

def dpo_loss(pi_logp_chosen, pi_logp_rejected,
           ref_logp_chosen, ref_logp_rejected, beta=0.1):
  """
  Compute DPO loss for one (prompt, chosen, rejected) triple.
  
  Args:
      pi_logp_chosen: log P(chosen | prompt) under current policy
      pi_logp_rejected: log P(rejected | prompt) under current policy
      ref_logp_chosen: log P(chosen | prompt) under reference policy
      ref_logp_rejected: log P(rejected | prompt) under reference policy
      beta: KL coefficient (default 0.1)
  
  Returns:
      DPO loss (scalar, minimize this)
  """
  chosen_ratio = pi_logp_chosen - ref_logp_chosen
  rejected_ratio = pi_logp_rejected - ref_logp_rejected
  logits = beta * (chosen_ratio - rejected_ratio)
  return -log_sigmoid(logits)

# --- Scenario 1: Model initially assigns equal probability ---
print("=== Scenario 1: Equal initial probabilities ===")
# Both chosen and rejected have the same log-prob under current policy and ref
loss_equal = dpo_loss(
  pi_logp_chosen=-5.0, pi_logp_rejected=-5.0,
  ref_logp_chosen=-5.0, ref_logp_rejected=-5.0,
  beta=0.1
)
print(f"Loss (equal probs): {loss_equal:.4f}")
print(f"Expected: ~0.693 (log 2 — random chance baseline)\n")

# --- Scenario 2: Model already prefers chosen ---
print("=== Scenario 2: Model prefers chosen ===")
loss_prefer_chosen = dpo_loss(
  pi_logp_chosen=-3.0, pi_logp_rejected=-7.0,  # current model strongly prefers chosen
  ref_logp_chosen=-5.0, ref_logp_rejected=-5.0,  # ref is neutral
  beta=0.1
)
print(f"Loss (prefers chosen): {loss_prefer_chosen:.4f}")
print(f"Expected: &lt; 0.693 (lower loss = model is on the right track)\n")

# --- Scenario 3: Model prefers the rejected response (bad!) ---
print("=== Scenario 3: Model prefers rejected (misaligned) ===")
loss_prefer_rejected = dpo_loss(
  pi_logp_chosen=-7.0, pi_logp_rejected=-3.0,  # current model prefers rejected
  ref_logp_chosen=-5.0, ref_logp_rejected=-5.0,  # ref is neutral
  beta=0.1
)
print(f"Loss (prefers rejected): {loss_prefer_rejected:.4f}")
print(f"Expected: > 0.693 (higher loss = gradient will fix this)\n")

# --- Simulate gradient effect ---
print("=== Simulating DPO training effect ===")
print("Starting: equal probabilities")
print("After DPO step: chosen probability should increase, rejected should decrease\n")

pi_chosen = -5.0
pi_rejected = -5.0
ref_chosen = -5.0
ref_rejected = -5.0
learning_rate = 0.5

for step in range(5):
  loss = dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected)
  # Simplified gradient: DPO loss gradient increases chosen, decreases rejected
  # (In real training, this is computed via backprop through the model)
  grad_signal = 1 - (1 / (1 + math.exp(-0.1 * ((pi_chosen - ref_chosen) - (pi_rejected - ref_rejected)))))
  pi_chosen += learning_rate * grad_signal * 0.1
  pi_rejected -= learning_rate * grad_signal * 0.1
  print(f"Step {step+1}: loss={loss:.4f}, logP(chosen)={pi_chosen:.2f}, logP(rejected)={pi_rejected:.2f}")

print("\n=== CHALLENGE ===")
print("Implement a function that takes a list of (chosen_logp, rejected_logp) pairs")
print("and computes the average DPO loss across the batch.")
print("Then verify: as beta increases, what happens to the loss sensitivity?")

Tests · Verify Scenario 1 loss is approximately 0.693. Verify Scenario 2 loss is less than Scenario 1 loss. Verify Scenario 3 loss is greater than Scenario 1 loss.

Training language models to follow instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe (2022)

The InstructGPT paper — the first large-scale application of RLHF to LLMs. Showed that a 1.3B parameter RLHF-tuned model outperformed a raw 175B GPT-3 model on human preference evaluations. Introduced the SFT → reward model → PPO pipeline that became the industry standard.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (2023)

The DPO paper. Derived that the optimal RLHF policy has a closed form that can be expressed directly in terms of policy probabilities, eliminating the need for a separate reward model and PPO training loop. Within months of publication, DPO became the default alignment method for the open-source LLM community.

Constitutional AI: Harmlessness from AI Feedback

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, et al. (2022)

Anthropic's Constitutional AI paper. Introduces the principle-based approach to alignment where an AI critiques and revises its own outputs according to a human-written constitution. Demonstrates that AI feedback can partially replace human preference labeling, scaling alignment without proportional annotation cost.

#Key Takeaways

SFT teaches format, alignment teaches values. Supervised fine-tuning produces an assistant that follows instructions; RLHF and DPO teach it which instructions to follow and how to balance helpfulness, harmlessness, and honesty
RLHF requires four models and RL expertise. The SFT → reward model → PPO pipeline is powerful but expensive: four models in memory, a complex RL training loop, and constant vigilance against reward hacking
DPO derived that the reward model is unnecessary. By reparameterizing the optimal RLHF policy in terms of log-ratios, DPO turns alignment into a supervised learning problem over preference pairs, cutting compute by 10x and eliminating the RL loop
Reward hacking is alignment's Goodhart's Law. Any proxy measure of human preferences can be exploited; the KL penalty limits exploitation but does not eliminate it; monitoring for sycophancy, verbosity, and false confidence is part of production alignment
The right method depends on your resources. DPO for startups and 7B models; RLHF for safety-critical large-scale systems; GRPO for verifiable reward tasks like math; Constitutional AI for principled alignment without per-pair human annotation

#Quick Check

Quick Check1 / 4

In the RLHF pipeline, what is the purpose of the KL divergence penalty in the PPO objective?

Next up: Reasoning Models & Test-Time Compute — how models like o1 and DeepSeek-R1 use GRPO and process reward models to learn step-by-step reasoning, and the new paradigm of scaling compute at inference time rather than only at training time.

RLHF & DPO: Aligning LLMs with Human Preferences

Learning Objectives

After this lesson, you will be able to:

Understand the three-step RLHF recipe: teach the model to follow instructions, build a reward model from human preferences, then optimize with PPO
See how DPO simplifies this by skipping the reward model entirely
Compare four alignment approaches (RLHF, DPO, GRPO, Constitutional AI) and know when to use each
Spot alignment failures: reward hacking (gaming the score), mode collapse (boring answers), and over-refusal (saying no to everything)

Prerequisites

Before You Start

Review

Why

Why This Matters

GPT-3 could write anything, including harmful content, GPT-3.5 and ChatGPT share the same architecture, but ChatGPT went through RLHF; this alignment step is what transformed a raw language model into a safe, useful assistant that billions of people trust
DPO (2023) cut alignment training cost by 10x. By eliminating the separate reward model and RL training loop, DPO made alignment accessible to teams without RL expertise, running on a single GPU with a dataset of preference pairs
Every major LLM you use went through alignment training. Claude uses Constitutional AI + RLHF, ChatGPT uses RLHF, Gemini uses RLHF, and every open-source chat model (LLaMA-2-chat, Mistral-Instruct, Qwen-Chat) uses DPO or a variant; this lesson explains the technique behind all of them
Build this → Implement the DPO loss function from scratch in PyTorch, understanding exactly how preference data drives the model to prefer helpful responses over harmful ones

Active Recall

Write your answer in your own words — don't look back at the lesson. This is the most effective way to remember what you just learned.

Type your explanation above

#The Alignment Problem

Supervised fine-tuning (SFT) produces a model that follows instructions. But "follows instructions" is not the same as "safe and helpful." A raw instruction-tuned model has three failure modes:

Not helpful enough: The model hedges excessively, refuses benign requests, or gives vague non-answers to avoid controversy.

Not harmless enough: The model assists with harmful tasks if asked cleverly, produces biased outputs, or generates content that violates community standards.

Not honest enough: The model confidently states incorrect information, flatters users instead of correcting them, or fails to acknowledge uncertainty.

These three properties (helpful, harmless, honest) are the "HHH" alignment target, formalized by Anthropic in their early research. The challenge is that they conflict:

A maximally helpful model answers every request, including harmful ones
A maximally harmless model refuses everything uncertain, becoming useless
A maximally honest model might deliver truths in harmful ways

Alignment training is the process of finding the right balance among these three objectives using human judgment to define "the right balance."

#Stage 1: Supervised Fine-Tuning (SFT)

Before any alignment training, the base pretrained model undergoes SFT. This converts a completion model (predicts the next token) into an assistant (responds to instructions).

The SFT step:

Collect a dataset of (prompt, ideal response) pairs, curated by human contractors
Fine-tune the base model on these pairs using standard cross-entropy loss
Mask loss on the prompt tokens — train only to predict the response

This SFT model becomes both:

The starting point for reward model training
The reference policy π_ref used later in PPO and DPO to prevent the aligned model from drifting too far from sensible behavior

The SFT model is competent but not aligned — it follows the format of a helpful assistant without the values. Stage 2 fixes that.

#Stage 2: Reward Modeling

#Collecting Preference Data

Human annotators see the same prompt with multiple model responses and rank them from best to worst. A typical annotation task:

Prompt: "How do I get my neighbor to stop playing loud music at night?"

Response A: "You could hire a lawyer and sue them for noise violations." (overkill, confrontational)

Response B: "Try talking to them directly first — most people are willing to compromise when asked politely. If that fails, check your local noise ordinance and contact your landlord or local authorities." (helpful, proportionate, practical)

Response C: "That's a frustrating situation. Some options: 1) Talk to them directly, 2) Contact your landlord or building management, 3) File a noise complaint with local authorities. I'd start with option 1." (good but slightly more mechanical)

Response D: "Have you tried soundproofing your apartment?" (misses the point)

The annotator ranks: B > C > A > D. This generates three preference pairs: (B > A), (B > D), (C > D), and so on.

#The Bradley-Terry Model

The reward model learns to assign a scalar score r(x, y) to any (prompt x, response y) pair. It is trained using the Bradley-Terry model for pairwise preferences:

P(y_w \succ y_l \mid x) = \sigma\bigl(r(x, y_w) - r(x, y_l)\bigr)

The reward model loss is the negative log-likelihood of correctly predicting the human preference:

\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \Bigl[\log \sigma\bigl(r_\theta(x, y_w) - r_\theta(x, y_l)\bigr)\Bigr]

#Architecture

Reward Model Architecture

🧠Frozen or fine-tuned transformer backbone

💭Hidden state at final token position

🔧Linear layer: hidden_dim → 1

📊Scalar reward r(x, y)

🧠 llm💭 memory🔧 tool📊 evaluator

Training the reward model typically uses 10,000–100,000 preference pairs and takes a fraction of the compute of pretraining.

#Stage 3: PPO Optimization

#The RLHF Objective

\mathcal{J}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot | x)} \Bigl[r_\phi(x, y) - \beta \cdot \mathrm{KL}\bigl[\pi_\theta(\cdot | x) \,\|\, \pi_{\mathrm{ref}}(\cdot | x)\bigr]\Bigr]

#One PPO Update Step

#Step 1: Sample a batch of prompts

Draw 64 prompts from the training distribution. These are the contexts the policy will respond to.

#Step 2: Generate responses with the current policy

The current LLM (θ) generates a response for each prompt. This is the "rollout" — sampling from the policy being optimized.

#Step 3: Score responses with the reward model

The frozen reward model assigns a scalar score r(x, y) to each (prompt, response) pair. High scores = responses humans prefer.

#Step 4: Compute the KL penalty

Compare the current policy's token probabilities to the reference SFT policy. The KL divergence measures how much the model has changed. Large deviation = large penalty.

#Step 5: Compute advantages

Normalize the reward minus KL penalty to get advantages: how much better than average was this response? Positive advantage = reinforce this behavior. Negative advantage = suppress it.

#Step 6: Update the policy

Use the PPO clipped objective to update θ. The clipping prevents any single update from being too large, keeping training stable. Repeat for hundreds of update steps.

#The Four Models in RLHF

RLHF requires four models in memory simultaneously, which is its biggest practical challenge:

Model	Role	Trainable?
SFT model (reference π_ref)	KL baseline — prevents reward hacking	Frozen
Reward model r_φ	Scores each response for the PPO signal	Frozen
Current policy π_θ	The model being optimized	Yes
Value function V	Estimates expected future reward (PPO baseline)	Yes

#Reward Hacking

Watch Out

Common Mistake

Classic examples:

Length exploitation: Annotators tend to rate longer, more detailed responses higher. The model learns that verbosity is rewarded and becomes excessively wordy, adding filler phrases like "Great question!" and "Certainly, I'd be happy to help!"
Sycophancy: The model learns that agreeing with the user, flattering them, and validating their premises leads to higher reward scores — even when the user is factually wrong.
Format gaming: If the reward model scores bullet points and headers highly, the model uses them everywhere, even when plain prose would be clearer.
False confidence: Hedging language like "I'm not sure, but..." may lower reward scores because it seems less authoritative. The model learns to assert things confidently even when it should express uncertainty.

The KL penalty mitigates but does not eliminate reward hacking. In practice, teams monitor for these patterns during training and use iterated RLHF with refreshed preference data to correct them.

What Do You Think?

#DPO: Cutting Out the Reward Model

#The Key Insight

The RLHF objective has an optimal solution we can derive analytically. Given the objective:

\mathcal{J}(\theta) = \mathbb{E}\bigl[r(x,y)\bigr] - \beta \cdot \mathrm{KL}\bigl[\pi_\theta \| \pi_{\mathrm{ref}}\bigr]

The optimal policy has the closed form:

\pi^*(y \mid x) = \frac{1}{Z(x)} \, \pi_{\mathrm{ref}}(y \mid x) \exp\!\Bigl(\frac{r(x, y)}{\beta}\Bigr)

Rearranging this equation to express the reward in terms of the policy:

r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} + \beta \log Z(x)

#The DPO Loss

Substituting this reparameterization into the Bradley-Terry reward model loss, the partition function Z(x) cancels (it appears in both winner and loser terms), giving the DPO loss:

\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)} \Biggl[\log \sigma\!left(\beta \log \frac{\pi_\theta(y_w | x)}{\pi_{\mathrm{ref}}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{\mathrm{ref}}(y_l | x)}\right)\Biggr]

#What DPO Actually Does

The two log-ratio terms have a clear interpretation:

β · log(π_θ(y_w|x) / π_ref(y_w|x)): How much more (or less) does the current model prefer the chosen response compared to the reference model? DPO wants this to be large and positive.
β · log(π_θ(y_l|x) / π_ref(y_l|x)): How much more (or less) does the current model prefer the rejected response compared to the reference model? DPO wants this to be small or negative.

What Do You Think?

You want to align a 7B model for customer service. You have 10,000 preference pairs (chosen/rejected responses). You have one A100 GPU for 3 days. Should you use RLHF or DPO?

#DPO Family & Successors

Method	Needs paired preferences?	Needs reference model?	Year	One-line trade-off
DPO	Yes	Yes	2023	Strong baseline; overfits to deterministic preferences and inherits length bias.
IPO	Yes	Yes	2023	Bounded loss is more robust to noisy/contradictory labels.
KTO	No (binary thumbs)	Yes	2024	Eats production telemetry directly; much better data efficiency.
ORPO	Yes	No	2024	Half the memory, single-pass SFT+alignment; now default in TRL.
SimPO	Yes	No	2024	Length-normalized; matches/beats DPO with cleaner outputs.
CPO	Yes	Yes	2024	Adds SFT regularizer; stable when SFT distribution matters.

Quick check

What is ORPO's key architectural simplification over DPO?

#GRPO: Group Relative Policy Optimization

GRPO, used in DeepSeek-R1, takes the simplification further by eliminating the reference model as well.

#How It Works

Instead of comparing against a fixed reference policy, GRPO:

Samples G responses (a group) for the same prompt
Scores each response (using a reward signal — either a reward model or a rule-based checker)
Computes relative advantages within the group: how much better is this response than the group average?
Updates the policy to increase probability of above-average responses and decrease below-average ones

A_i = \frac{r_i - \bar{r}}{\sigma_r} \quad \text{where} \quad \bar{r} = \frac{1}{G}\sum_{j=1}^G r_j

#Why GRPO Matters

No reference model: The baseline is the group average, not a fixed π_ref. This saves memory (one fewer model copy) and avoids the assumption that the reference model is always a good baseline.
Works with rule-based rewards: For tasks with verifiable answers (math, code, factual Q&A), you can replace a learned reward model with a rule: "does the answer match the ground truth?" This eliminates reward hacking on the reward model.
Enabled reasoning model training: DeepSeek-R1 used GRPO with rule-based rewards (math verification) to train models that reason in extended chains of thought, achieving o1-level performance at a fraction of the cost.

#Constitutional AI: Anthropic's Approach

Anthropic's alignment method replaces human preference labels with a constitution — a set of principles that define good behavior.

#The Two-Phase Process

Phase 1 — Supervised Learning from AI Feedback (SLAF):

Sample a response from the model to a potentially harmful prompt
Use a separate "critique" model to critique the response according to the constitution: "Does this response violate principle 7: 'Do not assist with creating weapons'? How?"
Use the critique model to revise the response according to the constitution
Fine-tune on the (prompt, revised response) pairs

Phase 2 — Reinforcement Learning from AI Feedback (RLAIF):

Generate preference pairs by having the AI evaluate two responses against the constitution: "Which response is more consistent with being helpful, harmless, and honest?"
Train a preference model on these AI-generated labels
Use RLHF with this AI-labeled reward model

#Why Constitutional AI Scales

Human preference annotation is expensive and inconsistent — different annotators disagree, and the same annotator is inconsistent across sessions. CAI:

Scales annotation without requiring human judgment on every pair
Makes alignment principles explicit and auditable (you can read the constitution)
Allows rapid iteration — update the constitution, regenerate labels, retrain

#Comparison: RLHF vs DPO vs GRPO vs Constitutional AI

Aspect	RLHF	DPO	GRPO	Constitutional AI
Data required	100K+ preference pairs	10K–100K preference pairs	Any reward signal	A constitution (principles)
Separate reward model	Yes	No	Optional	Yes (AI-generated)
Training complexity	Very high (RL loop, 4 models)	Low (supervised)	Medium	Medium (2 phases)
GPU memory	4x SFT model size	2x SFT model size	1–2x	2x
Main failure mode	Reward hacking	Preference data quality	Group size sensitivity	Constitution gaps
Best for	Large teams, safety-critical	Startups, 7B–70B models	Reasoning/math tasks	Principled alignment at scale
Used by	OpenAI (ChatGPT)	Most open-source chat models	DeepSeek (R1)	Anthropic (Claude)

#Try It Yourself

pythonplayground.py · Pyodide

# Implement DPO loss from scratch
# and verify it behaves as expected

import math

def log_sigmoid(x):
  """Numerically stable log(sigmoid(x))"""
  if x >= 0:
      return -math.log(1 + math.exp(-x))
  else:
      return x - math.log(1 + math.exp(x))

pi_chosen = -5.0
pi_rejected = -5.0
ref_chosen = -5.0
ref_rejected = -5.0
learning_rate = 0.5

Tests · Verify Scenario 1 loss is approximately 0.693. Verify Scenario 2 loss is less than Scenario 1 loss. Verify Scenario 3 loss is greater than Scenario 1 loss.

Training language models to follow instructions with human feedback

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (2023)

Constitutional AI: Harmlessness from AI Feedback

#Key Takeaways

SFT teaches format, alignment teaches values. Supervised fine-tuning produces an assistant that follows instructions; RLHF and DPO teach it which instructions to follow and how to balance helpfulness, harmlessness, and honesty
RLHF requires four models and RL expertise. The SFT → reward model → PPO pipeline is powerful but expensive: four models in memory, a complex RL training loop, and constant vigilance against reward hacking
DPO derived that the reward model is unnecessary. By reparameterizing the optimal RLHF policy in terms of log-ratios, DPO turns alignment into a supervised learning problem over preference pairs, cutting compute by 10x and eliminating the RL loop
Reward hacking is alignment's Goodhart's Law. Any proxy measure of human preferences can be exploited; the KL penalty limits exploitation but does not eliminate it; monitoring for sycophancy, verbosity, and false confidence is part of production alignment
The right method depends on your resources. DPO for startups and 7B models; RLHF for safety-critical large-scale systems; GRPO for verifiable reward tasks like math; Constitutional AI for principled alignment without per-pair human annotation

#Quick Check

Quick Check1 / 4

In the RLHF pipeline, what is the purpose of the KL divergence penalty in the PPO objective?

RLHF & DPO: Aligning LLMs with Human Preferences

#The Alignment Problem

#Stage 1: Supervised Fine-Tuning (SFT)

#Stage 2: Reward Modeling

#Collecting Preference Data

#The Bradley-Terry Model

#Architecture

#Stage 3: PPO Optimization

#The RLHF Objective

#One PPO Update Step

#Step 1: Sample a batch of prompts

#Step 2: Generate responses with the current policy

#Step 3: Score responses with the reward model

#Step 4: Compute the KL penalty

#Step 5: Compute advantages

#Step 6: Update the policy

#The Four Models in RLHF

#Reward Hacking

#DPO: Cutting Out the Reward Model

#The Key Insight

#The DPO Loss

#What DPO Actually Does

#DPO Family & Successors

#GRPO: Group Relative Policy Optimization

#How It Works

#Why GRPO Matters

#Constitutional AI: Anthropic's Approach

#The Two-Phase Process

#Why Constitutional AI Scales

#Comparison: RLHF vs DPO vs GRPO vs Constitutional AI

#Try It Yourself

Deep Dive

#Key Takeaways

#Quick Check

Related Lessons

Your Reflection

Continue Learning

Discussion

RLHF & DPO: Aligning LLMs with Human Preferences

#The Alignment Problem

#Stage 1: Supervised Fine-Tuning (SFT)

#Stage 2: Reward Modeling

#Collecting Preference Data

#The Bradley-Terry Model

#Architecture

#Stage 3: PPO Optimization

#The RLHF Objective

#One PPO Update Step

#Step 1: Sample a batch of prompts

#Step 2: Generate responses with the current policy

#Step 3: Score responses with the reward model

#Step 4: Compute the KL penalty

#Step 5: Compute advantages

#Step 6: Update the policy

#The Four Models in RLHF

#Reward Hacking

#DPO: Cutting Out the Reward Model

#The Key Insight

#The DPO Loss

#What DPO Actually Does

#DPO Family & Successors

#GRPO: Group Relative Policy Optimization

#How It Works

#Why GRPO Matters

#Constitutional AI: Anthropic's Approach

#The Two-Phase Process

#Why Constitutional AI Scales

#Comparison: RLHF vs DPO vs GRPO vs Constitutional AI

#Try It Yourself

Deep Dive

#Key Takeaways

#Quick Check

Related Lessons

Your Reflection

Continue Learning

Discussion