GRPO & DeepSeek R1: Training Reasoning Models

#The Problem with PPO

To understand why GRPO exists, you first need to understand what makes PPO expensive.

PPO (Proximal Policy Optimization) is the RL algorithm that powers RLHF — the training recipe that produced ChatGPT and the first generation of aligned LLMs. It works well, but it requires four models in memory simultaneously:

For a 7B model at bf16 precision, that is roughly 4 × 14 GB = 56 GB of parameters before activations, optimizer states, or gradient buffers. In practice, a PPO run on a 7B model needs 8-16 A100s just to fit in memory.

#The Value Function's Three Costs

1. Memory: An entire copy of the LLM architecture, trained in parallel with the policy. For a 70B model, this adds ~140 GB of VRAM — effectively doubling the hardware requirement.
2. Training instability: The value function and the policy are co-dependent. If the value function underestimates expected reward, the policy gets overconfident positive advantages and takes too-large gradient steps. If it overestimates, the policy is pessimistic and barely updates. Getting this interaction stable requires careful hyperparameter tuning, learning rate scheduling, and clipping parameters — a significant engineering burden.
3. Architectural mismatch: A standard transformer generates sequences autoregressively. But the value function needs to predict the scalar reward that will come at the end of a sequence given only partial tokens. This is fundamentally a different task than language modeling, and forcing the same architecture to do both creates tension.

#GRPO: Group Relative Policy Optimization

#The Core Idea

#The GRPO Advantage Formula

#The GRPO Training Objective

#DeepSeek R1's Training Recipe

DeepSeek did not simply swap PPO for GRPO and call it done. The R1 technical report describes a careful four-stage pipeline designed to avoid failure modes that appeared in earlier experiments.

#Stage 0: The Failed Experiment (DeepSeek R1-Zero)

Before R1, DeepSeek ran an experiment: train a base LLM (DeepSeek V3 Base, 671B MoE parameters) with pure GRPO, starting with no SFT, using only rule-based math rewards. No human demonstrations, no chain-of-thought examples — just reinforcement learning from scratch.

#Stage 1: Cold-Start SFT (Fixing R1-Zero's Problems)

To give the model coherent output behavior before RL training begins, DeepSeek first ran a short SFT phase on a carefully curated dataset of a few thousand examples with:

• Multi-step reasoning formatted in <think>...</think> tags
• Clear, human-readable language throughout (no language mixing)
• Explicit self-reflection and verification steps
• Well-structured final answers in the correct format

#Stage 2: GRPO with Accuracy + Format Rewards

With the cold-start model in hand, DeepSeek ran the main GRPO training phase. The reward function combined two signals:

• +1 if the final answer (extracted from the response) exactly matches the ground-truth answer
• 0 otherwise
• Applied to math problems with deterministic correct answers

• +0.5 for correctly using the <think>...</think> format
• Penalties for language mixing (switching between English and Chinese mid-response)
• No reward for response length — the model was not directly incentivized to reason longer

#Stage 3: Rejection Sampling + SFT on R1-Generated Data

After GRPO training converged, DeepSeek ran rejection sampling: for each problem, generate many responses from the GRPO-trained model and keep only the ones that reach the correct answer. These correct reasoning traces then became a new SFT dataset.

This stage served two purposes:

1. Stabilize the model: Supervised training on high-quality R1 outputs smoothed out remaining inconsistencies from RL training
2. Create distillation data: The correct reasoning traces from the 671B model could then be used to fine-tune much smaller models (1.5B, 7B, 14B, 32B, 70B) — giving them strong reasoning ability without running GRPO themselves

#Stage 4: Final GRPO Refinement

A short final GRPO pass on the Stage 3 model, using both math rewards and a broader set of reward signals (helpfulness, format, safety) to produce the final R1 model that was released publicly.

#The "Aha Moment": Emergent Reasoning

The most scientifically interesting finding in the R1 paper was not the final benchmark numbers — it was the spontaneous emergence of self-reflective reasoning during GRPO training.

At a specific point in training (roughly when the model's accuracy on training problems crossed ~60%), the reasoning traces in rollouts began containing phrases like:

"Wait, I think I made an error. Let me reconsider..." "Actually, that approach doesn't work because..." "Let me try a different method..."

These self-corrections were not in the training data. No human demonstration showed the model how to catch and fix its own mistakes. The model discovered that self-correction was a useful strategy — if it noticed it was going down a wrong path, backtracking and trying again produced more correct final answers, which produced more reward.

This is a profound observation: reasoning and self-correction are learnable strategies, not architectural features. The model did not become smarter in any fundamental sense — its weights changed to produce behaviors that historically led to higher reward, and self-reflective reasoning was one of those behaviors.

#Process Reward Models vs Outcome Reward Models

#Outcome Reward Models (ORMs)

An ORM assigns a scalar reward only to the complete response, based on whether the final answer is correct:

def outcome_reward(response: str, ground_truth: str) -> float:
    """
    Rule-based ORM for math problems.
    Returns 1.0 if the extracted answer matches, 0.0 otherwise.
    """
    extracted = extract_boxed_answer(response)
    if extracted is None:
        return 0.0
    return 1.0 if extracted.strip() == ground_truth.strip() else 0.0

Advantages of ORMs

• Simple to implement — no need to label intermediate steps
• Rule-based when ground truth is verifiable (no reward model training needed)
• Reward hacking is harder — the model cannot "game" a rule-based checker the way it can game a learned reward model

Disadvantages of ORMs

• Sparse rewards: The model gets no signal about why a response was wrong, only that it was. Long reasoning chains with a single error get zero reward, even if 95% of the reasoning was correct.
• Credit assignment problem: In a 500-token reasoning chain, which of the 500 steps caused the wrong answer? ORM cannot tell you.

#Process Reward Models (PRMs)

A PRM evaluates each step of the reasoning chain independently, assigning a score (correct / incorrect / uncertain) to each intermediate step:

def process_reward(
    problem: str,
    reasoning_steps: list[str],
    prm_model: torch.nn.Module
) -> list[float]:
    """
    PRM evaluates each step in the reasoning chain.
    Returns a score per step rather than a single final reward.
    """
    step_rewards = []
    context = problem
    for step in reasoning_steps:
        # Score this step given the problem and all previous steps
        score = prm_model(context, step)
        step_rewards.append(score)
        context += "\n" + step
    return step_rewards

Advantages of PRMs

• Dense rewards: Every step in a long reasoning chain gets a signal, not just the final answer
• Better credit assignment: The model learns which specific steps lead to correct conclusions
• OpenAI's "Let's Verify Step by Step" (2023) showed PRMs significantly outperform ORMs on competitive math

Disadvantages of PRMs

• Hard to build: Requires labeling thousands of reasoning steps, which demands annotators who can verify mathematical reasoning at each step — expensive and slow
• Reward hacking risk: A learned PRM is itself a model that can be exploited. A smart policy might learn to produce steps that look correct to the PRM without actually being correct — a subtle and hard-to-detect failure mode
• Step ambiguity: Where does one "step" end and another begin? Mathematical reasoning does not always decompose cleanly into discrete, independently-evaluable steps

The answer depends on your annotation capacity. If you can build a rule-based checker that verifies all three elements (ORM — simple and scalable), do that first. If you find the model is consistently getting one element wrong but you cannot tell which, switching to PRM for that element specifically can help. The legal domain is actually well-suited for hybrid approaches: verifiable elements (dates, party names) get rule-based ORM rewards; open-ended elements (obligation characterization) might need a learned PRM or human evaluation.

#GRPO in Practice: Implementing the Reward Loop

Here is a practical GRPO reward function and sampling loop for a math reasoning task:

#GRPO vs DPO vs PPO: When to Use Each

#Try It Yourself

#Key Takeaways

• PPO's value function is its biggest cost. Four models in memory, a complex co-training problem, and a second optimization loop that requires careful tuning; GRPO eliminates the value function by using group-relative advantages instead
• GRPO replaces learned baselines with empirical ones. Instead of training a model to predict "how good is the average response to this prompt?", GRPO samples G responses and measures it directly; the baseline is the group mean, not a neural network
• DeepSeek R1's cold-start SFT was a practical necessity. Pure RL from scratch (R1-Zero) produced powerful but unreadable reasoning; a brief cold-start SFT phase established output format before GRPO shaped content
• Chain-of-thought reasoning is a learnable strategy. When rewarded only for correct final answers, the model discovered that generating extended reasoning traces and self-correcting improved accuracy; this was not trained, it emerged from the reward signal
• PRMs provide denser signal than ORMs but cost more to build. Process rewards evaluate each step, eliminating the credit assignment problem; outcome rewards are simpler and work well with rule-based checkers; most teams start with ORMs and add PRM evaluation if they hit a ceiling
• Choose GRPO for verifiable tasks. If you can write a rule-based checker for correctness (math, code, structured output), GRPO with that checker is the most efficient path to reasoning improvement; use DPO for preference alignment and PPO only when you have dedicated RLHF infrastructure

#Quick Check

Aᵢ = (rᵢ - mean(r₁...rG)) / std(r₁...rG)

This is the key insight: the baseline is the group average, not a learned value function. Responses above average get positive advantage (reinforce them). Responses below average get negative advantage (suppress them).

import re
import torch
from typing import Optional
from transformers import AutoModelForCausalLM, AutoTokenizer

def extract_boxed_answer(text: str) -> Optional[str]:
    """Extract the answer from \\boxed{...} notation."""
    match = re.search(r'\\boxed\{([^}]+)\}', text)
    return match.group(1) if match else None

def compute_grpo_reward(
    response: str,
    ground_truth: str,
    format_reward: float = 0.5,
    accuracy_reward: float = 1.0,
) -> float:
    """
    Rule-based reward function for math reasoning.
    Combines accuracy reward (primary) with format reward (secondary).
    
    Args:
        response: Full model response including reasoning chain
        ground_truth: Correct answer string
        format_reward: Reward for using correct \\boxed{} format
        accuracy_reward: Reward for correct final answer
    
    Returns:
        Scalar reward in [0, 1.5]
    """
    reward = 0.0
    
    # Format reward: did the model use \\boxed{} notation?
    if extract_boxed_answer(response) is not None:
        reward += format_reward
    
    # Accuracy reward: is the extracted answer correct?
    extracted = extract_boxed_answer(response)
    if extracted is not None and extracted.strip() == ground_truth.strip():
        reward += accuracy_reward
    
    return reward

def grpo_rollout(
    model: AutoModelForCausalLM,
    tokenizer: AutoTokenizer,
    prompt: str,
    ground_truth: str,
    G: int = 8,
    max_new_tokens: int = 2048,
    temperature: float = 0.9,
) -> dict:
    """
    GRPO rollout: sample G responses, compute rewards, compute advantages.
    
    Returns:
        dict with 'responses', 'rewards', 'advantages', 'log_probs'
    """
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    responses = []
    rewards = []
    log_probs_list = []
    
    for _ in range(G):
        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=max_new_tokens,
                temperature=temperature,
                do_sample=True,
                return_dict_in_generate=True,
                output_scores=True,
            )
        
        # Decode only the new tokens (exclude the prompt)
        new_tokens = outputs.sequences[0][inputs['input_ids'].shape[1]:]
        response = tokenizer.decode(new_tokens, skip_special_tokens=True)
        
        # Compute reward
        reward = compute_grpo_reward(response, ground_truth)
        
        # Compute token-level log probabilities for the generated tokens
        log_prob = sum(
            torch.log_softmax(score, dim=-1).max().item()
            for score in outputs.scores
        )
        
        responses.append(response)
        rewards.append(reward)
        log_probs_list.append(log_prob)
    
    # Compute group-relative advantages
    rewards_tensor = torch.tensor(rewards)
    mean_r = rewards_tensor.mean()
    std_r = rewards_tensor.std(unbiased=False) + 1e-8
    advantages = (rewards_tensor - mean_r) / std_r
    
    return {
        "responses": responses,
        "rewards": rewards,
        "advantages": advantages.tolist(),
        "log_probs": log_probs_list,
        "mean_reward": mean_r.item(),
        "std_reward": std_r.item(),
    }

# Example usage
if __name__ == "__main__":
    # Simulate GRPO rollout results (without loading a real model)
    simulated_rewards = [1.5, 0.0, 1.5, 0.5, 0.0, 1.5, 0.5, 1.0]
    rewards_t = torch.tensor(simulated_rewards, dtype=torch.float32)
    advantages = (rewards_t - rewards_t.mean()) / (rewards_t.std(unbiased=False) + 1e-8)
    
    print("Simulated GRPO rollout for G=8:")
    print(f"  Rewards:    {simulated_rewards}")
    print(f"  Mean reward: {rewards_t.mean():.3f}")
    print(f"  Advantages: {advantages.tolist()}")
    print()
    print("Interpretation:")
    for i, (r, a) in enumerate(zip(simulated_rewards, advantages.tolist())):
        direction = "REINFORCE" if a > 0 else "SUPPRESS "
        print(f"  Response {i+1}: reward={r:.1f}, advantage={a:+.3f} -> {direction}")