LLM & VLM courseBlog
70Track 4: Post-training and RL · session 18 of 25

GRPO and RL with verifiable rewards (RLVR)

GRPO (Group Relative Policy Optimization) removes PPO's critic: it samples a group of answers for each prompt and uses how much better each answer is than its group as the advantage. Combined with rewards that a program can check (correct math answers, passing unit tests, format rules), it became the recipe behind DeepSeek-R1 and most open reasoning models. This session builds the objective, counts the memory it saves, follows the path from DeepSeekMath to DeepSeek-R1, and explains the known biases and their fixes in Dr. GRPO and DAPO.

Time: 3 to 4 hours (reading 2 to 2.5 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 38, 64, 65 and 66

What you will learn

  • How the group-relative advantage works, why it sums to zero, and why a group where every answer gets the same reward teaches nothing
  • The full GRPO objective: the PPO clip per token, the k3 KL estimator, and outcome versus process supervision
  • What dropping the critic saves in memory (126 GB against 252 GB of model state for a 7B policy) and what it costs in credit assignment
  • What a verifiable reward is, how math, code, format and IoU checkers are built, and how a policy can still exploit a loose verifier
  • The path from DeepSeekMath to DeepSeek-R1-Zero and DeepSeek-R1, with the real hyperparameters and results
  • The length bias and the difficulty bias of the original loss, and the fixes in Dr. GRPO and DAPO (token-level loss, clip-higher, dynamic sampling, overlong shaping)
  • A runnable GRPO loop in PyTorch with a verifier, a reference model and a switch between GRPO and Dr. GRPO aggregation

Why this matters

In session 66 we saw that classic RLHF keeps four models on the GPUs: the policy, a frozen reference, a reward model and a critic. Two of them are trained. The critic alone costs a full model of weights, gradients and optimizer state, and it is hard to train well on long text. GRPO (Group Relative Policy Optimization, from the DeepSeekMath paper, Shao et al., 2024) removes the critic. For each prompt it samples a group of answers, scores them all, and uses “how much better than the other answers in my group” as the advantage. [Certain]

The second idea of this session is the reward itself. For math, code and many formatted tasks we do not need a learned reward model: a short program can check the answer. The final number matches, the unit tests pass, the output is valid JSON. This is RLVR (reinforcement learning with verifiable rewards), a name introduced by the Tülu 3 paper (Lambert et al., 2024). [Certain] GRPO plus rule-based rewards is the recipe of DeepSeek-R1-Zero, which learned long step-by-step reasoning from a base model with no supervised reasoning data. [Certain] Today almost every open reasoning model is trained with GRPO or a close variant. [Likely]

  • One less model. No critic: one trained model instead of two. With the KL term turned off, no reference model either.
  • Rewards you can trust. A checker cannot be flattered by long or confident text the way a learned reward model can. It can still be fooled, and we will see how.
  • Simple code. The advantage is a mean and a standard deviation over a few numbers. The rest is the PPO clipped loss you already know.
  • Known problems. The original GRPO loss has a bias toward long wrong answers and a hidden weighting by question difficulty. Dr. GRPO and DAPO fixed these in 2025, and most libraries now use the fixed versions. [Likely]

One prompt, many answers: the group baseline

Recall the policy gradient from session 64: push up the log-probability of each sampled answer in proportion to its advantage, the reward minus a baseline. A good baseline is the expected reward of the prompt: “what score do I usually get here?”. PPO learns this baseline with a critic network (session 65). GRPO estimates it directly. For one prompt \(q\), sample \(G\) answers \(o_1, \dots, o_G\) from the current policy and score each one, giving rewards \(r_1, \dots, r_G\). The mean of the group is a sample estimate of the prompt’s expected reward. Then

\[ \hat A_{i,t} = \tilde r_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)} \qquad \text{for every token } t \text{ of answer } i \tag{70.1} \]
Read the symbols
\(q\)
one prompt (in the papers, a “question”)
\(G\)
the group size: how many answers we sample for the same prompt; DeepSeekMath used 64, DeepSeek-R1 used 16
\(o_i\), \(r_i\)
the \(i\)-th sampled answer (“output”) and its scalar reward
\(\tilde r_i\)
the normalized reward: centered by the group mean, scaled by the group standard deviation
\(\hat A_{i,t}\)
the advantage of token \(t\) in answer \(i\); with one reward per answer (“outcome supervision”), every token of the answer gets the same value

Three facts follow at once. First, the advantages of a group sum to zero: some answers are pushed up and the others are pushed down by the same total amount. Second, if all answers get the same reward (all right or all wrong), every advantage is zero, and the prompt teaches nothing. In code a small \(\varepsilon\) is added to the denominator so that this case gives 0 and not a division by zero. Third, the baseline is per prompt: a hard question where 1 answer in 8 is right and an easy one where 7 in 8 are right each get their own mean. This is exactly what a critic tries to learn, but GRPO gets it for free from the samples.

Session 64 compared this with two cousins. RLOO (REINFORCE leave-one-out) uses the mean of the other \(G-1\) answers as the baseline, which equals \(\tfrac{G}{G-1}(r_i - \bar r)\) and is unbiased. Dividing by the standard deviation is GRPO’s own choice. It makes the advantages unit-free, so the learning rate does not depend on the reward scale. But it also changes how much each prompt counts, and we will see that this is one of GRPO’s known problems.

Worked example: a group of eight answers to one math question

The verifier gives 1 for a correct final answer and 0 otherwise. Rewards: \([1, 0, 0, 1, 0, 0, 0, 1]\). The mean is \(3/8 = 0.375\).

The standard deviation, dividing by \(G\) (the “population” version), is \(\sqrt{0.375 \times 0.625} = 0.4841\). So each correct answer gets \(\hat A = 0.625 / 0.4841 = 1.291\) and each wrong answer gets \(-0.375 / 0.4841 = -0.775\). Check: \(3 \times 1.291 - 5 \times 0.775 = 0\).

PyTorch’s std() divides by \(G - 1\) by default, which gives \(0.5175\) and advantages \(1.208\) and \(-0.725\). Libraries differ here; the ratio between right and wrong is the same, only the scale changes.

Without the division (mean only, as in Dr. GRPO): \(+0.625\) and \(-0.375\). RLOO: \(\tfrac{8}{7} \times 0.625 = 0.714\) and \(\tfrac{8}{7} \times (-0.375) = -0.429\).

correct answer, reward 1wrong answer, reward 0
Figure 1. The group-relative advantage (70.1) for one prompt. Each button is one sampled answer; click it to mark it correct (green) or wrong. Bars show the advantage that every token of that answer receives. The standard deviation divides by \(G\).
How to read Figure 1

Start with 3 of 8 correct: correct answers get +1.291, wrong ones −0.775. Click until only one answer is correct: its bar grows to +2.646, because a rare success on a hard question is a big surprise. Make all answers wrong, or all correct: every bar is zero and the readout says the group has no signal. Switch to “Mean only”: bars become \(r_i - \bar r\) and no longer grow for rare successes. Switch to RLOO: the same shape as mean only, scaled by \(G/(G-1)\).

The GRPO objective

With the advantage in hand, GRPO reuses the PPO clipped objective from session 66 token by token, and adds a KL penalty to a frozen reference model directly in the loss (not in the reward, as PPO-RLHF does). DeepSeekMath writes the objective, to be maximized, as [Certain]

\[ \mathcal{J}_{\text{GRPO}}(\theta) = \E_{q,\ \{o_i\} \sim \pi_{\theta_\text{old}}} \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \Big\{ \min\!\big( \rho_{i,t} \hat A_{i,t},\ \operatorname{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\, \hat A_{i,t} \big) - \beta\, \mathbb{D}_{i,t} \Big\} \tag{70.2} \]
Read the symbols
\(\rho_{i,t}\)
the probability ratio \(\pi_\theta(o_{i,t} \mid q, o_{i,\lt t}) / \pi_{\theta_\text{old}}(o_{i,t} \mid q, o_{i,\lt t})\) of token \(t\) of answer \(i\), as in PPO
\(\pi_{\theta_\text{old}}\)
the policy that generated the group; its token log-probabilities are stored once
\(|o_i|\)
the number of tokens in answer \(i\); the inner sum is an average over the tokens of one answer
\(\epsilon\)
the clip range, usually 0.2
\(\beta\)
the KL coefficient; 0.04 in DeepSeekMath, 0.001 in DeepSeek-R1
\(\mathbb{D}_{i,t}\)
a per-token estimate of the KL divergence from the policy to the reference model, formula (70.3)

Notice the two averages. First over the \(|o_i|\) tokens of one answer, then over the \(G\) answers. This “mean of means” looks harmless, but it is the source of the length bias discussed later. The DeepSeek-R1 paper writes the objective without the \(1/|o_i|\) factor. [Certain]

The KL term uses Schulman’s \(k_3\) estimator (we met it in session 66), evaluated on the sampled token: [Certain]

\[ \mathbb{D}_{i,t} = \frac{\pi_\text{ref}(o_{i,t} \mid \cdot)}{\pi_\theta(o_{i,t} \mid \cdot)} - \log \frac{\pi_\text{ref}(o_{i,t} \mid \cdot)}{\pi_\theta(o_{i,t} \mid \cdot)} - 1 \tag{70.3} \]
Read the symbols
\(\pi_\text{ref}\)
the frozen reference model, usually the SFT model the run started from
\(\cdot\)
short for the context \(q, o_{i,\lt t}\)
\(\mathbb{D}_{i,t}\)
never negative (the function \(x - \log x - 1\) has its minimum 0 at \(x = 1\)), and its average over samples from \(\pi_\theta\) is the true KL

For example, a token the reference finds half as likely as the policy (\(x = 0.5\)) costs \(0.5 + 0.693 - 1 = 0.193\); a token the reference finds twice as likely (\(x = 2\)) costs \(2 - 0.693 - 1 = 0.307\).

Outcome supervision and process supervision

Formula (70.1) is outcome supervision: one reward at the end of each answer. DeepSeekMath also tried process supervision, where a process reward model scores each reasoning step. The step rewards of all answers in the group are normalized together (\(\mathbf{R}\) below is the set of all step rewards of the group), and a token’s advantage is the sum of the normalized rewards of the steps that end at or after it: [Certain]

\[ \tilde r_i^{(j)} = \frac{r_i^{(j)} - \operatorname{mean}(\mathbf{R})}{\operatorname{std}(\mathbf{R})}, \qquad \hat A_{i,t} = \sum_{j:\ \text{index}(j) \ge t} \tilde r_i^{(j)} \tag{70.4} \]

Here \(r_i^{(j)}\) is the reward of step \(j\) of answer \(i\), and \(\text{index}(j)\) is the token where step \(j\) ends. In DeepSeekMath, process supervision did slightly better than outcome supervision. [Certain] DeepSeek-R1 later went back to outcome rewards from rules, and listed process reward models among its unsuccessful attempts: steps are hard to define, step correctness is hard to judge, and a learned step scorer invites reward hacking. [Certain] Process rewards are covered in session 73.

No critic: what we save and what we lose

Using the memory rule from session 38 (about 16 bytes per trained parameter with mixed-precision AdamW, 2 bytes per frozen bf16 parameter), GRPO needs

\[ M_\text{GRPO} \approx 16\, N_\pi + 2\, N_\text{ref}\, \mathbb{1}[\beta \gt 0] \ \text{bytes} \;+\; \text{activations} + \text{KV cache} \tag{70.5} \]
Read the symbols
\(N_\pi\), \(N_\text{ref}\)
parameters of the policy (trained) and of the reference (frozen, same size)
\(\mathbb{1}[\beta \gt 0]\)
1 if the KL term is used, else 0: with \(\beta = 0\) the reference model is not loaded at all
no reward-model term
with verifiable rewards, the reward is a program on the CPU; with a learned reward model, add \(2 N_\text{RM}\)
Worked example: a 7B policy

PPO with a 7B reward model and a 7B critic (from session 66): \(16 \times 7 + 16 \times 7 + 2 \times 7 + 2 \times 7 = 252\) GB of model state.

GRPO with a verifier and a KL term: \(16 \times 7 + 2 \times 7 = 126\) GB, half of PPO. GRPO with \(\beta = 0\): 112 GB, the same as plain SFT of the model.

The KV cache for generation is not in these numbers. With groups of 16 answers of up to 32k tokens, generation memory is often the real limit, which is why rollouts usually run in a separate inference engine such as vLLM (session 75). [Likely]

What do we lose? Three things.

  • Per-token credit. A critic can say “the answer went wrong at token 140”. GRPO gives every token of an answer the same advantage. The good first steps of a wrong answer are pushed down together with the mistake. Over many groups this averages out, because the same good steps also appear in correct answers. [Likely]
  • More samples per prompt. GRPO needs \(G\) answers per prompt to estimate the baseline. PPO can learn from one answer per prompt. But generation is batched, and for reasoning tasks several samples per prompt are useful anyway. [Likely]
  • No learning from uniform groups. A prompt where all \(G\) answers get the same reward produces zero gradient. The next section measures how often this happens.
PPO-RLHF (session 66)GRPO with verifiable rewards
AdvantageGAE from a learned critic, per tokengroup-normalized reward, same for all tokens of an answer
Trained modelspolicy and criticpolicy only
Frozen modelsreference and reward modelreference (optional); the reward is a program
KL to the referenceinside the per-token rewarda separate loss term, \(k_3\) estimator (70.3), or none
Samples per promptusually 1a group: 8 to 64
Typical usechat alignment with human preferencesmath, code, formatted output, reasoning

Verifiable rewards (RLVR)

A verifiable reward is a reward computed by a deterministic program from the answer and some ground truth, with no learned model in between. The Tülu 3 paper defines RLVR as giving a constant reward when a completion is verified correct and 0 otherwise; they used a reward of 10 for a correct answer and trained with PPO on GSM8K, MATH and instruction-following constraints (IFEval-style). [Certain] So RLVR is a kind of reward, not an algorithm. It works with PPO, GRPO, RLOO or any other policy-gradient method.

TaskVerifierTypical traps
Math with a final answerExtract the answer (for example from \boxed{...} or an answer tag), normalize it, compare with the ground truth; symbolic checkers (such as sympy) accept \(1/2 = 0.5\)Bad extraction marks right answers wrong; multiple-choice answers can be guessed
CodeRun the program on unit tests in a sandbox; reward 1 if all pass, or the fraction passedToo few tests: the model learns special cases; code that edits or skips the tests
FormatRegular expression: reasoning inside <think>...</think>, answer inside <answer>...</answer>; valid JSON; a required languageFormat reward much bigger than accuracy reward: the model learns the format only
Instruction constraintsCheck “at most 3 sentences”, “include the word X”, “no commas”Satisfies the letter of the constraint with a useless answer
Vision and videoIoU (intersection over union) of a predicted box or time span with the true one; exact match of a counted numberReward on the box only, not on the explanation; ambiguous ground truth

DeepSeek-R1-Zero used two rule-based rewards with equal weight: an accuracy reward (is the final answer right; for code, do the tests pass) and a format reward (is the thinking inside think tags). [Certain] For one answer \(o\) to prompt \(q\):

\[ r(q, o) = r_\text{acc}(q, o) + r_\text{format}(o), \qquad r_\text{acc},\, r_\text{format} \in \{0, 1\} \tag{70.6} \]

The R1 paper explains why it did not use a neural reward model for reasoning: such models are “susceptible to reward hacking” in large-scale RL, and retraining them costs compute and adds complexity. [Certain] In the second stage of DeepSeek-R1 a language-consistency reward was added, the fraction of chain-of-thought words in the target language, because the model mixed languages in its reasoning. [Certain]

That last sentence is the weakness of RLVR. The verifier checks the outcome, not the reasoning. With enough samples, a policy finds any gap in the checker: answers in a form the parser accepts too easily, code that special-cases the visible tests, or guessing on multiple-choice questions. Good RLVR data therefore has hidden tests, answers that are hard to guess (numbers, not A/B/C/D), and strict but correct parsing. [Likely]

Figure 2. One GRPO iteration with the reward (70.6) on the prompt “What is 17 × 24?” (the true answer is 408). Four answers are sampled. Step through: sampling, verification, group statistics, advantages, and the update. The standard deviation divides by \(G\).
How to read Figure 2

Step 2 shows the verifier: answer C has the right number but no answer tags, so the parser cannot extract it and both rewards are 0. Step 3 gives the group mean 1.25 and standard deviation 0.829. Step 4 turns these into advantages +0.905, −0.302, −1.508, +0.905: answer B is pushed down only a little, because it got the format right. Step 5 shows that every token of an answer shares its advantage, and that the reference model is needed only when \(\beta \gt 0\).

When does a group teach anything?

Suppose the policy solves a prompt with probability \(p\) and the reward is 0 or 1. A group of \(G\) independent answers has a non-zero advantage only if it contains at least one right and at least one wrong answer:

\[ P(\text{signal}) = 1 - p^{G} - (1-p)^{G} \tag{70.7} \]
Read the symbols
\(p\)
the chance that one sampled answer is correct, for this prompt and the current policy
\(p^G\), \((1-p)^G\)
the chance that all \(G\) answers are right, or all are wrong; then all advantages are zero

For a hard prompt with \(p = 0.05\): with \(G = 8\) only 33.7% of groups carry any signal, with \(G = 16\) it is 56.0%, and with \(G = 64\) it is 96.3%. Prompts the model always solves are equally useless. As training goes on, more prompts become “always solved”, and a fixed batch of prompts carries less and less gradient. DAPO’s dynamic sampling fixes this by over-sampling and dropping every group whose accuracy is exactly 0 or 1, until the batch is full of useful groups. [Certain] Many recipes also filter the prompt set before training to keep prompts of medium difficulty. [Likely]

The same figure shows the second effect: the scale of the advantage. With 0/1 rewards the group std is \(\sqrt{p(1-p)}\) (for a group with exactly a fraction \(p\) correct). Dividing by it gives a rare correct answer a big advantage: with 1 correct out of 16, \(\hat A = (1 - 1/16)/0.242 = 3.873\), against 1.0 for each correct answer when half are right. Summed over a group, the total push \(\sum_i |\hat A_i|\) is \(2G\sqrt{p(1-p)}\) with the std division and \(2G\,p(1-p)\) without it. So relative to a medium prompt, GRPO gives very hard and very easy prompts more weight than an unscaled advantage would.

selected \(G\)other group sizestotal push \(\sum|\hat A|/G\) with std division, dashedwithout std division, dotted
Figure 3. Solid curves: formula (70.7), the chance that a group of \(G\) answers contains both a right and a wrong answer, for \(G = 4, 8, 16, 64\). Dashed and dotted curves: the expected total advantage per answer, \(2\sqrt{p(1-p)}\) with GRPO’s std division and \(2p(1-p)\) without it, for a group whose fraction correct equals \(p\).
How to read Figure 3

At \(p = 0.05\) and \(G = 8\), only about a third of the groups give any gradient. Move to \(G = 64\): almost all do, at 8 times the generation cost. Slide \(p\) to 0.5: every group size gives signal. Compare the orange dashed and the green dotted curves near the edges: the std division keeps the push large for prompts that are almost always solved or failed; without it, these prompts fade out smoothly.

From DeepSeekMath to DeepSeek-R1

DeepSeekMath (February 2024). GRPO was introduced to train DeepSeekMath-RL 7B from DeepSeekMath-Instruct 7B. The reward was a learned reward model, not a rule. Training used about 144K math questions, 64 samples per question, a learning rate of \(10^{-6}\), \(\beta = 0.04\), a maximum length of 1024 tokens, a batch of 1024, and a single policy update per batch of samples (so the ratio is 1 and the clip never acts). [Certain] Results: GSM8K rose from 82.9% to 88.2% and MATH from 46.8% to 51.7%. [Certain] The paper also found that RL improved majority-vote accuracy (Maj@K) but not Pass@K, and suggested RL makes the output distribution more robust rather than adding new abilities. [Certain] It also ran “iterative RL”, where the reward model was retrained on new policy samples with a replay of 10% old data. [Certain]

DeepSeek-R1-Zero (January 2025). GRPO applied directly to DeepSeek-V3-Base, with no supervised fine-tuning first, and only the two rule-based rewards (70.6). The prompt template only asked the model to think inside think tags and then answer. [Certain] AIME 2024 pass@1 rose from 15.6% to 71.0% during training, and 86.7% with majority voting over 64 samples (cons@64). [Certain] The average response length grew steadily by itself, and the paper describes an “aha moment”: an intermediate checkpoint starts to re-check its own work, with a sudden rise in words like “wait”. [Certain] The later version of the paper lists the R1-Zero settings: learning rate \(3 \times 10^{-6}\), \(\beta = 0.001\), 16 answers per question, 32 questions per step (batch 512), maximum length 32,768 tokens raised to 65,536 after step 8.2k, about 10,400 steps, and the reference model replaced by the current policy every 400 steps. [Certain] (That paper version reports 77.9% for the final R1-Zero; the first version reported 71.0%.)

R1-Zero had problems: poor readability and mixed languages. DeepSeek-R1 fixed them with a multi-stage pipeline: a few thousand long chain-of-thought examples as a “cold start” SFT; reasoning RL with GRPO plus the language-consistency reward; rejection sampling of about 600k reasoning and 200k non-reasoning examples for a second SFT; and a final RL stage for all scenarios, with rule rewards for reasoning and reward models for helpfulness and harmlessness. [Certain] DeepSeek-R1 reached 79.8% on AIME 2024 and 97.3% on MATH-500. [Certain]

The same paper distilled R1 into six smaller dense models (1.5B to 70B, Qwen and Llama). DeepSeek-R1-Distill-Qwen-32B scored 72.6% on AIME 2024, while RL run directly on Qwen-32B (“R1-Zero-Qwen-32B”) scored only 47.0%. [Certain] The lesson: for small models, SFT on a strong reasoner’s outputs (session 59) can beat RL from scratch. Test-time reasoning is the topic of session 72.

SettingDeepSeekMath-RLDeepSeek-R1-ZeroDAPO
Start modelDeepSeekMath-Instruct 7BDeepSeek-V3-BaseQwen2.5-32B base
Rewardlearned reward modelrules: accuracy + formatrules: correct or not, plus overlong shaping
Group size \(G\)641616
Learning rate\(10^{-6}\)\(3 \times 10^{-6}\)\(10^{-6}\), 20-step warmup
KL \(\beta\)0.040.0010 (removed)
Max length102432,768, then 65,53620,480 (16,384 + 4,096 buffer)
Updates per batch of samples116 mini-batches, 1 epoch16 gradient updates
Headline resultMATH 46.8% → 51.7%AIME 2024 15.6% → 71.0%AIME 2024: 50

All values in this table are from the three papers. [Certain]

Known issues and fixes

After R1, many groups reproduced R1-Zero-style training on open models and looked closely at the loss. Two papers from March 2025 became the standard references: Dr. GRPO (“GRPO Done Right”, in Liu et al., Understanding R1-Zero-Like Training) and DAPO (Yu et al.). [Certain]

Problem 1: the length bias of \(1/|o_i|\)

In (70.2), each answer’s token sum is divided by its own length. So each answer gets the same total weight \(\hat A_i / G\), whatever its length, and the weight per token is \(\hat A_i / (G\, |o_i|)\). For a correct answer (\(\hat A \gt 0\)), a short answer gets a bigger push per token, which favors short correct answers. For a wrong answer (\(\hat A \lt 0\)), a long answer gets a smaller penalty per token, so long wrong answers are punished less. Liu et al. argue this makes wrong answers grow longer and longer, and that some of the famous length growth of R1-Zero-style runs comes from this bias, not from better reasoning. [Certain] They also found that open-source implementations (TRL, OpenRLHF, verl and others) all normalized the loss by response length. [Certain]

Problem 2: the difficulty bias of the std

As shown in Figure 3, dividing by the group std gives more weight to prompts with a low std, that is, prompts the policy almost always solves or almost always fails. Liu et al. call this a question-level difficulty bias. [Certain]

Fix: Dr. GRPO

Dr. GRPO simply removes both normalizations: no \(1/|o_i|\) and no division by the std. To keep the loss scale fixed, the token sum is divided by a constant, such as the generation budget \(L_\text{max}\), instead of the answer length: [Certain]

\[ \mathcal{J}_{\text{Dr.GRPO}}(\theta) = \E \ \frac{1}{G\, L_\text{max}} \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \min\!\big( \rho_{i,t} \hat A_i,\ \operatorname{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\, \hat A_i \big), \qquad \hat A_i = r_i - \operatorname{mean}(r_1, \dots, r_G) \tag{70.8} \]
Read the symbols
\(L_\text{max}\)
a constant, for example the maximum generation length; the same for every answer and every step
\(\hat A_i\)
the centered reward, with no std division; this is the plain REINFORCE-with-group-mean-baseline advantage

With this change, every token gets weight \(\hat A_i / (G L_\text{max})\), so a long wrong answer is penalized in proportion to its length. Their minimal recipe reached 43.3% on AIME 2024 with a 7B base model in 27 hours on 8 A100 GPUs, and used fewer tokens per answer than GRPO, especially for wrong answers. [Certain] The paper also found that Qwen2.5 base models already reason well with no prompt template, and that nearly all base models it tested, including DeepSeek-V3-Base, already show “aha”-style self-reflection before any RL, so the aha moment is not created by RL alone. [Certain]

DAPO: four fixes for long reasoning

DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) proposed four techniques. [Certain]

  1. Clip-higher. Use a larger upper clip than lower clip: \(\epsilon_\text{low} = 0.2\), \(\epsilon_\text{high} = 0.28\). With a symmetric clip, a token with probability 0.01 can only rise to 0.012 in one update, so rare tokens stay rare and the entropy collapses. A higher upper bound keeps exploration alive.
  2. Dynamic sampling. Drop groups whose accuracy is 0 or 1 and keep sampling until the batch is full (formula (70.7) explains why).
  3. Token-level policy-gradient loss. Average over all tokens in the batch instead of first over each answer:
\[ \mathcal{J}_{\text{DAPO}}(\theta) = \E \ \frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \min\!\big( \rho_{i,t} \hat A_{i,t},\ \operatorname{clip}(\rho_{i,t}, 1-\epsilon_\text{low}, 1+\epsilon_\text{high})\, \hat A_{i,t} \big) \tag{70.9} \]
Read the symbols
\(\sum_i |o_i|\)
the total number of answer tokens in the group (or batch); every token has the same weight
\(\epsilon_\text{low}\), \(\epsilon_\text{high}\)
the separate lower and upper clip ranges of clip-higher
\(\hat A_{i,t}\)
still the GRPO advantage (70.1), with the std division
  1. Overlong reward shaping. Answers cut off at the length limit are not simply scored 0 (they may contain good reasoning). Instead a soft length penalty is added near the limit:
\[ R_\text{length}(y) = \begin{cases} 0, & |y| \le L_\text{max} - L_\text{cache} \\ \dfrac{(L_\text{max} - L_\text{cache}) - |y|}{L_\text{cache}}, & L_\text{max} - L_\text{cache} \lt |y| \le L_\text{max} \\ -1, & |y| \gt L_\text{max} \end{cases} \tag{70.10} \]

With \(L_\text{max} = 16{,}384\) and \(L_\text{cache} = 4{,}096\), an answer of 14,000 tokens gets \((12{,}288 - 14{,}000)/4{,}096 = -0.418\). DAPO also removed the KL term, arguing that in long chain-of-thought training the policy should be allowed to move far from the start model. [Certain] With these changes DAPO reached 50 points on AIME 2024 from Qwen2.5-32B base, above DeepSeek-R1-Zero-Qwen-32B’s 47, in half the training steps. [Certain]

Worked example: per-token weights for one group

Group of \(G = 4\): two correct answers of 200 and 400 tokens, two wrong answers of 300 and 1200 tokens. Rewards \([1, 1, 0, 0]\): mean 0.5, std 0.5, so \(\hat A = [+1, +1, -1, -1]\) with GRPO’s formula (and \(\pm 0.5\) without the std).

GRPO (70.2), weight per token \(\hat A_i/(G|o_i|)\): \(+0.00125\), \(+0.000625\), \(-0.000833\), \(-0.000208\). Each answer’s total is \(\pm 0.25\), so the 1200-token wrong answer is penalized per token four times less than the 300-token one.

DAPO (70.9), weight per token \(\hat A/\sum|o_i| = 1/2100 = 0.000476\) for everyone. Totals: \(+0.095\), \(+0.190\), \(-0.143\), \(-0.571\). The long wrong answer now carries the largest total penalty.

Dr. GRPO (70.8) with \(L_\text{max} = 2048\) and \(\hat A = \pm 0.5\): weight per token \(0.5/(4 \times 2048) = 0.000061\). Totals: \(+0.0122\), \(+0.0244\), \(-0.0183\), \(-0.0732\): the same proportions as DAPO, at a fixed scale that does not change from batch to batch.

correct answerwrong answer
Figure 4. How the loss aggregation spreads the advantage over tokens, for the group in the worked example (answers A and B correct with 200 and 400 tokens; C and D wrong with 300 tokens and an adjustable length). Left: weight per token. Right: total weight of each answer (per-token weight times length). Dr. GRPO uses \(L_\text{max} = 2048\).
How to read Figure 4

In GRPO mode the right panel has four bars of equal size: every answer gets a total of ±0.25, whatever its length. Drag D’s length up: its per-token penalty shrinks, so the model loses little by making a wrong answer longer. Switch to DAPO or Dr. GRPO: all tokens have the same weight, and the total penalty of D grows with its length. Note that in DAPO the per-token weight still changes when D changes, because the denominator is the total token count of the batch.

Other changes you will meet

  • No KL, no reference model. TRL’s GRPO trainer uses \(\beta = 0\) by default, citing work showing the KL term is not essential; then the reference model is not loaded. [Likely] Removing the leash is reasonable when the reward is a strict verifier; it is risky with a learned reward model.
  • Std over the batch. Some libraries center per group but divide by the std of the whole batch, which removes the difficulty bias while keeping a fixed scale. TRL calls this scale_rewards="batch". [Likely]
  • Loss type as a switch. Current TRL exposes the aggregations as loss_type, with values such as "grpo", "dr_grpo" and "dapo", and in late 2026 its documented default is "dapo". [Likely] Defaults change between versions: always check which one you run.

The algorithm

Algorithm 1: GRPO with a verifiable reward (outcome supervision)
  1. input: policy \(\pi_\theta\) (from an SFT or base model); reference \(\pi_\text{ref} \leftarrow \pi_\theta\); verifier \(r(q, o)\); group size \(G\); \(\epsilon\), \(\beta\)
  2. for step \(= 1, 2, \dots\):
  3.   sample a batch of prompts \(q\); \(\pi_{\theta_\text{old}} \leftarrow \pi_\theta\)
  4.   for each \(q\): sample \(G\) answers \(o_1 \dots o_G \sim \pi_{\theta_\text{old}}(\cdot \mid q)\) at temperature 1 // the slow part
  5.     \(r_i \leftarrow r(q, o_i)\) // a program: parse, compare, run tests
  6.     \(\hat A_i \leftarrow (r_i - \operatorname{mean}(r)) / (\operatorname{std}(r) + \varepsilon)\) // (70.1); Dr. GRPO: no division
  7.     (optional, DAPO) drop the group if all \(r_i\) are equal; sample more prompts
  8.   store \(\log \pi_{\theta_\text{old}}\) of every answer token; if \(\beta \gt 0\), also \(\log \pi_\text{ref}\)
  9.   for each mini-batch (often one epoch): \(\rho \leftarrow \exp(\log \pi_\theta - \log \pi_{\theta_\text{old}})\)
  10.     per-token term \(\min(\rho \hat A, \operatorname{clip}(\rho)\hat A) - \beta\, \mathbb{D}\) // (70.2), (70.3)
  11.     aggregate over answer tokens (per answer, per token, or by a constant), negate, Adam step
  12.   (optional, as in R1) every few hundred steps set \(\pi_\text{ref} \leftarrow \pi_\theta\)
Algorithm 2: a math verifier with format check (reward 70.6)
  1. input: answer text \(o\), ground truth \(a^*\)
  2. \(r_\text{format} \leftarrow 1\) if \(o\) matches “<think> … </think> <answer> … </answer>” exactly once, else 0
  3. \(s \leftarrow\) text inside the answer tag (or inside the last \boxed{}); if none, return \(r_\text{format}\)
  4. normalize \(s\) and \(a^*\): strip spaces and units, remove trailing “.0”, parse fractions
  5. \(r_\text{acc} \leftarrow 1\) if they are equal as strings, or equal as symbolic or numeric values (with a small tolerance); else 0
  6. return \(r_\text{acc} + r_\text{format}\) // keep the format reward small or equal; never bigger

Watch Algorithm 1 run, line by line

Figure 5 runs Algorithm 1 for real in your browser on a model small enough to follow every number. There are four prompts, “2+3=”, “4+3=”, “6+2=” and “3+3=”. An answer has two tokens: a number from 0 to 9, then either the closing tag “</a>” or a period. The verifier is a tiny version of Algorithm 2 and reward (70.6): \(r = r_\text{acc} + r_\text{fmt}\), so 2 for the right number with a closed tag, 1 for a wrong number with a closed tag, 0 without the tag (the parser cannot read the answer). The reference model, and the starting policy, says the right number with probability 0.18, prefers the number one too high (0.30), and closes the tag with probability 0.40.

The policy’s parameters are its logits: 10 scores per prompt for the number and 2 per prompt for the tag. The gradient of one token’s term of (70.2) with respect to the logit \(z_k\) has a simple form, which is all the simulation needs:

\[ \frac{\partial}{\partial z_k} \Big[ \min\!\big(\rho \hat A, \operatorname{clip}(\rho)\hat A\big) - \beta\, \mathbb{D} \Big] = \Big( \mathbb{1}[\text{not clipped}]\, \hat A\, \rho \;+\; \beta \Big( \frac{\pi_\text{ref}(y)}{\pi_\theta(y)} - 1 \Big) \Big) \big( \mathbb{1}[k = y] - \pi_\theta(k) \big) \tag{70.11} \]
Read the symbols
\(y\)
the token that was sampled at this position; \(\pi_\theta(k)\) is the softmax probability of token \(k\) in the same context
\(\hat A \rho\)
the PPO part: it is switched off when the token is clipped
\(\beta(\pi_\text{ref}/\pi_\theta - 1)\)
the gradient of the \(k_3\) term (70.3): it pushes \(y\) up when the reference likes it more than the policy does, and down when it likes it less. It is never clipped, and it grows large for a token that the policy has made rare

Each mini-batch gradient \(g\) is then clipped to norm 0.1, as trainers clip the gradient norm, and the logits move by \(\alpha g\) (plain gradient ascent; real runs use Adam).

Algorithm 1, as the simulation runs it
  1. for step \(= 1, 2, \dots\) (at most 200):
  2.   \(\pi_{\theta_\text{old}} \leftarrow \pi_\theta\); for each prompt sample \(G\) answers \(o_i\)
  3.   \(r_i \leftarrow r_\text{acc} + r_\text{fmt}\) // (70.6)
  4.   \(\hat A_i \leftarrow (r_i - \bar r) / (s_r + 10^{-4})\), group mean and std // (70.1)
  5.   store \(\log \pi_{\theta_\text{old}}\) and \(\log \pi_\text{ref}\) of every token
  6.   for epoch \(= 1 \dots K\): shuffle, mini-batches of \(M\) answers
  7.     \(\rho_{i,t} \leftarrow \exp(\log \pi_\theta - \log \pi_{\theta_\text{old}})\)
  8.     \(\min(\rho \hat A, \operatorname{clip}(\rho)\hat A) - \beta \mathbb{D}\) per token // (70.2), (70.3)
  9.     mean over the tokens of each answer, then over answers
  10.     \(\theta \leftarrow \theta + \alpha\, g\), \(g\) norm-clipped // (70.11)
  11.   log accuracy, KL to \(\pi_\text{ref}\), clip fraction, dead groups
accuracy (exact)format rate (dashed), clip fractiondead groups (dashed): all rewards equal, not all rightKL to reference; purple ticks: \(\pi_\text{ref}\)right number
Figure 5. GRPO (Algorithm 1) running live on the toy model, 4 prompts per step. The highlighted line is the line that just ran; the panel shows its formula with the numbers of the followed answer. Answers are shown in their groups, with the group mean and std. Card edge: green for reward 2, orange for 1, red for 0; blue background: current mini-batch; red ratio: clipped token. Accuracy and KL are exact, computed from the probabilities. The same seed always gives the same run.
How to read Figure 5

Step to line 4 with seed 1. The group of “2+3=” has rewards [1, 0, 1, 0, 0, 0, 0, 1]: three closed tags, all with wrong numbers. This is the worked example of the first section, and the advantages are the same, +1.291 and −0.774 (the \(10^{-4}\) in the denominator changes the third digit). The group of “3+3=” has only one closed tag, so that answer gets +2.646. Step on to line 8 and watch the ratios leave 1 in the second mini-batch. Line 8 also shows the \(k_3\) term: tiny at first, because the policy is still close to the reference. Then choose Fast and Play: the format rate rises first, then accuracy. Try “No clip”: the ratio histogram grows a red bar at 2 and above. Try “Strong KL”: the KL stays low, but the accuracy is noisy, because the \(k_3\) gradient (70.11) is large whenever a token the policy has made rare is sampled.

Worked example: the first step of Figure 5 (seed 1, default settings)

The four groups have rewards “2+3=” [1, 0, 1, 0, 0, 0, 0, 1], “4+3=” [2, 0, 1, 0, 0, 1, 0, 1], “6+2=” [0, 0, 0, 1, 1, 0, 2, 1] and “3+3=” [0, 0, 0, 0, 0, 0, 0, 1]. No group has all rewards equal, so all four give signal.

For “2+3=”: mean 0.375, std 0.484, so the answer “6</a>” gets \(\hat A = 0.625 / 0.4842 = 1.291\). Both of its tokens get this advantage: the closed tag deserves it, the wrong number 6 does not. This is the credit problem of outcome rewards in its smallest form.

After step 1 (4 epochs × 2 mini-batches of 16 answers), \(\pi_\theta(6 \mid \text{2+3=})\) went from 0.302 to 0.466 and \(\pi_\theta(\text{</a>} \mid \text{2+3=})\) from 0.401 to 0.564. 70% of the token updates were clipped: with \(\alpha = 20\) the steps are large, and the clip is working hard. The exact accuracy is 0.107 after step 1, 0.742 after 10 steps and 0.942 after 20, with a KL to the reference of 2.295 nats.

Two runs side by side: clip, KL weight and group size

Figure 6 runs the same simulation twice with the same seed and one different setting. Three comparisons are worth seeing.

The clip. With \(\beta = 0\) and no clip (\(\epsilon = 100\)), one step can move a prompt very far. For seed 4, two prompts lock onto wrong numbers with closed tags in the first steps. Each then gets the same reward 1 for every answer in its group: the std is 0, the advantage is 0 (formula 70.1), and the group teaches nothing. The accuracy sits at 0.50 until step 38 and reaches 0.994 only by step 150, when rare right answers finally get sampled. With \(\epsilon = 0.2\), the run is at 0.973 after 10 steps. The curve of dead groups (all rewards equal, but not all right) shows the cause directly: 46% of the groups are dead in the first 40 steps of run B, against under 1% in run A.

The KL weight. With \(\beta = 0.5\) the KL to the reference stays far lower (0.348 nats at step 150 for seed 4, against 2.605 with \(\beta = 0\)), but the accuracy is low and noisy (0.193 at step 150). The \(k_3\) gradient pushes rare tokens back hard. Real runs use small values such as 0.04 or 0.001, and many recent recipes drop the term.

The group size. With \(\beta = 0.04\) and seed 4, \(G = 4\) moves between 0.79 and 0.90 after step 30, while \(G = 16\) reaches 0.996. A small group more often has all rewards equal (13% dead groups in the first 40 steps with \(G = 4\), none with \(G = 16\), as formula 70.7 predicts), and its mean is a noisier baseline, while the KL term pulls back at every step. [Likely]

Run ARun B (dashed)purple ticks: \(\pi_\text{ref}\)
Figure 6. Two GRPO runs of the Figure 5 simulation with the same seed and one different setting, 150 steps each (\(\alpha = 20\), \(K = 4\), \(M = 16\)). Run A is solid blue, run B dashed red. The third chart is the share of dead groups in each step: groups whose rewards were all equal but not all 2 (a solved group also gives no signal, but it is not a problem). The bottom panels show each run’s final policy.
How to read Figure 6

Press “Run to the end” with the clip comparison and seed 4. Run B’s accuracy is flat at 0.50 until step 38, and in the same steps its dashed dead-group curve stays at 0.5: two of the four groups are stuck. Run A climbs in about 10 steps. Change the seed: in some seeds run B is lucky and fast, which is the point: without the clip, the outcome depends on luck in the first few batches. Choose the group comparison: \(G = 4\) has dead groups in early steps, \(G = 16\) has none.

The PyTorch code below runs GRPO end to end on a toy task in a few seconds on a CPU. The “prompt” is two digits \(a, b \in \{0, \dots, 4\}\). The model writes up to 8 tokens from 11 (digits 0 to 9 and an end token). The verifier gives reward 1 if the digit just before the first end token equals \(a + b\). There is no reward model and no critic; the reference model is a frozen copy of the start policy. The flag loss_type switches between the GRPO aggregation (70.2) and Dr. GRPO (70.8). It is deterministic (fixed seed, CPU).

import copy, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
EOS, V, T = 10, 11, 8            # tokens 0-9 are digits, 10 ends the answer; at most 8 tokens
G, P = 8, 16                     # answers per prompt, prompts per step

class TinyLM(nn.Module):         # logits from (prompt a, prompt b, previous token, position)
    def __init__(s):
        super().__init__()
        s.a, s.b = nn.Embedding(5, 32), nn.Embedding(5, 32)
        s.tok, s.pos = nn.Embedding(V + 1, 32), nn.Embedding(T, 32)
        s.mlp = nn.Sequential(nn.Linear(32, 64), nn.Tanh(), nn.Linear(64, V))
    def forward(s, a, b, prev):  # a, b: (N,), prev: (N, L)
        h = s.a(a)[:, None] + s.b(b)[:, None] + s.tok(prev) + s.pos.weight[None, :prev.size(1)]
        return s.mlp(h)

def verify(a, b, y):             # verifiable reward: 1 if the digit before EOS equals a + b
    r = torch.zeros(len(y))
    for i in range(len(y)):
        toks = y[i].tolist()
        if EOS in toks:
            k = toks.index(EOS)
            if k > 0 and toks[k - 1] == int(a[i] + b[i]): r[i] = 1.0
    return r

def prev_of(y):                  # previous token at each position; V = "start"
    return torch.cat([torch.full((y.size(0), 1), V), y[:, :-1]], 1)

def token_logp(model, a, b, y):
    lp = F.log_softmax(model(a, b, prev_of(y)), -1)
    return lp.gather(-1, y[..., None]).squeeze(-1)

def run(loss_type, steps=150, beta=0.04, eps=0.2):
    torch.manual_seed(0)
    policy = TinyLM(); ref = copy.deepcopy(policy).eval()
    for p in ref.parameters(): p.requires_grad_(False)
    opt = torch.optim.Adam(policy.parameters(), lr=3e-3)
    for step in range(steps):
        a = torch.randint(0, 5, (P,)).repeat_interleave(G)     # each prompt G times
        b = torch.randint(0, 5, (P,)).repeat_interleave(G)
        with torch.no_grad():                                   # 1) rollout
            y = torch.full((P * G, T), EOS)
            for t in range(T):
                logits = policy(a, b, prev_of(y))[:, t]
                y[:, t] = torch.distributions.Categorical(logits=logits).sample()
            done = (y == EOS).long().cumsum(1)                  # mask: tokens up to the first EOS
            mask = ((done == 0) | ((done == 1) & (y == EOS))).float()
            r = verify(a, b, y)                                 # 2) verifier, no reward model
            rg = r.view(P, G)                                   # 3) group-relative advantage
            adv = rg - rg.mean(1, keepdim=True)
            if loss_type == "grpo": adv = adv / (rg.std(1, keepdim=True) + 1e-4)
            adv = adv.view(-1, 1)
            old_lp = token_logp(policy, a, b, y)
        for _ in range(2):                                      # 4) two gradient steps per rollout
            lp = token_logp(policy, a, b, y)
            ratio = torch.exp(lp - old_lp)
            pg = torch.min(ratio * adv, ratio.clamp(1 - eps, 1 + eps) * adv)
            d = token_logp(ref, a, b, y) - lp                   # k3 estimator of KL(pi || ref)
            kl = torch.exp(d) - d - 1
            per_tok = -(pg - beta * kl) * mask
            if loss_type == "grpo":                             # mean over each answer, then over answers
                loss = (per_tok.sum(1) / mask.sum(1)).mean()
            else:                                               # Dr. GRPO: divide by a constant
                loss = per_tok.sum(1).mean() / T
            opt.zero_grad(); loss.backward(); opt.step()
        if step % 30 == 0 or step == steps - 1:
            right, wrong = r == 1, r == 0
            ml = lambda m: mask.sum(1)[m].mean().item() if m.any() else float("nan")
            print(f"{loss_type:7s} step {step:3d}  acc {r.mean():.3f}  len right {ml(right):.2f}"
                  f"  len wrong {ml(wrong):.2f}  no-signal groups {(rg.std(1) == 0).float().mean():.2f}")

for lt in ["grpo", "dr_grpo"]: run(lt)
# Output (CPU, torch 2.8):
# grpo    step   0  acc 0.031  len right 4.00  len wrong 5.27  no-signal groups 0.75
# grpo    step  30  acc 0.367  len right 2.40  len wrong 3.05  no-signal groups 0.25
# grpo    step  60  acc 0.516  len right 2.47  len wrong 3.39  no-signal groups 0.38
# grpo    step  90  acc 0.758  len right 2.74  len wrong 3.61  no-signal groups 0.44
# grpo    step 120  acc 0.594  len right 2.36  len wrong 4.44  no-signal groups 0.62
# grpo    step 149  acc 0.703  len right 2.98  len wrong 5.79  no-signal groups 0.62
# dr_grpo step   0  acc 0.031  len right 4.00  len wrong 5.27  no-signal groups 0.75
# dr_grpo step  30  acc 0.234  len right 3.90  len wrong 4.50  no-signal groups 0.19
# dr_grpo step  60  acc 0.289  len right 3.51  len wrong 4.95  no-signal groups 0.19
# dr_grpo step  90  acc 0.617  len right 3.81  len wrong 4.20  no-signal groups 0.38
# dr_grpo step 120  acc 0.680  len right 3.70  len wrong 6.44  no-signal groups 0.56
# dr_grpo step 149  acc 0.656  len right 3.89  len wrong 5.86  no-signal groups 0.50

What the output shows. Accuracy rises from 3.1% (random guessing) to about 65% to 75% for both losses, with no critic and no reward model. At step 0, 75% of the groups have no signal: the task is hard for a random policy, so most groups are all wrong. Later, 50% to 62% of groups have no signal again, now because many prompts are always solved; this is the situation dynamic sampling is designed for. The two runs differ in accuracy and length from step to step, but with only 16 prompts per step this toy is too noisy to show the length bias clearly; do not read the length columns as evidence for or against Dr. GRPO. The hands-on version of this loop on a real small model is session 77.

Where this is used

SettingVerifiable rewardNotes
Math reasoning models (DeepSeek-R1, many open “-Zero” reproductions)Final answer equals the ground truth, plus a format checkGRPO, Dr. GRPO or DAPO; groups of 8 to 64; long answers
Code generation and agents for software tasksUnit tests pass in a sandboxSandboxing and hidden tests are needed against test hacking [Likely]
Instruction following (Tülu 3 RLVR)Programmatic constraint checks (length, keywords, format)Tülu 3 used PPO, not GRPO: RLVR is about the reward, not the algorithm
Structured output and tool callsThe output parses as JSON or matches a schema; the tool call runsOften mixed with accuracy rewards [Likely]
VLM reasoning and grounding (session 190)Exact answer for visual math or counting; IoU of a predicted boxSame GRPO loop; image tokens are part of the prompt and get no loss [Likely]
Video reasoning and video QA (Video-R1)Exact-match answers on video questionsVideo-R1 proposes T-GRPO, a variant that rewards use of temporal information [Certain]
Video temporal grounding and key-moment detection (VideoChat-R1)Temporal IoU of a predicted time span with the true span; tracking overlapVideoChat-R1 reports large gains on temporal grounding and tracking with rule-based rewards [Certain]; see session 205
Video classification and action recognition by a Video-LLMPredicted label equals the true labelEasy to verify, but labels are easy to guess: random accuracy may be high [Speculation]

Common mistakes

  • All-equal groups treated as signal. Dividing by a std of 0 without an \(\varepsilon\) gives NaN; with an \(\varepsilon\) it gives 0, which is correct but wastes the samples. Log the fraction of no-signal groups and filter them (dynamic sampling) or make the prompt set harder.
  • Mixing up group statistics. The mean and std must be taken over the \(G\) answers of the same prompt. Computing them over the whole batch (mixing prompts) gives a different method: easy prompts then look good and hard prompts look bad, whatever the answers.
  • Not knowing your aggregation. “Mean per answer, then mean over answers” (original GRPO), “mean over all tokens” (DAPO) and “sum divided by a constant” (Dr. GRPO) have different length behavior. Check the loss_type of your library.
  • A format reward that dominates. If the format reward is larger than the accuracy reward, the model first learns tags and can stop there. Keep it equal or smaller, and log the two rewards separately.
  • A loose verifier. Extracting “the last number in the text” rewards answers that list many numbers. Multiple-choice answers can be guessed at 25%. Use strict extraction, numeric answers and hidden tests.
  • Truncated answers scored as normal. An answer cut off at the length limit has no final answer; scoring it 0 together with real wrong answers punishes long reasoning blindly. Mask it or use overlong shaping (70.10).
  • Loss on the prompt. Only answer tokens are actions. Mask prompt and padding tokens in the policy loss and in the KL.
  • Mismatched log-probabilities. If the inference engine (for example vLLM) and the trainer compute slightly different log-probabilities, the ratio is not 1 at the start, and the clip acts on noise. Recompute \(\log \pi_{\theta_\text{old}}\) in the trainer, or check that the ratio starts at 1.000. [Likely]

Interview questions

How does GRPO compute the advantage?basic
For each prompt it samples a group of \(G\) answers from the current policy and scores each one. The advantage of answer \(i\) is \((r_i - \operatorname{mean}(r))/\operatorname{std}(r)\) over the group, and every token of the answer gets this same value (with outcome rewards). The group mean is a per-prompt baseline, estimated from samples instead of being predicted by a critic. Example: rewards \([1,0,0,1,0,0,0,1]\) give +1.291 for the correct answers and −0.775 for the wrong ones (std dividing by \(G\)).
Why does GRPO not need a critic, and what does it lose by dropping it?core
The critic’s job in PPO is to provide a baseline, the expected reward from a state. GRPO gets a baseline for the whole answer by averaging the rewards of several answers to the same prompt. It saves a trained model: for a 7B policy, 112 GB of critic state (16 bytes per parameter), plus its forward and backward passes, and the trouble of training a value model on long text. It loses per-token credit assignment (all tokens share one advantage), needs \(G\) samples per prompt, and gets no gradient from prompts where all answers score the same.
What is a verifiable reward? Give examples and one weakness.basic
A reward computed by a program from the answer and a ground truth, with no learned model: the final math answer equals the key; code passes unit tests; the output matches a format or a schema; an instruction constraint (word count, keyword) holds; a predicted box or time span has high IoU with the truth. Weakness: it checks only the outcome, so the policy can exploit gaps (loose answer extraction, guessing multiple-choice answers, code that special-cases visible tests), and it gives no credit to good reasoning that ends in a small error.
What happens when all answers in a group get the same reward? How often does it happen?core
All advantages are zero (the std is zero; with an \(\varepsilon\) in the denominator the result is 0), so the group contributes no policy gradient, only the KL term. With 0/1 rewards and solve rate \(p\), the chance of a useful group is \(1 - p^G - (1-p)^G\): for \(p = 0.05\), 33.7% at \(G = 8\) and 96.3% at \(G = 64\). As training succeeds, more prompts become always solved. Fixes: larger groups, filtering prompts by difficulty, or DAPO’s dynamic sampling, which drops groups with accuracy 0 or 1 and samples more.
Explain the length bias in the original GRPO loss.advanced
GRPO averages the per-token terms inside each answer (divides by \(|o_i|\)) and then averages over answers. So every answer gets the same total weight \(\hat A_i/G\), and the per-token weight is \(\hat A_i/(G|o_i|)\). For a wrong answer, a longer answer spreads the same total penalty over more tokens, so each token is punished less; for correct answers, short ones are favored. Liu et al. argue this lets wrong answers grow longer and inflates the length growth seen in R1-Zero-style runs. Dr. GRPO divides the token sum by a constant instead; DAPO averages over all tokens in the batch.
What is the difficulty bias from the std normalization?advanced
Dividing by the group std scales each prompt by \(1/\sigma\). With 0/1 rewards \(\sigma = \sqrt{p(1-p)}\), which is small for prompts that are almost always solved or almost always failed. So those prompts get larger advantages: one correct answer in 16 gets \(\hat A = 3.873\), against 1.0 when half are right. The total push per answer is \(2\sqrt{p(1-p)}\) with the std and \(2p(1-p)\) without it. Dr. GRPO removes the std; another fix is to divide by the batch std, which keeps a fixed scale without per-prompt reweighting.
What did DeepSeek-R1-Zero show, and why was DeepSeek-R1 still needed?core
R1-Zero applied GRPO with only rule-based accuracy and format rewards directly to DeepSeek-V3-Base, with no SFT. AIME 2024 pass@1 rose from 15.6% to 71.0% (86.7% with majority voting over 64), response length grew by itself, and the model started to re-check its work (“aha moment”). But the outputs were hard to read and mixed languages. DeepSeek-R1 added a small cold-start SFT on long chain-of-thought data, a language-consistency reward, a rejection-sampling SFT round (about 600k reasoning and 200k other examples) and a final RL stage for all tasks, reaching 79.8% on AIME 2024.
GRPO has a PPO-style clip. When does it actually do anything?core
Only when the policy takes more than one gradient step on the same samples. On the first step after sampling, \(\pi_\theta = \pi_{\theta_\text{old}}\), every ratio is 1 and the clip is inactive; the loss is plain policy gradient with the group advantage. DeepSeekMath made a single update per batch of samples, so its clip never acted. DeepSeek-R1 split each batch of 8,192 outputs into 16 mini-batches, so later mini-batches are slightly off-policy and the clip matters. DAPO’s clip-higher (0.2 below, 0.28 above) exists because the clip can also block rare tokens from rising.
Where does the KL term go in GRPO versus PPO-RLHF, and why do many recipes now drop it?advanced
PPO-RLHF subtracts \(\beta \log(\pi/\pi_\text{ref})\) inside the per-token reward, so the critic and GAE see it. GRPO has no critic, so it adds the \(k_3\) estimator \(x - \log x - 1\) (with \(x = \pi_\text{ref}/\pi_\theta\)) directly to the loss, averaged like the policy term. The leash mainly protects against exploiting a learned reward model. With a strict verifier there is less to exploit, and long reasoning needs the policy to move far from the start, so DAPO and others set \(\beta = 0\), which also removes the reference model from memory. DeepSeek-R1 kept a small \(\beta = 0.001\) and refreshed the reference every 400 steps.
Design an RLVR setup for training a Video-LLM to find key moments in videos.advanced
Prompt: the video frames plus “give the start and end time of the moment where X happens”. Format reward: the answer is two numbers in a fixed tag, start below end, inside the video length. Accuracy reward: temporal IoU between the predicted and true span (a continuous value in [0, 1], which gives more signal than 0/1). Sample \(G = 8\) answers per video, group-normalize, use a Dr. GRPO or DAPO-style loss, mask the video and prompt tokens. Watch for hacks: always predicting the whole video gives a non-zero IoU, so consider a penalty for very long spans or reward only above a threshold. VideoChat-R1 reports large gains on temporal grounding with rule-based rewards of this kind; the exact reward design here is an example, not their recipe.

Exercises

  1. A group of 4 answers has rewards \([1, 0, 0, 0]\). Compute the GRPO advantages (std dividing by \(G\)), the mean-only advantages and the RLOO advantages.
  2. With rewards \([2, 1, 0, 2]\) (accuracy plus format), compute the group mean, the std (dividing by \(G\)) and the four GRPO advantages.
  3. For \(p = 0.1\), compute the chance that a group has signal for \(G = 4\) and \(G = 16\).
  4. Under the original GRPO aggregation, a correct answer of 100 tokens and a correct answer of 500 tokens both have \(\hat A = 1\), with \(G = 8\). What is the weight per token of each? What is it under Dr. GRPO with \(L_\text{max} = 1000\) (same \(\hat A\))?
  5. Compute the memory of model state for GRPO on a 32B policy with \(\beta = 0.001\), and with \(\beta = 0\), using (70.5).
  6. With DAPO’s overlong shaping, \(L_\text{max} = 16{,}384\), \(L_\text{cache} = 4{,}096\): what is the length reward for answers of 10,000, 14,336 and 17,000 tokens?
Answerscheck
1. Mean 0.25, std \(\sqrt{0.25 \times 0.75} = 0.433\). GRPO: \(0.75/0.433 = +1.732\) and \(-0.25/0.433 = -0.577\) for the three wrong ones. Mean only: \(+0.75\), \(-0.25\). RLOO: \(\tfrac{4}{3} \times 0.75 = +1.0\) and \(\tfrac{4}{3} \times (-0.25) = -0.333\) (the correct answer’s baseline is the mean of the other three, 0; a wrong answer’s baseline is \(1/3\)). 2. Mean 1.25; deviations 0.75, −0.25, −1.25, 0.75; variance \((0.5625 + 0.0625 + 1.5625 + 0.5625)/4 = 0.6875\), std 0.829; advantages \(+0.905, -0.302, -1.508, +0.905\). 3. \(G = 4\): \(1 - 0.1^4 - 0.9^4 = 1 - 0.0001 - 0.6561 = 0.3438\). \(G = 16\): \(1 - 0.9^{16} - 10^{-16} = 0.8147\). 4. GRPO: \(1/(8 \times 100) = 0.00125\) and \(1/(8 \times 500) = 0.00025\); the short answer gets 5 times more per token, and both get the same total 0.125. Dr. GRPO: \(1/(8 \times 1000) = 0.000125\) per token for both; totals 0.0125 and 0.0625. 5. \(16 \times 32 + 2 \times 32 = 512 + 64 = 576\) GB with the reference; 512 GB with \(\beta = 0\). 6. 10,000 is below \(12{,}288\): 0. 14,336: \((12{,}288 - 14{,}336)/4{,}096 = -0.5\). 17,000 is above 16,384: −1.

Key takeaways

  • GRPO samples a group of \(G\) answers per prompt and uses the group-normalized reward (70.1) as the advantage of every token of an answer. The group mean replaces the critic.
  • The loss is PPO’s clipped objective plus a \(k_3\) KL term to a reference (70.2), (70.3). No critic halves the model-state memory of PPO for a 7B model (126 GB against 252 GB), and \(\beta = 0\) also removes the reference.
  • RLVR means rewards from programs: answer checks, unit tests, format and constraint checks, IoU. It resists reward-model hacking but checks only the outcome, so verifiers must be strict.
  • Groups where all answers score the same give no gradient; the chance of signal is \(1 - p^G - (1-p)^G\) (70.7). Dynamic sampling filters such groups.
  • DeepSeekMath introduced GRPO with a learned reward model; DeepSeek-R1-Zero used it with rule-based rewards on a base model (AIME 2024: 15.6% → 71.0%); DeepSeek-R1 added cold-start SFT, rejection sampling and more RL.
  • In the live toy run (Figures 5 and 6), the format reward alone pushes up wrong numbers with closed tags; without the clip a prompt can lock onto such an answer, its group rewards become all equal, and it stops learning until a rare right answer is sampled.
  • The original loss has a length bias (\(1/|o_i|\)) and a difficulty bias (std). Dr. GRPO removes both (70.8); DAPO uses a token-level loss, clip-higher, dynamic sampling and overlong shaping (70.9), (70.10).

Key formulas

\[ \hat A_{i} = \frac{r_i - \operatorname{mean}(r_{1..G})}{\operatorname{std}(r_{1..G})}, \qquad \mathbb{D}_{i,t} = \frac{\pi_\text{ref}}{\pi_\theta} - \log\frac{\pi_\text{ref}}{\pi_\theta} - 1 \]
\[ \mathcal{J}_{\text{GRPO}} = \E\, \frac{1}{G}\sum_{i} \frac{1}{|o_i|}\sum_{t} \Big\{ \min\big(\rho_{i,t}\hat A_{i},\ \operatorname{clip}(\rho_{i,t}, 1 \pm \epsilon)\hat A_{i}\big) - \beta\, \mathbb{D}_{i,t} \Big\} \]
\[ \text{Dr. GRPO: } \frac{1}{G L_\text{max}} \sum_i \sum_t (\cdot), \ \hat A_i = r_i - \bar r; \qquad \text{DAPO: } \frac{1}{\sum_i |o_i|} \sum_i \sum_t (\cdot) \]
\[ P(\text{signal}) = 1 - p^G - (1-p)^G, \qquad M_\text{GRPO} \approx 16 N_\pi + 2 N_\text{ref}\,\mathbb{1}[\beta \gt 0] \]

References for this session

  1. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
  2. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  3. Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective
  4. Raschka, Build a Reasoning Model (From Scratch), chapter 6: Training reasoning models with RL
  5. Raschka, Build a Reasoning Model (From Scratch), chapter 7: Improving GRPO
  6. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
  7. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  8. Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective
  9. Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  10. Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
  11. Schulman et al. (2017). Proximal Policy Optimization Algorithms
  12. Schulman (2020). Approximating KL Divergence (blog)
  13. Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  14. Hugging Face (2026). TRL documentation: GRPO Trainer
  15. Feng et al. (2025). Video-R1: Reinforcing Video Reasoning in MLLMs
  16. Li et al. (2025). VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning
  17. Raschka, Build a Reasoning Model (From Scratch), chapter 6: Training reasoning models with RL
  18. Raschka, Build a Reasoning Model (From Scratch), chapter 7: Improving GRPO