GRPO and RL with verifiable rewards (RLVR)
GRPO (Group Relative Policy Optimization) removes PPO's critic: it samples a group of answers for each prompt and uses how much better each answer is than its group as the advantage. Combined with rewards that a program can check (correct math answers, passing unit tests, format rules), it became the recipe behind DeepSeek-R1 and most open reasoning models. This session builds the objective, counts the memory it saves, follows the path from DeepSeekMath to DeepSeek-R1, and explains the known biases and their fixes in Dr. GRPO and DAPO.
What you will learn
- How the group-relative advantage works, why it sums to zero, and why a group where every answer gets the same reward teaches nothing
- The full GRPO objective: the PPO clip per token, the k3 KL estimator, and outcome versus process supervision
- What dropping the critic saves in memory (126 GB against 252 GB of model state for a 7B policy) and what it costs in credit assignment
- What a verifiable reward is, how math, code, format and IoU checkers are built, and how a policy can still exploit a loose verifier
- The path from DeepSeekMath to DeepSeek-R1-Zero and DeepSeek-R1, with the real hyperparameters and results
- The length bias and the difficulty bias of the original loss, and the fixes in Dr. GRPO and DAPO (token-level loss, clip-higher, dynamic sampling, overlong shaping)
- A runnable GRPO loop in PyTorch with a verifier, a reference model and a switch between GRPO and Dr. GRPO aggregation
Why this matters
In session 66 we saw that classic RLHF keeps four models on the GPUs: the policy, a frozen reference, a reward model and a critic. Two of them are trained. The critic alone costs a full model of weights, gradients and optimizer state, and it is hard to train well on long text. GRPO (Group Relative Policy Optimization, from the DeepSeekMath paper, Shao et al., 2024) removes the critic. For each prompt it samples a group of answers, scores them all, and uses “how much better than the other answers in my group” as the advantage. [Certain]
The second idea of this session is the reward itself. For math, code and many formatted tasks we do not need a learned reward model: a short program can check the answer. The final number matches, the unit tests pass, the output is valid JSON. This is RLVR (reinforcement learning with verifiable rewards), a name introduced by the Tülu 3 paper (Lambert et al., 2024). [Certain] GRPO plus rule-based rewards is the recipe of DeepSeek-R1-Zero, which learned long step-by-step reasoning from a base model with no supervised reasoning data. [Certain] Today almost every open reasoning model is trained with GRPO or a close variant. [Likely]
- One less model. No critic: one trained model instead of two. With the KL term turned off, no reference model either.
- Rewards you can trust. A checker cannot be flattered by long or confident text the way a learned reward model can. It can still be fooled, and we will see how.
- Simple code. The advantage is a mean and a standard deviation over a few numbers. The rest is the PPO clipped loss you already know.
- Known problems. The original GRPO loss has a bias toward long wrong answers and a hidden weighting by question difficulty. Dr. GRPO and DAPO fixed these in 2025, and most libraries now use the fixed versions. [Likely]
One prompt, many answers: the group baseline
Recall the policy gradient from session 64: push up the log-probability of each sampled answer in proportion to its advantage, the reward minus a baseline. A good baseline is the expected reward of the prompt: “what score do I usually get here?”. PPO learns this baseline with a critic network (session 65). GRPO estimates it directly. For one prompt \(q\), sample \(G\) answers \(o_1, \dots, o_G\) from the current policy and score each one, giving rewards \(r_1, \dots, r_G\). The mean of the group is a sample estimate of the prompt’s expected reward. Then
- Read the symbols
- \(q\)
- one prompt (in the papers, a “question”)
- \(G\)
- the group size: how many answers we sample for the same prompt; DeepSeekMath used 64, DeepSeek-R1 used 16
- \(o_i\), \(r_i\)
- the \(i\)-th sampled answer (“output”) and its scalar reward
- \(\tilde r_i\)
- the normalized reward: centered by the group mean, scaled by the group standard deviation
- \(\hat A_{i,t}\)
- the advantage of token \(t\) in answer \(i\); with one reward per answer (“outcome supervision”), every token of the answer gets the same value
Three facts follow at once. First, the advantages of a group sum to zero: some answers are pushed up and the others are pushed down by the same total amount. Second, if all answers get the same reward (all right or all wrong), every advantage is zero, and the prompt teaches nothing. In code a small \(\varepsilon\) is added to the denominator so that this case gives 0 and not a division by zero. Third, the baseline is per prompt: a hard question where 1 answer in 8 is right and an easy one where 7 in 8 are right each get their own mean. This is exactly what a critic tries to learn, but GRPO gets it for free from the samples.
Session 64 compared this with two cousins. RLOO (REINFORCE leave-one-out) uses the mean of the other \(G-1\) answers as the baseline, which equals \(\tfrac{G}{G-1}(r_i - \bar r)\) and is unbiased. Dividing by the standard deviation is GRPO’s own choice. It makes the advantages unit-free, so the learning rate does not depend on the reward scale. But it also changes how much each prompt counts, and we will see that this is one of GRPO’s known problems.
The verifier gives 1 for a correct final answer and 0 otherwise. Rewards: \([1, 0, 0, 1, 0, 0, 0, 1]\). The mean is \(3/8 = 0.375\).
The standard deviation, dividing by \(G\) (the “population” version), is \(\sqrt{0.375 \times 0.625} = 0.4841\). So each correct answer gets \(\hat A = 0.625 / 0.4841 = 1.291\) and each wrong answer gets \(-0.375 / 0.4841 = -0.775\). Check: \(3 \times 1.291 - 5 \times 0.775 = 0\).
PyTorch’s std() divides by \(G - 1\) by default, which gives \(0.5175\) and advantages \(1.208\) and \(-0.725\). Libraries differ here; the ratio between right and wrong is the same, only the scale changes.
Without the division (mean only, as in Dr. GRPO): \(+0.625\) and \(-0.375\). RLOO: \(\tfrac{8}{7} \times 0.625 = 0.714\) and \(\tfrac{8}{7} \times (-0.375) = -0.429\).
How to read Figure 1
Start with 3 of 8 correct: correct answers get +1.291, wrong ones −0.775. Click until only one answer is correct: its bar grows to +2.646, because a rare success on a hard question is a big surprise. Make all answers wrong, or all correct: every bar is zero and the readout says the group has no signal. Switch to “Mean only”: bars become \(r_i - \bar r\) and no longer grow for rare successes. Switch to RLOO: the same shape as mean only, scaled by \(G/(G-1)\).
The GRPO objective
With the advantage in hand, GRPO reuses the PPO clipped objective from session 66 token by token, and adds a KL penalty to a frozen reference model directly in the loss (not in the reward, as PPO-RLHF does). DeepSeekMath writes the objective, to be maximized, as [Certain]
- Read the symbols
- \(\rho_{i,t}\)
- the probability ratio \(\pi_\theta(o_{i,t} \mid q, o_{i,\lt t}) / \pi_{\theta_\text{old}}(o_{i,t} \mid q, o_{i,\lt t})\) of token \(t\) of answer \(i\), as in PPO
- \(\pi_{\theta_\text{old}}\)
- the policy that generated the group; its token log-probabilities are stored once
- \(|o_i|\)
- the number of tokens in answer \(i\); the inner sum is an average over the tokens of one answer
- \(\epsilon\)
- the clip range, usually 0.2
- \(\beta\)
- the KL coefficient; 0.04 in DeepSeekMath, 0.001 in DeepSeek-R1
- \(\mathbb{D}_{i,t}\)
- a per-token estimate of the KL divergence from the policy to the reference model, formula (70.3)
Notice the two averages. First over the \(|o_i|\) tokens of one answer, then over the \(G\) answers. This “mean of means” looks harmless, but it is the source of the length bias discussed later. The DeepSeek-R1 paper writes the objective without the \(1/|o_i|\) factor. [Certain]
The KL term uses Schulman’s \(k_3\) estimator (we met it in session 66), evaluated on the sampled token: [Certain]
- Read the symbols
- \(\pi_\text{ref}\)
- the frozen reference model, usually the SFT model the run started from
- \(\cdot\)
- short for the context \(q, o_{i,\lt t}\)
- \(\mathbb{D}_{i,t}\)
- never negative (the function \(x - \log x - 1\) has its minimum 0 at \(x = 1\)), and its average over samples from \(\pi_\theta\) is the true KL
For example, a token the reference finds half as likely as the policy (\(x = 0.5\)) costs \(0.5 + 0.693 - 1 = 0.193\); a token the reference finds twice as likely (\(x = 2\)) costs \(2 - 0.693 - 1 = 0.307\).
Outcome supervision and process supervision
Formula (70.1) is outcome supervision: one reward at the end of each answer. DeepSeekMath also tried process supervision, where a process reward model scores each reasoning step. The step rewards of all answers in the group are normalized together (\(\mathbf{R}\) below is the set of all step rewards of the group), and a token’s advantage is the sum of the normalized rewards of the steps that end at or after it: [Certain]
Here \(r_i^{(j)}\) is the reward of step \(j\) of answer \(i\), and \(\text{index}(j)\) is the token where step \(j\) ends. In DeepSeekMath, process supervision did slightly better than outcome supervision. [Certain] DeepSeek-R1 later went back to outcome rewards from rules, and listed process reward models among its unsuccessful attempts: steps are hard to define, step correctness is hard to judge, and a learned step scorer invites reward hacking. [Certain] Process rewards are covered in session 73.
No critic: what we save and what we lose
Using the memory rule from session 38 (about 16 bytes per trained parameter with mixed-precision AdamW, 2 bytes per frozen bf16 parameter), GRPO needs
- Read the symbols
- \(N_\pi\), \(N_\text{ref}\)
- parameters of the policy (trained) and of the reference (frozen, same size)
- \(\mathbb{1}[\beta \gt 0]\)
- 1 if the KL term is used, else 0: with \(\beta = 0\) the reference model is not loaded at all
- no reward-model term
- with verifiable rewards, the reward is a program on the CPU; with a learned reward model, add \(2 N_\text{RM}\)
PPO with a 7B reward model and a 7B critic (from session 66): \(16 \times 7 + 16 \times 7 + 2 \times 7 + 2 \times 7 = 252\) GB of model state.
GRPO with a verifier and a KL term: \(16 \times 7 + 2 \times 7 = 126\) GB, half of PPO. GRPO with \(\beta = 0\): 112 GB, the same as plain SFT of the model.
The KV cache for generation is not in these numbers. With groups of 16 answers of up to 32k tokens, generation memory is often the real limit, which is why rollouts usually run in a separate inference engine such as vLLM (session 75). [Likely]
What do we lose? Three things.
- Per-token credit. A critic can say “the answer went wrong at token 140”. GRPO gives every token of an answer the same advantage. The good first steps of a wrong answer are pushed down together with the mistake. Over many groups this averages out, because the same good steps also appear in correct answers. [Likely]
- More samples per prompt. GRPO needs \(G\) answers per prompt to estimate the baseline. PPO can learn from one answer per prompt. But generation is batched, and for reasoning tasks several samples per prompt are useful anyway. [Likely]
- No learning from uniform groups. A prompt where all \(G\) answers get the same reward produces zero gradient. The next section measures how often this happens.
| PPO-RLHF (session 66) | GRPO with verifiable rewards | |
|---|---|---|
| Advantage | GAE from a learned critic, per token | group-normalized reward, same for all tokens of an answer |
| Trained models | policy and critic | policy only |
| Frozen models | reference and reward model | reference (optional); the reward is a program |
| KL to the reference | inside the per-token reward | a separate loss term, \(k_3\) estimator (70.3), or none |
| Samples per prompt | usually 1 | a group: 8 to 64 |
| Typical use | chat alignment with human preferences | math, code, formatted output, reasoning |
Verifiable rewards (RLVR)
A verifiable reward is a reward computed by a deterministic program from the answer and some ground truth, with no learned model in between. The Tülu 3 paper defines RLVR as giving a constant reward when a completion is verified correct and 0 otherwise; they used a reward of 10 for a correct answer and trained with PPO on GSM8K, MATH and instruction-following constraints (IFEval-style). [Certain] So RLVR is a kind of reward, not an algorithm. It works with PPO, GRPO, RLOO or any other policy-gradient method.
| Task | Verifier | Typical traps |
|---|---|---|
| Math with a final answer | Extract the answer (for example from \boxed{...} or an answer tag), normalize it, compare with the ground truth; symbolic checkers (such as sympy) accept \(1/2 = 0.5\) | Bad extraction marks right answers wrong; multiple-choice answers can be guessed |
| Code | Run the program on unit tests in a sandbox; reward 1 if all pass, or the fraction passed | Too few tests: the model learns special cases; code that edits or skips the tests |
| Format | Regular expression: reasoning inside <think>...</think>, answer inside <answer>...</answer>; valid JSON; a required language | Format reward much bigger than accuracy reward: the model learns the format only |
| Instruction constraints | Check “at most 3 sentences”, “include the word X”, “no commas” | Satisfies the letter of the constraint with a useless answer |
| Vision and video | IoU (intersection over union) of a predicted box or time span with the true one; exact match of a counted number | Reward on the box only, not on the explanation; ambiguous ground truth |
DeepSeek-R1-Zero used two rule-based rewards with equal weight: an accuracy reward (is the final answer right; for code, do the tests pass) and a format reward (is the thinking inside think tags). [Certain] For one answer \(o\) to prompt \(q\):
The R1 paper explains why it did not use a neural reward model for reasoning: such models are “susceptible to reward hacking” in large-scale RL, and retraining them costs compute and adds complexity. [Certain] In the second stage of DeepSeek-R1 a language-consistency reward was added, the fraction of chain-of-thought words in the target language, because the model mixed languages in its reasoning. [Certain]
That last sentence is the weakness of RLVR. The verifier checks the outcome, not the reasoning. With enough samples, a policy finds any gap in the checker: answers in a form the parser accepts too easily, code that special-cases the visible tests, or guessing on multiple-choice questions. Good RLVR data therefore has hidden tests, answers that are hard to guess (numbers, not A/B/C/D), and strict but correct parsing. [Likely]
How to read Figure 2
Step 2 shows the verifier: answer C has the right number but no answer tags, so the parser cannot extract it and both rewards are 0. Step 3 gives the group mean 1.25 and standard deviation 0.829. Step 4 turns these into advantages +0.905, −0.302, −1.508, +0.905: answer B is pushed down only a little, because it got the format right. Step 5 shows that every token of an answer shares its advantage, and that the reference model is needed only when \(\beta \gt 0\).
When does a group teach anything?
Suppose the policy solves a prompt with probability \(p\) and the reward is 0 or 1. A group of \(G\) independent answers has a non-zero advantage only if it contains at least one right and at least one wrong answer:
- Read the symbols
- \(p\)
- the chance that one sampled answer is correct, for this prompt and the current policy
- \(p^G\), \((1-p)^G\)
- the chance that all \(G\) answers are right, or all are wrong; then all advantages are zero
For a hard prompt with \(p = 0.05\): with \(G = 8\) only 33.7% of groups carry any signal, with \(G = 16\) it is 56.0%, and with \(G = 64\) it is 96.3%. Prompts the model always solves are equally useless. As training goes on, more prompts become “always solved”, and a fixed batch of prompts carries less and less gradient. DAPO’s dynamic sampling fixes this by over-sampling and dropping every group whose accuracy is exactly 0 or 1, until the batch is full of useful groups. [Certain] Many recipes also filter the prompt set before training to keep prompts of medium difficulty. [Likely]
The same figure shows the second effect: the scale of the advantage. With 0/1 rewards the group std is \(\sqrt{p(1-p)}\) (for a group with exactly a fraction \(p\) correct). Dividing by it gives a rare correct answer a big advantage: with 1 correct out of 16, \(\hat A = (1 - 1/16)/0.242 = 3.873\), against 1.0 for each correct answer when half are right. Summed over a group, the total push \(\sum_i |\hat A_i|\) is \(2G\sqrt{p(1-p)}\) with the std division and \(2G\,p(1-p)\) without it. So relative to a medium prompt, GRPO gives very hard and very easy prompts more weight than an unscaled advantage would.
How to read Figure 3
At \(p = 0.05\) and \(G = 8\), only about a third of the groups give any gradient. Move to \(G = 64\): almost all do, at 8 times the generation cost. Slide \(p\) to 0.5: every group size gives signal. Compare the orange dashed and the green dotted curves near the edges: the std division keeps the push large for prompts that are almost always solved or failed; without it, these prompts fade out smoothly.
From DeepSeekMath to DeepSeek-R1
DeepSeekMath (February 2024). GRPO was introduced to train DeepSeekMath-RL 7B from DeepSeekMath-Instruct 7B. The reward was a learned reward model, not a rule. Training used about 144K math questions, 64 samples per question, a learning rate of \(10^{-6}\), \(\beta = 0.04\), a maximum length of 1024 tokens, a batch of 1024, and a single policy update per batch of samples (so the ratio is 1 and the clip never acts). [Certain] Results: GSM8K rose from 82.9% to 88.2% and MATH from 46.8% to 51.7%. [Certain] The paper also found that RL improved majority-vote accuracy (Maj@K) but not Pass@K, and suggested RL makes the output distribution more robust rather than adding new abilities. [Certain] It also ran “iterative RL”, where the reward model was retrained on new policy samples with a replay of 10% old data. [Certain]
DeepSeek-R1-Zero (January 2025). GRPO applied directly to DeepSeek-V3-Base, with no supervised fine-tuning first, and only the two rule-based rewards (70.6). The prompt template only asked the model to think inside think tags and then answer. [Certain] AIME 2024 pass@1 rose from 15.6% to 71.0% during training, and 86.7% with majority voting over 64 samples (cons@64). [Certain] The average response length grew steadily by itself, and the paper describes an “aha moment”: an intermediate checkpoint starts to re-check its own work, with a sudden rise in words like “wait”. [Certain] The later version of the paper lists the R1-Zero settings: learning rate \(3 \times 10^{-6}\), \(\beta = 0.001\), 16 answers per question, 32 questions per step (batch 512), maximum length 32,768 tokens raised to 65,536 after step 8.2k, about 10,400 steps, and the reference model replaced by the current policy every 400 steps. [Certain] (That paper version reports 77.9% for the final R1-Zero; the first version reported 71.0%.)
R1-Zero had problems: poor readability and mixed languages. DeepSeek-R1 fixed them with a multi-stage pipeline: a few thousand long chain-of-thought examples as a “cold start” SFT; reasoning RL with GRPO plus the language-consistency reward; rejection sampling of about 600k reasoning and 200k non-reasoning examples for a second SFT; and a final RL stage for all scenarios, with rule rewards for reasoning and reward models for helpfulness and harmlessness. [Certain] DeepSeek-R1 reached 79.8% on AIME 2024 and 97.3% on MATH-500. [Certain]
The same paper distilled R1 into six smaller dense models (1.5B to 70B, Qwen and Llama). DeepSeek-R1-Distill-Qwen-32B scored 72.6% on AIME 2024, while RL run directly on Qwen-32B (“R1-Zero-Qwen-32B”) scored only 47.0%. [Certain] The lesson: for small models, SFT on a strong reasoner’s outputs (session 59) can beat RL from scratch. Test-time reasoning is the topic of session 72.
| Setting | DeepSeekMath-RL | DeepSeek-R1-Zero | DAPO |
|---|---|---|---|
| Start model | DeepSeekMath-Instruct 7B | DeepSeek-V3-Base | Qwen2.5-32B base |
| Reward | learned reward model | rules: accuracy + format | rules: correct or not, plus overlong shaping |
| Group size \(G\) | 64 | 16 | 16 |
| Learning rate | \(10^{-6}\) | \(3 \times 10^{-6}\) | \(10^{-6}\), 20-step warmup |
| KL \(\beta\) | 0.04 | 0.001 | 0 (removed) |
| Max length | 1024 | 32,768, then 65,536 | 20,480 (16,384 + 4,096 buffer) |
| Updates per batch of samples | 1 | 16 mini-batches, 1 epoch | 16 gradient updates |
| Headline result | MATH 46.8% → 51.7% | AIME 2024 15.6% → 71.0% | AIME 2024: 50 |
All values in this table are from the three papers. [Certain]
Known issues and fixes
After R1, many groups reproduced R1-Zero-style training on open models and looked closely at the loss. Two papers from March 2025 became the standard references: Dr. GRPO (“GRPO Done Right”, in Liu et al., Understanding R1-Zero-Like Training) and DAPO (Yu et al.). [Certain]
Problem 1: the length bias of \(1/|o_i|\)
In (70.2), each answer’s token sum is divided by its own length. So each answer gets the same total weight \(\hat A_i / G\), whatever its length, and the weight per token is \(\hat A_i / (G\, |o_i|)\). For a correct answer (\(\hat A \gt 0\)), a short answer gets a bigger push per token, which favors short correct answers. For a wrong answer (\(\hat A \lt 0\)), a long answer gets a smaller penalty per token, so long wrong answers are punished less. Liu et al. argue this makes wrong answers grow longer and longer, and that some of the famous length growth of R1-Zero-style runs comes from this bias, not from better reasoning. [Certain] They also found that open-source implementations (TRL, OpenRLHF, verl and others) all normalized the loss by response length. [Certain]
Problem 2: the difficulty bias of the std
As shown in Figure 3, dividing by the group std gives more weight to prompts with a low std, that is, prompts the policy almost always solves or almost always fails. Liu et al. call this a question-level difficulty bias. [Certain]
Fix: Dr. GRPO
Dr. GRPO simply removes both normalizations: no \(1/|o_i|\) and no division by the std. To keep the loss scale fixed, the token sum is divided by a constant, such as the generation budget \(L_\text{max}\), instead of the answer length: [Certain]
- Read the symbols
- \(L_\text{max}\)
- a constant, for example the maximum generation length; the same for every answer and every step
- \(\hat A_i\)
- the centered reward, with no std division; this is the plain REINFORCE-with-group-mean-baseline advantage
With this change, every token gets weight \(\hat A_i / (G L_\text{max})\), so a long wrong answer is penalized in proportion to its length. Their minimal recipe reached 43.3% on AIME 2024 with a 7B base model in 27 hours on 8 A100 GPUs, and used fewer tokens per answer than GRPO, especially for wrong answers. [Certain] The paper also found that Qwen2.5 base models already reason well with no prompt template, and that nearly all base models it tested, including DeepSeek-V3-Base, already show “aha”-style self-reflection before any RL, so the aha moment is not created by RL alone. [Certain]
DAPO: four fixes for long reasoning
DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) proposed four techniques. [Certain]
- Clip-higher. Use a larger upper clip than lower clip: \(\epsilon_\text{low} = 0.2\), \(\epsilon_\text{high} = 0.28\). With a symmetric clip, a token with probability 0.01 can only rise to 0.012 in one update, so rare tokens stay rare and the entropy collapses. A higher upper bound keeps exploration alive.
- Dynamic sampling. Drop groups whose accuracy is 0 or 1 and keep sampling until the batch is full (formula (70.7) explains why).
- Token-level policy-gradient loss. Average over all tokens in the batch instead of first over each answer:
- Read the symbols
- \(\sum_i |o_i|\)
- the total number of answer tokens in the group (or batch); every token has the same weight
- \(\epsilon_\text{low}\), \(\epsilon_\text{high}\)
- the separate lower and upper clip ranges of clip-higher
- \(\hat A_{i,t}\)
- still the GRPO advantage (70.1), with the std division
- Overlong reward shaping. Answers cut off at the length limit are not simply scored 0 (they may contain good reasoning). Instead a soft length penalty is added near the limit:
With \(L_\text{max} = 16{,}384\) and \(L_\text{cache} = 4{,}096\), an answer of 14,000 tokens gets \((12{,}288 - 14{,}000)/4{,}096 = -0.418\). DAPO also removed the KL term, arguing that in long chain-of-thought training the policy should be allowed to move far from the start model. [Certain] With these changes DAPO reached 50 points on AIME 2024 from Qwen2.5-32B base, above DeepSeek-R1-Zero-Qwen-32B’s 47, in half the training steps. [Certain]
Group of \(G = 4\): two correct answers of 200 and 400 tokens, two wrong answers of 300 and 1200 tokens. Rewards \([1, 1, 0, 0]\): mean 0.5, std 0.5, so \(\hat A = [+1, +1, -1, -1]\) with GRPO’s formula (and \(\pm 0.5\) without the std).
GRPO (70.2), weight per token \(\hat A_i/(G|o_i|)\): \(+0.00125\), \(+0.000625\), \(-0.000833\), \(-0.000208\). Each answer’s total is \(\pm 0.25\), so the 1200-token wrong answer is penalized per token four times less than the 300-token one.
DAPO (70.9), weight per token \(\hat A/\sum|o_i| = 1/2100 = 0.000476\) for everyone. Totals: \(+0.095\), \(+0.190\), \(-0.143\), \(-0.571\). The long wrong answer now carries the largest total penalty.
Dr. GRPO (70.8) with \(L_\text{max} = 2048\) and \(\hat A = \pm 0.5\): weight per token \(0.5/(4 \times 2048) = 0.000061\). Totals: \(+0.0122\), \(+0.0244\), \(-0.0183\), \(-0.0732\): the same proportions as DAPO, at a fixed scale that does not change from batch to batch.
How to read Figure 4
In GRPO mode the right panel has four bars of equal size: every answer gets a total of ±0.25, whatever its length. Drag D’s length up: its per-token penalty shrinks, so the model loses little by making a wrong answer longer. Switch to DAPO or Dr. GRPO: all tokens have the same weight, and the total penalty of D grows with its length. Note that in DAPO the per-token weight still changes when D changes, because the denominator is the total token count of the batch.
Other changes you will meet
- No KL, no reference model. TRL’s GRPO trainer uses \(\beta = 0\) by default, citing work showing the KL term is not essential; then the reference model is not loaded. [Likely] Removing the leash is reasonable when the reward is a strict verifier; it is risky with a learned reward model.
- Std over the batch. Some libraries center per group but divide by the std of the whole batch, which removes the difficulty bias while keeping a fixed scale. TRL calls this
scale_rewards="batch". [Likely] - Loss type as a switch. Current TRL exposes the aggregations as
loss_type, with values such as"grpo","dr_grpo"and"dapo", and in late 2026 its documented default is"dapo". [Likely] Defaults change between versions: always check which one you run.
The algorithm
- input: policy \(\pi_\theta\) (from an SFT or base model); reference \(\pi_\text{ref} \leftarrow \pi_\theta\); verifier \(r(q, o)\); group size \(G\); \(\epsilon\), \(\beta\)
- for step \(= 1, 2, \dots\):
- sample a batch of prompts \(q\); \(\pi_{\theta_\text{old}} \leftarrow \pi_\theta\)
- for each \(q\): sample \(G\) answers \(o_1 \dots o_G \sim \pi_{\theta_\text{old}}(\cdot \mid q)\) at temperature 1 // the slow part
- \(r_i \leftarrow r(q, o_i)\) // a program: parse, compare, run tests
- \(\hat A_i \leftarrow (r_i - \operatorname{mean}(r)) / (\operatorname{std}(r) + \varepsilon)\) // (70.1); Dr. GRPO: no division
- (optional, DAPO) drop the group if all \(r_i\) are equal; sample more prompts
- store \(\log \pi_{\theta_\text{old}}\) of every answer token; if \(\beta \gt 0\), also \(\log \pi_\text{ref}\)
- for each mini-batch (often one epoch): \(\rho \leftarrow \exp(\log \pi_\theta - \log \pi_{\theta_\text{old}})\)
- per-token term \(\min(\rho \hat A, \operatorname{clip}(\rho)\hat A) - \beta\, \mathbb{D}\) // (70.2), (70.3)
- aggregate over answer tokens (per answer, per token, or by a constant), negate, Adam step
- (optional, as in R1) every few hundred steps set \(\pi_\text{ref} \leftarrow \pi_\theta\)
- input: answer text \(o\), ground truth \(a^*\)
- \(r_\text{format} \leftarrow 1\) if \(o\) matches “<think> … </think> <answer> … </answer>” exactly once, else 0
- \(s \leftarrow\) text inside the answer tag (or inside the last
\boxed{}); if none, return \(r_\text{format}\) - normalize \(s\) and \(a^*\): strip spaces and units, remove trailing “.0”, parse fractions
- \(r_\text{acc} \leftarrow 1\) if they are equal as strings, or equal as symbolic or numeric values (with a small tolerance); else 0
- return \(r_\text{acc} + r_\text{format}\) // keep the format reward small or equal; never bigger
Watch Algorithm 1 run, line by line
Figure 5 runs Algorithm 1 for real in your browser on a model small enough to follow every number. There are four prompts, “2+3=”, “4+3=”, “6+2=” and “3+3=”. An answer has two tokens: a number from 0 to 9, then either the closing tag “</a>” or a period. The verifier is a tiny version of Algorithm 2 and reward (70.6): \(r = r_\text{acc} + r_\text{fmt}\), so 2 for the right number with a closed tag, 1 for a wrong number with a closed tag, 0 without the tag (the parser cannot read the answer). The reference model, and the starting policy, says the right number with probability 0.18, prefers the number one too high (0.30), and closes the tag with probability 0.40.
The policy’s parameters are its logits: 10 scores per prompt for the number and 2 per prompt for the tag. The gradient of one token’s term of (70.2) with respect to the logit \(z_k\) has a simple form, which is all the simulation needs:
- Read the symbols
- \(y\)
- the token that was sampled at this position; \(\pi_\theta(k)\) is the softmax probability of token \(k\) in the same context
- \(\hat A \rho\)
- the PPO part: it is switched off when the token is clipped
- \(\beta(\pi_\text{ref}/\pi_\theta - 1)\)
- the gradient of the \(k_3\) term (70.3): it pushes \(y\) up when the reference likes it more than the policy does, and down when it likes it less. It is never clipped, and it grows large for a token that the policy has made rare
Each mini-batch gradient \(g\) is then clipped to norm 0.1, as trainers clip the gradient norm, and the logits move by \(\alpha g\) (plain gradient ascent; real runs use Adam).
- for step \(= 1, 2, \dots\) (at most 200):
- \(\pi_{\theta_\text{old}} \leftarrow \pi_\theta\); for each prompt sample \(G\) answers \(o_i\)
- \(r_i \leftarrow r_\text{acc} + r_\text{fmt}\) // (70.6)
- \(\hat A_i \leftarrow (r_i - \bar r) / (s_r + 10^{-4})\), group mean and std // (70.1)
- store \(\log \pi_{\theta_\text{old}}\) and \(\log \pi_\text{ref}\) of every token
- for epoch \(= 1 \dots K\): shuffle, mini-batches of \(M\) answers
- \(\rho_{i,t} \leftarrow \exp(\log \pi_\theta - \log \pi_{\theta_\text{old}})\)
- \(\min(\rho \hat A, \operatorname{clip}(\rho)\hat A) - \beta \mathbb{D}\) per token // (70.2), (70.3)
- mean over the tokens of each answer, then over answers
- \(\theta \leftarrow \theta + \alpha\, g\), \(g\) norm-clipped // (70.11)
- log accuracy, KL to \(\pi_\text{ref}\), clip fraction, dead groups
How to read Figure 5
Step to line 4 with seed 1. The group of “2+3=” has rewards [1, 0, 1, 0, 0, 0, 0, 1]: three closed tags, all with wrong numbers. This is the worked example of the first section, and the advantages are the same, +1.291 and −0.774 (the \(10^{-4}\) in the denominator changes the third digit). The group of “3+3=” has only one closed tag, so that answer gets +2.646. Step on to line 8 and watch the ratios leave 1 in the second mini-batch. Line 8 also shows the \(k_3\) term: tiny at first, because the policy is still close to the reference. Then choose Fast and Play: the format rate rises first, then accuracy. Try “No clip”: the ratio histogram grows a red bar at 2 and above. Try “Strong KL”: the KL stays low, but the accuracy is noisy, because the \(k_3\) gradient (70.11) is large whenever a token the policy has made rare is sampled.
The four groups have rewards “2+3=” [1, 0, 1, 0, 0, 0, 0, 1], “4+3=” [2, 0, 1, 0, 0, 1, 0, 1], “6+2=” [0, 0, 0, 1, 1, 0, 2, 1] and “3+3=” [0, 0, 0, 0, 0, 0, 0, 1]. No group has all rewards equal, so all four give signal.
For “2+3=”: mean 0.375, std 0.484, so the answer “6</a>” gets \(\hat A = 0.625 / 0.4842 = 1.291\). Both of its tokens get this advantage: the closed tag deserves it, the wrong number 6 does not. This is the credit problem of outcome rewards in its smallest form.
After step 1 (4 epochs × 2 mini-batches of 16 answers), \(\pi_\theta(6 \mid \text{2+3=})\) went from 0.302 to 0.466 and \(\pi_\theta(\text{</a>} \mid \text{2+3=})\) from 0.401 to 0.564. 70% of the token updates were clipped: with \(\alpha = 20\) the steps are large, and the clip is working hard. The exact accuracy is 0.107 after step 1, 0.742 after 10 steps and 0.942 after 20, with a KL to the reference of 2.295 nats.
Two runs side by side: clip, KL weight and group size
Figure 6 runs the same simulation twice with the same seed and one different setting. Three comparisons are worth seeing.
The clip. With \(\beta = 0\) and no clip (\(\epsilon = 100\)), one step can move a prompt very far. For seed 4, two prompts lock onto wrong numbers with closed tags in the first steps. Each then gets the same reward 1 for every answer in its group: the std is 0, the advantage is 0 (formula 70.1), and the group teaches nothing. The accuracy sits at 0.50 until step 38 and reaches 0.994 only by step 150, when rare right answers finally get sampled. With \(\epsilon = 0.2\), the run is at 0.973 after 10 steps. The curve of dead groups (all rewards equal, but not all right) shows the cause directly: 46% of the groups are dead in the first 40 steps of run B, against under 1% in run A.
The KL weight. With \(\beta = 0.5\) the KL to the reference stays far lower (0.348 nats at step 150 for seed 4, against 2.605 with \(\beta = 0\)), but the accuracy is low and noisy (0.193 at step 150). The \(k_3\) gradient pushes rare tokens back hard. Real runs use small values such as 0.04 or 0.001, and many recent recipes drop the term.
The group size. With \(\beta = 0.04\) and seed 4, \(G = 4\) moves between 0.79 and 0.90 after step 30, while \(G = 16\) reaches 0.996. A small group more often has all rewards equal (13% dead groups in the first 40 steps with \(G = 4\), none with \(G = 16\), as formula 70.7 predicts), and its mean is a noisier baseline, while the KL term pulls back at every step. [Likely]
How to read Figure 6
Press “Run to the end” with the clip comparison and seed 4. Run B’s accuracy is flat at 0.50 until step 38, and in the same steps its dashed dead-group curve stays at 0.5: two of the four groups are stuck. Run A climbs in about 10 steps. Change the seed: in some seeds run B is lucky and fast, which is the point: without the clip, the outcome depends on luck in the first few batches. Choose the group comparison: \(G = 4\) has dead groups in early steps, \(G = 16\) has none.
The PyTorch code below runs GRPO end to end on a toy task in a few seconds on a CPU. The “prompt” is two digits \(a, b \in \{0, \dots, 4\}\). The model writes up to 8 tokens from 11 (digits 0 to 9 and an end token). The verifier gives reward 1 if the digit just before the first end token equals \(a + b\). There is no reward model and no critic; the reference model is a frozen copy of the start policy. The flag loss_type switches between the GRPO aggregation (70.2) and Dr. GRPO (70.8). It is deterministic (fixed seed, CPU).
import copy, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) EOS, V, T = 10, 11, 8 # tokens 0-9 are digits, 10 ends the answer; at most 8 tokens G, P = 8, 16 # answers per prompt, prompts per step class TinyLM(nn.Module): # logits from (prompt a, prompt b, previous token, position) def __init__(s): super().__init__() s.a, s.b = nn.Embedding(5, 32), nn.Embedding(5, 32) s.tok, s.pos = nn.Embedding(V + 1, 32), nn.Embedding(T, 32) s.mlp = nn.Sequential(nn.Linear(32, 64), nn.Tanh(), nn.Linear(64, V)) def forward(s, a, b, prev): # a, b: (N,), prev: (N, L) h = s.a(a)[:, None] + s.b(b)[:, None] + s.tok(prev) + s.pos.weight[None, :prev.size(1)] return s.mlp(h) def verify(a, b, y): # verifiable reward: 1 if the digit before EOS equals a + b r = torch.zeros(len(y)) for i in range(len(y)): toks = y[i].tolist() if EOS in toks: k = toks.index(EOS) if k > 0 and toks[k - 1] == int(a[i] + b[i]): r[i] = 1.0 return r def prev_of(y): # previous token at each position; V = "start" return torch.cat([torch.full((y.size(0), 1), V), y[:, :-1]], 1) def token_logp(model, a, b, y): lp = F.log_softmax(model(a, b, prev_of(y)), -1) return lp.gather(-1, y[..., None]).squeeze(-1) def run(loss_type, steps=150, beta=0.04, eps=0.2): torch.manual_seed(0) policy = TinyLM(); ref = copy.deepcopy(policy).eval() for p in ref.parameters(): p.requires_grad_(False) opt = torch.optim.Adam(policy.parameters(), lr=3e-3) for step in range(steps): a = torch.randint(0, 5, (P,)).repeat_interleave(G) # each prompt G times b = torch.randint(0, 5, (P,)).repeat_interleave(G) with torch.no_grad(): # 1) rollout y = torch.full((P * G, T), EOS) for t in range(T): logits = policy(a, b, prev_of(y))[:, t] y[:, t] = torch.distributions.Categorical(logits=logits).sample() done = (y == EOS).long().cumsum(1) # mask: tokens up to the first EOS mask = ((done == 0) | ((done == 1) & (y == EOS))).float() r = verify(a, b, y) # 2) verifier, no reward model rg = r.view(P, G) # 3) group-relative advantage adv = rg - rg.mean(1, keepdim=True) if loss_type == "grpo": adv = adv / (rg.std(1, keepdim=True) + 1e-4) adv = adv.view(-1, 1) old_lp = token_logp(policy, a, b, y) for _ in range(2): # 4) two gradient steps per rollout lp = token_logp(policy, a, b, y) ratio = torch.exp(lp - old_lp) pg = torch.min(ratio * adv, ratio.clamp(1 - eps, 1 + eps) * adv) d = token_logp(ref, a, b, y) - lp # k3 estimator of KL(pi || ref) kl = torch.exp(d) - d - 1 per_tok = -(pg - beta * kl) * mask if loss_type == "grpo": # mean over each answer, then over answers loss = (per_tok.sum(1) / mask.sum(1)).mean() else: # Dr. GRPO: divide by a constant loss = per_tok.sum(1).mean() / T opt.zero_grad(); loss.backward(); opt.step() if step % 30 == 0 or step == steps - 1: right, wrong = r == 1, r == 0 ml = lambda m: mask.sum(1)[m].mean().item() if m.any() else float("nan") print(f"{loss_type:7s} step {step:3d} acc {r.mean():.3f} len right {ml(right):.2f}" f" len wrong {ml(wrong):.2f} no-signal groups {(rg.std(1) == 0).float().mean():.2f}") for lt in ["grpo", "dr_grpo"]: run(lt) # Output (CPU, torch 2.8): # grpo step 0 acc 0.031 len right 4.00 len wrong 5.27 no-signal groups 0.75 # grpo step 30 acc 0.367 len right 2.40 len wrong 3.05 no-signal groups 0.25 # grpo step 60 acc 0.516 len right 2.47 len wrong 3.39 no-signal groups 0.38 # grpo step 90 acc 0.758 len right 2.74 len wrong 3.61 no-signal groups 0.44 # grpo step 120 acc 0.594 len right 2.36 len wrong 4.44 no-signal groups 0.62 # grpo step 149 acc 0.703 len right 2.98 len wrong 5.79 no-signal groups 0.62 # dr_grpo step 0 acc 0.031 len right 4.00 len wrong 5.27 no-signal groups 0.75 # dr_grpo step 30 acc 0.234 len right 3.90 len wrong 4.50 no-signal groups 0.19 # dr_grpo step 60 acc 0.289 len right 3.51 len wrong 4.95 no-signal groups 0.19 # dr_grpo step 90 acc 0.617 len right 3.81 len wrong 4.20 no-signal groups 0.38 # dr_grpo step 120 acc 0.680 len right 3.70 len wrong 6.44 no-signal groups 0.56 # dr_grpo step 149 acc 0.656 len right 3.89 len wrong 5.86 no-signal groups 0.50
What the output shows. Accuracy rises from 3.1% (random guessing) to about 65% to 75% for both losses, with no critic and no reward model. At step 0, 75% of the groups have no signal: the task is hard for a random policy, so most groups are all wrong. Later, 50% to 62% of groups have no signal again, now because many prompts are always solved; this is the situation dynamic sampling is designed for. The two runs differ in accuracy and length from step to step, but with only 16 prompts per step this toy is too noisy to show the length bias clearly; do not read the length columns as evidence for or against Dr. GRPO. The hands-on version of this loop on a real small model is session 77.
Where this is used
| Setting | Verifiable reward | Notes |
|---|---|---|
| Math reasoning models (DeepSeek-R1, many open “-Zero” reproductions) | Final answer equals the ground truth, plus a format check | GRPO, Dr. GRPO or DAPO; groups of 8 to 64; long answers |
| Code generation and agents for software tasks | Unit tests pass in a sandbox | Sandboxing and hidden tests are needed against test hacking [Likely] |
| Instruction following (Tülu 3 RLVR) | Programmatic constraint checks (length, keywords, format) | Tülu 3 used PPO, not GRPO: RLVR is about the reward, not the algorithm |
| Structured output and tool calls | The output parses as JSON or matches a schema; the tool call runs | Often mixed with accuracy rewards [Likely] |
| VLM reasoning and grounding (session 190) | Exact answer for visual math or counting; IoU of a predicted box | Same GRPO loop; image tokens are part of the prompt and get no loss [Likely] |
| Video reasoning and video QA (Video-R1) | Exact-match answers on video questions | Video-R1 proposes T-GRPO, a variant that rewards use of temporal information [Certain] |
| Video temporal grounding and key-moment detection (VideoChat-R1) | Temporal IoU of a predicted time span with the true span; tracking overlap | VideoChat-R1 reports large gains on temporal grounding and tracking with rule-based rewards [Certain]; see session 205 |
| Video classification and action recognition by a Video-LLM | Predicted label equals the true label | Easy to verify, but labels are easy to guess: random accuracy may be high [Speculation] |
Common mistakes
- All-equal groups treated as signal. Dividing by a std of 0 without an \(\varepsilon\) gives NaN; with an \(\varepsilon\) it gives 0, which is correct but wastes the samples. Log the fraction of no-signal groups and filter them (dynamic sampling) or make the prompt set harder.
- Mixing up group statistics. The mean and std must be taken over the \(G\) answers of the same prompt. Computing them over the whole batch (mixing prompts) gives a different method: easy prompts then look good and hard prompts look bad, whatever the answers.
- Not knowing your aggregation. “Mean per answer, then mean over answers” (original GRPO), “mean over all tokens” (DAPO) and “sum divided by a constant” (Dr. GRPO) have different length behavior. Check the
loss_typeof your library. - A format reward that dominates. If the format reward is larger than the accuracy reward, the model first learns tags and can stop there. Keep it equal or smaller, and log the two rewards separately.
- A loose verifier. Extracting “the last number in the text” rewards answers that list many numbers. Multiple-choice answers can be guessed at 25%. Use strict extraction, numeric answers and hidden tests.
- Truncated answers scored as normal. An answer cut off at the length limit has no final answer; scoring it 0 together with real wrong answers punishes long reasoning blindly. Mask it or use overlong shaping (70.10).
- Loss on the prompt. Only answer tokens are actions. Mask prompt and padding tokens in the policy loss and in the KL.
- Mismatched log-probabilities. If the inference engine (for example vLLM) and the trainer compute slightly different log-probabilities, the ratio is not 1 at the start, and the clip acts on noise. Recompute \(\log \pi_{\theta_\text{old}}\) in the trainer, or check that the ratio starts at 1.000. [Likely]
Interview questions
How does GRPO compute the advantage?basic
Why does GRPO not need a critic, and what does it lose by dropping it?core
What is a verifiable reward? Give examples and one weakness.basic
What happens when all answers in a group get the same reward? How often does it happen?core
Explain the length bias in the original GRPO loss.advanced
What is the difficulty bias from the std normalization?advanced
What did DeepSeek-R1-Zero show, and why was DeepSeek-R1 still needed?core
GRPO has a PPO-style clip. When does it actually do anything?core
Where does the KL term go in GRPO versus PPO-RLHF, and why do many recipes now drop it?advanced
Design an RLVR setup for training a Video-LLM to find key moments in videos.advanced
Exercises
- A group of 4 answers has rewards \([1, 0, 0, 0]\). Compute the GRPO advantages (std dividing by \(G\)), the mean-only advantages and the RLOO advantages.
- With rewards \([2, 1, 0, 2]\) (accuracy plus format), compute the group mean, the std (dividing by \(G\)) and the four GRPO advantages.
- For \(p = 0.1\), compute the chance that a group has signal for \(G = 4\) and \(G = 16\).
- Under the original GRPO aggregation, a correct answer of 100 tokens and a correct answer of 500 tokens both have \(\hat A = 1\), with \(G = 8\). What is the weight per token of each? What is it under Dr. GRPO with \(L_\text{max} = 1000\) (same \(\hat A\))?
- Compute the memory of model state for GRPO on a 32B policy with \(\beta = 0.001\), and with \(\beta = 0\), using (70.5).
- With DAPO’s overlong shaping, \(L_\text{max} = 16{,}384\), \(L_\text{cache} = 4{,}096\): what is the length reward for answers of 10,000, 14,336 and 17,000 tokens?
Answerscheck
Key takeaways
- GRPO samples a group of \(G\) answers per prompt and uses the group-normalized reward (70.1) as the advantage of every token of an answer. The group mean replaces the critic.
- The loss is PPO’s clipped objective plus a \(k_3\) KL term to a reference (70.2), (70.3). No critic halves the model-state memory of PPO for a 7B model (126 GB against 252 GB), and \(\beta = 0\) also removes the reference.
- RLVR means rewards from programs: answer checks, unit tests, format and constraint checks, IoU. It resists reward-model hacking but checks only the outcome, so verifiers must be strict.
- Groups where all answers score the same give no gradient; the chance of signal is \(1 - p^G - (1-p)^G\) (70.7). Dynamic sampling filters such groups.
- DeepSeekMath introduced GRPO with a learned reward model; DeepSeek-R1-Zero used it with rule-based rewards on a base model (AIME 2024: 15.6% → 71.0%); DeepSeek-R1 added cold-start SFT, rejection sampling and more RL.
- In the live toy run (Figures 5 and 6), the format reward alone pushes up wrong numbers with closed tags; without the clip a prompt can lock onto such an answer, its group rewards become all equal, and it stops learning until a rare right answer is sampled.
- The original loss has a length bias (\(1/|o_i|\)) and a difficulty bias (std). Dr. GRPO removes both (70.8); DAPO uses a token-level loss, clip-higher, dynamic sampling and overlong shaping (70.9), (70.10).
Key formulas
References for this session
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective
- Raschka, Build a Reasoning Model (From Scratch), chapter 6: Training reasoning models with RL
- Raschka, Build a Reasoning Model (From Scratch), chapter 7: Improving GRPO
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective
- Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
- Schulman et al. (2017). Proximal Policy Optimization Algorithms
- Schulman (2020). Approximating KL Divergence (blog)
- Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
- Hugging Face (2026). TRL documentation: GRPO Trainer
- Feng et al. (2025). Video-R1: Reinforcing Video Reasoning in MLLMs
- Li et al. (2025). VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning
- Raschka, Build a Reasoning Model (From Scratch), chapter 6: Training reasoning models with RL
- Raschka, Build a Reasoning Model (From Scratch), chapter 7: Improving GRPO