GRPO with a verifiable reward
In this lab you train a small chat model, Qwen2.5-0.5B-Instruct, on GSM8K math word problems with GRPO and two rewards that a program checks: is the final number right, and is the answer in the required layout. You test the reward functions, compute one GRPO step by hand, plan the group size and token budget, train with TRL, watch reward, length and KL, and read answers before and after to find out what the model really learned.
What you will learn
- How to turn GSM8K into prompts with exact gold answers and a fixed answer layout
- How to write and unit-test a correctness reward and a format reward, and which loopholes a loose checker leaves open
- How TRL computes rewards, group advantages and the GRPO loss, step by step with real numbers
- How to plan a run: the share of groups with a learning signal, the generation budget and the memory
- How to train with GRPOTrainer and read its logs: reward per function, length, clipped ratio, KL and frac_reward_zero_std
- How to evaluate before and after with a strict and a loose checker, standard errors and paired counts
- How to read samples and tell format gains, finishing gains, real math gains and reward hacking apart
Why this matters
You take a small chat model, give it grade-school math word problems whose answers are single numbers, and let it practice. A short Python function checks each answer: right number, reward 1; wrong, reward 0. A second function checks the layout of the answer. GRPO turns these checks into weight updates. In this lab you write the two reward functions, train with TRL’s GRPOTrainer, watch the reward, the answer length and the KL during training, and then read the model’s answers before and after to find out what it actually learned.
Session 70 gave the theory: the group-relative advantage, the clipped objective, the KL term, and the length and difficulty biases. Session 75 explained why generation is the expensive part of an RL step. This lab turns that theory into a run you control. That matters because RL is easy to start and hard to read: the reward curve can go up while the model gets worse, and the only way to know is to measure the right things and to read samples.
You will:
- Prepare GSM8K problems with exact answers and a prompt that asks for a fixed layout.
- Write and test two reward functions: correctness and format (Figure 1).
- Compute one GRPO step by hand: rewards, group advantages, loss (Figure 2).
- Plan the run: group size, batch, length budget, memory (Figure 3).
- Train a toy policy in your browser to see what healthy and broken runs look like (Figure 4).
- Train
Qwen2.5-0.5B-Instructwith GRPO, watch five curves, evaluate on the test set, and read samples before and after.
This is the same recipe, at a tiny scale, that real reasoning models use. DeepSeekMath introduced GRPO and trained on GSM8K- and MATH-style questions [Certain]. DeepSeek-R1-Zero used only two rule-based rewards, an accuracy reward and a format reward with think tags, which is exactly the pair you build here [Certain]. TRL’s GRPOTrainer, used in this lab, is one of the common open-source implementations; others are verl and OpenRLHF [Likely].
Setup
Hardware. One NVIDIA GPU with 24 GB (L4, A10G, RTX 4090) is comfortable for the 0.5B model with full fine-tuning. A 16 GB card works if you lower the batch or the completion length [Likely]. Without vLLM, TRL generates with the normal Hugging Face generate, which is slow; generation will take most of the time of each step (session 75).
pip install "transformers>=4.56" "trl>=0.24" datasets accelerate
The code on this page was checked with transformers 4.57, trl 0.24 and torch 2.8. We ran every script end to end on a CPU with a tiny model (SmolLM2-135M-Instruct, two GRPO steps, 24-token completions) to check that it runs. We did not run the full GPU training for you, and this page reports no after-training results: you will measure them. TRL changes fast; if an argument fails, check the GRPO trainer documentation for your version.
Model. Qwen/Qwen2.5-0.5B-Instruct: 494,032,768 parameters, 24 layers, a vocabulary of 151,936 tokens, with a chat template [Certain]. We use the instruct model, not the base model, because it already solves many of the problems: before training, the last number of its answer is right on 23 of the first 64 test problems (Step 8). It does not yet use the answer layout, but that is the easy part to learn, and the partial format credit is there to give it an early signal (Step 4 explains why the signal matters).
Data. openai/gsm8k, config main: 7,473 training and 1,319 test problems (we counted them). Each solution is written step by step and ends with a line like #### 72, the exact final answer. In the training split, every final answer is an integer once commas are removed (79 of them contain a comma, such as 1,250, and 3 are negative); we checked this with a script. Exact integer answers are what make the reward verifiable: a program can check them with no judgment.
The table you will fill. Copy it into your notes now. Every empty cell is a number you measure.
| Model | Test accuracy (strict) | Format rate | Loose accuracy | Mean answer tokens | Cut at the limit |
|---|---|---|---|---|---|
| Before training | |||||
GRPO, \(\beta = 0.04\), loss_type="grpo" | |||||
| Your variant (Step 9) |
“Strict” means the answer is inside the answer tags and is exactly the right number. “Loose” means the last number anywhere in the text is right, a common but weaker check. The gap between the two columns tells you how much of a change is about format and how much is about math.
Step 1: problems, gold answers and the prompt
Each training row needs two things: the prompt the model sees and the gold answer the reward function needs. The gold answer is the text after ####, with commas removed. The prompt is a chat: a system message that fixes the layout, and the question as the user message.
# the system prompt (in rewards.py below)
You are a careful math tutor. Think step by step inside <think> </think>. Then give only the final number inside <answer> </answer>.
Example:
<think>
3 apples + 2 apples = 5 apples
</think>
<answer>
5
</answer>
Why a fixed layout? So that the reward function knows where the final answer is. Without it, the checker must guess which number in a long text is the answer, and guesses can be exploited (Step 2). The tiny example in the prompt is there because a small model follows a shown example more reliably than a described one [Likely]. Do not expect too much from it: even with the example, the greedy answers of the 0.5B model before training almost never use the full layout (Step 8). In an earlier version of this lab we used a system prompt that described the layout in words but showed no example: in the first GRPO step, none of the 16 sampled answers used any of the four tags, so every reward was 0 (frac_reward_zero_std was 1.0) and the step taught nothing.
TRL applies the model’s chat template to a conversational prompt column for you [Certain]. Any other column of the dataset, such as gold, is passed to every reward function as a keyword argument [Certain]. That is how the checker gets the right answer for each completion.
Step 2: write the reward functions
A reward function in TRL is a plain Python function. It receives the list of completions (for chat data, each completion is a list of messages) plus the extra dataset columns, and returns one number per completion [Certain]. With several functions, TRL adds their outputs with weights:
- Read the symbols
- \(o_i\)
- the \(i\)-th completion (the model’s answer text) for one prompt
- \(y\)
- the gold answer of that prompt, for example 72
- \(r_{\text{acc}}\)
- correctness: 1 if there is exactly one answer block and it holds exactly the gold number, else 0
- \(r_{\text{fmt}}\)
- format: 1 for the exact layout (think block, then answer block, nothing else); otherwise 0.2 for each of the four tags that appears exactly once
- \(w_{\text{acc}}, w_{\text{fmt}}\)
- the weights, set with
reward_weights=[1.0, 0.5]; so the total reward is between 0 and 1.5
Correctness is the reward we care about. The format reward is a helper: it gives a gradient toward the layout even when no answer in a group is right, and its partial credit gives a signal before the model uses all four tags. It is worth at most a third of the total, so a correct answer always beats a nicely formatted wrong one.
# rewards.py: verifiable rewards for GSM8K. A program checks the answer; no model is involved. import re SYSTEM = ("You are a careful math tutor. Think step by step inside <think> </think>. " "Then give only the final number inside <answer> </answer>.\n" "Example:\n<think>\n3 apples + 2 apples = 5 apples\n</think>\n<answer>\n5\n</answer>") def gold_of(solution): # GSM8K solutions end with "#### 72" return solution.split("####")[-1].strip().replace(",", "") ANSWER = re.compile(r"<answer>(.*?)</answer>", re.S) NUMBER = re.compile(r"-?\d+(?:\.\d+)?") LAYOUT = re.compile(r"\s*<think>.+?</think>\s*<answer>.+?</answer>\s*", re.S) TAGS = ("<think>", "</think>", "<answer>", "</answer>") def clean(s): # "$1,250." -> "1250" return s.strip().replace(",", "").replace("$", "").rstrip(".").strip() def correct(text, gold): """1.0 only if there is exactly one answer block and it holds exactly the gold number.""" blocks = ANSWER.findall(text) if len(blocks) != 1: return 0.0 a = clean(blocks[0]) if not NUMBER.fullmatch(a): # "24 or 72" is not a number: no credit return 0.0 return 1.0 if abs(float(a) - float(gold)) < 1e-6 else 0.0 def layout(text): """1.0 for the exact layout; partial credit (0.2 per tag used exactly once) otherwise.""" if LAYOUT.fullmatch(text) and all(text.count(t) == 1 for t in TAGS): return 1.0 return 0.2 * sum(text.count(t) == 1 for t in TAGS) def loose(text, gold): """A LOOSE checker, shown as a trap: is the last number anywhere in the text the gold?""" nums = NUMBER.findall(text.replace(",", "")) return 1.0 if nums and abs(float(nums[-1]) - float(gold)) < 1e-6 else 0.0 # TRL calls each reward function with the completions and every extra dataset column as a keyword. def correctness_reward(completions, gold, **kwargs): return [correct(c[0]["content"], g) for c, g in zip(completions, gold)] def format_reward(completions, **kwargs): return [layout(c[0]["content"]) for c in completions]
Test the reward functions before you train. A bug in a reward function is a bug in the goal: the optimizer will faithfully learn the wrong thing. Write a few test answers by hand, including the cases you want to reject, and check every output. Figure 1 runs the same three checkers in your browser; the presets are our unit tests, and we checked each value with the Python code above.
rewards.py, computed live on the text you type. The total uses formula (77.1) with \(w_{\text{fmt}} = 0.5\). The question is GSM8K training problem 0 (Natalia’s clips: 48 in April, half as many in May), gold answer 72.How to read Figure 1
“Correct” gets \(1 + 0.5 \times 1 = 1.5\). “No tags” has the right number but earns 0 from the strict checker and 1 from the loose one. “Hedging” lists three numbers in the answer box: the loose checker reads the last one, 72, and pays 1; the strict checker pays 0 because “24 or 48 or 72” is not a number. A policy under the loose checker can learn to list many numbers. “Two answers” puts 24 and then 72 in two answer blocks: correctness 0, format 0.4 (only the two think tags are used exactly once). Edit the text and watch which rule fires.
Three rules in correct close the loopholes that a policy finds most often [Likely]: exactly one answer block (no “try several answers”), the block must be a single number (no hedging), and light cleaning only (a dollar sign, commas, a final period). Each loophole you leave open is a way to get reward without solving the problem. This is called reward hacking: the policy optimizes the checker, not the task. You will see it happen in Figure 4 when you switch to the loose checker.
Step 3: one GRPO step by hand
Now follow one prompt through one GRPO step. The trainer samples \(G = 8\) completions for the same question, at temperature 1, so they differ. Each completion gets a total reward \(r_i\) from (77.1). GRPO does not train a value network (a critic). Instead, it compares each answer with the other answers to the same question:
- Read the symbols
- \(\hat A_i\)
- the advantage of completion \(i\): how much better it is than its group, in units of the group’s spread; every token of the completion gets this same value
- \(\bar r, s\)
- the mean and the standard deviation of the \(G\) rewards of this group. TRL uses the sample standard deviation (divide by \(G - 1\)) and adds \(10^{-4}\) so it never divides by zero [Certain]
- \(G\)
- the group size,
num_generationsin TRL (default 8)
Advantages in a group sum to zero: the answers above the mean are pushed up and the ones below are pushed down. If all \(G\) rewards are equal, then \(s = 0\), every \(\hat A_i = 0\), and this prompt adds nothing to the gradient. TRL logs the share of such groups as frac_reward_zero_std [Certain]. Step through Figure 2 and click rows to change the answers.
How to read Figure 2
Step 1 shows the eight answers, step 2 the correctness and format rewards, step 3 the total, step 4 the group mean and spread, and step 5 the advantages as bars. The three correct answers get +1.165 each; the answer with no tags gets −1.090 even though its number is right, because under this reward it is worse than the group. Click “All answers correct”: the spread becomes 0 and all advantages are 0. That prompt teaches nothing, and the readout says so.
The eight rewards are 1.5, 0.5, 0, 1.5, 0.1, 0.5, 0.2 and 1.5 (correct, wrong number, no tags, correct, cut off, hedging, two answers, “$72.”). Their sum is 5.8, so \(\bar r = 0.725\). The squared deviations sum to 3.095, so \(s = \sqrt{3.095 / 7} = 0.665\). The advantages are \((1.5 - 0.725)/0.665 = 1.165\) for each correct answer, \(-0.338\) for “wrong number” and “hedging”, \(-1.090\) for “no tags”, \(-0.940\) for “cut off” and \(-0.789\) for “two answers”. They sum to 0.
Then comes the loss. For each token \(t\) of completion \(i\), the policy compares its new probability with the probability when the sample was made, the ratio \(\rho_{i,t}\), and uses the PPO clip. With loss_type="grpo", TRL averages over the tokens of each completion first, then over the completions, as in DeepSeekMath [Certain]:
- Read the symbols
- \(\pi_\theta, \pi_{\theta_\text{old}}, \pi_\text{ref}\)
- the policy being trained, the policy that sampled the answers, and the frozen starting model (the reference)
- \(|o_i|\)
- the number of tokens of completion \(i\)
- \(\epsilon\)
- the clip range,
epsilon=0.2by default - \(\beta\)
- the weight of the KL penalty; TRL’s default is 0 (no penalty and no reference model in memory), we set 0.04, the value of DeepSeekMath, so that you can watch the KL [Certain]
- \(\mathbb{D}_{i,t}\)
- the “k3” estimate of the KL divergence from the reference at one token; it is never negative and is 0 when the two models agree
With TRL’s default of one update per batch (num_iterations=1), the sampling policy and the trained policy are the same at the moment of the update, so \(\rho_{i,t} = 1\) and the clip never acts. The gradient is then just “raise the log-probability of each token in proportion to \(\hat A_i\)”, plus the KL pull toward the reference. The clip matters when you reuse a batch for several updates.
- input: policy \(\pi_\theta\), reference \(\pi_\text{ref}\) (only if \(\beta \gt 0\)), reward functions with weights, \(B\) prompts, group size \(G\)
- for each prompt \(q\): sample \(G\) completions at temperature 1, up to
max_completion_lengthtokens // the slow part - run every reward function on every completion; \(r_i \leftarrow\) weighted sum (77.1)
- per group: \(\bar r, s\); \(\hat A_i \leftarrow (r_i - \bar r)/(s + 10^{-4})\) (77.2); log the share of groups with \(s = 0\)
- one forward pass of \(\pi_\theta\) (and of \(\pi_\text{ref}\)) over prompt + completion to get token log-probabilities
- loss (77.3); backward; optimizer step (with gradient accumulation if the batch was split)
- log reward, reward per function, completion length, clipped ratio, KL, entropy
Step 4: plan the run
Before you spend GPU hours, check three things on paper: will the groups have a signal, how many tokens will you generate, and does it fit in memory.
Signal. Think only about correctness for a moment. If a prompt is solved with probability \(p\) by the current policy, a group of \(G\) independent samples is all wrong with probability \((1-p)^G\) and all right with probability \(p^G\). Only mixed groups teach:
With \(p = 0.1\) and \(G = 8\), only 57% of prompts give a signal; with \(G = 16\) it is 81%. With \(p = 0.5\) and \(G = 8\) it is 99%. The format reward adds signal even when correctness gives none, as long as the answers differ in layout. Problems that are much too hard or much too easy for the model are wasted compute; this is why some recipes filter or re-sample prompts (DAPO calls it dynamic sampling, session 70).
Tokens. Every optimizer step generates up to
where \(B\) is the number of prompts per step and \(L_\text{max}\) the completion limit. In our config, \(B = 8\) (64 completions per step divided by \(G = 8\)), \(L_\text{max} = 384\): at most 24,576 tokens per step, and at most 6,144,000 tokens for 250 steps. The run sees \(250 \times 8 = 2{,}000\) prompts, about 27% of the 7,473 training problems: less than one epoch.
Memory. Full fine-tuning in mixed precision costs about 16 bytes per parameter for weights, gradients and Adam states (session 38): \(16 \times 494{,}032{,}768 = 7.9\) GB. With \(\beta \gt 0\), the reference model adds a BF16 copy: 0.99 GB. Then come the activations of the forward and backward pass over 64 sequences of up to 320 + 384 tokens, the logits over a 151,936-token vocabulary, and the generation KV cache. Gradient checkpointing keeps the activations small. If you run out of memory, lower per_device_train_batch_size and raise gradient_accumulation_steps by the same factor: the step stays the same.
How to read Figure 3
The curve is symmetric: a prompt the model always solves is as useless as one it never solves. Move \(p\) to 0.1 and raise \(G\) from 8 to 16: the useful share goes from 0.570 to 0.815, but the tokens per step double. Larger groups buy signal on hard prompts with generation time. Measure \(p\) for your prompts with the before-training evaluation (Step 8) and pick \(G\) and the data with this trade-off in mind.
Step 5: a toy you can train in your browser
A real run takes hours. To build intuition first, Figure 4 trains a toy policy in a fraction of a second. The toy model does not write tokens. It chooses one of seven answer styles, which stand for things a small model really does: answer short without tags, reason at length without tags, use the tags with short, medium or long careful reasoning, ramble until the length limit cuts the answer, or hedge by listing several numbers in the answer box. Each style has a length, a layout reward, and a skill: the chance to be right on a prompt of difficulty \(d\) is \(\sigma(a - d)\), with \(d\) drawn from a normal distribution for each prompt. The starting policy mostly answers without tags, like the 0.5B model before training.
Each step samples 16 prompts with \(G\) answers each, computes rewards with (77.1), advantages with (77.2), and takes a policy-gradient step on the seven logits with a KL pull of weight \(\beta\) toward the start. “True accuracy” is what a human grader would say: right number, and a real answer (no hedging), whether or not the tags are used. It is computed on 200 fixed test difficulties.
How to read Figure 4
With the defaults (strict checker, \(\beta = 0.04\), \(w_\text{fmt} = 0.5\), \(G = 8\)), the tags are learned first: the styles without tags shrink, and in the first 50 steps the short tagged answers and the hedges grow too, so the true accuracy even dips from 0.370 to 0.336. Then long careful reasoning takes over: the true accuracy ends at 0.728 and the mean length grows from 161 to 244 tokens. Set \(w_\text{fmt} = 0\) and the dip disappears (the lowest value stays 0.370). Now click “Loose checker”: the batch reward ends even higher (about 1.29), but true accuracy falls to 0.018, because the policy learned to hedge. Raise \(\beta\) to 0.3: the KL stays lower and the policy keeps a mix of styles. Set \(G = 2\): almost half the groups have zero spread and learning slows.
The toy is honest about the shape of the effects, not about real numbers: real models have millions of “styles”, and real length growth also depends on the loss aggregation (session 70 explains the length bias of loss_type="grpo"). Here is a version with a real tiny network that writes tokens, so you can see GRPO work end to end in PyTorch. A small MLP must learn to answer “a + b” in the layout <a> d d </a> (two digits inside answer tags), with a correctness reward and a format reward, exactly like the lab. It runs on a CPU in about ten seconds and is deterministic.
import copy, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # vocab: 0-9 digits, 10 "<a>", 11 "</a>", 12 "x" (junk); 13 = start V, T, G, P = 13, 4, 8, 32 # vocab, completion length, group size, prompts per step OPEN, CLOSE = 10, 11 class TinyLM(nn.Module): # next-token logits from (prompt a, prompt b, previous token, position) def __init__(s): super().__init__() s.a, s.b = nn.Embedding(10, 64), nn.Embedding(10, 64) s.tok, s.pos = nn.Embedding(V + 1, 64), nn.Embedding(T, 64) s.mlp = nn.Sequential(nn.Linear(64, 128), nn.Tanh(), nn.Linear(128, V)) def forward(s, a, b, prev): # a, b: (N,); prev: (N, T) h = s.a(a)[:, None] + s.b(b)[:, None] + s.tok(prev) + s.pos.weight[None] return s.mlp(h) # (N, T, V) def prev_of(y): # shift right, start token = V return torch.cat([torch.full((y.size(0), 1), V), y[:, :-1]], 1) def rewards(a, b, y): # the verifiable reward: a checker, not a model fmt = ((y[:, 0] == OPEN) & (y[:, 3] == CLOSE)).float() s = a + b # 0..18 written as two digits, e.g. "07" ok = (y[:, 1] == s // 10) & (y[:, 2] == s % 10) return fmt * ok.float(), fmt # correctness counts only inside the tags def logp(model, a, b, y): lp = F.log_softmax(model(a, b, prev_of(y)), -1) return lp.gather(-1, y[..., None]).squeeze(-1) # (N, T) policy = TinyLM() ref = copy.deepcopy(policy).eval() for p in ref.parameters(): p.requires_grad_(False) opt = torch.optim.AdamW(policy.parameters(), lr=3e-3) beta, eps, mu = 0.04, 0.2, 2 for step in range(401): a = torch.randint(0, 10, (P,)).repeat_interleave(G) # each prompt G times b = torch.randint(0, 10, (P,)).repeat_interleave(G) with torch.no_grad(): # 1) sample a group per prompt y = torch.zeros(P * G, T, dtype=torch.long) for t in range(T): logits = policy(a, b, prev_of(y))[:, t] y[:, t] = torch.distributions.Categorical(logits=logits).sample() corr, fmt = rewards(a, b, y) r = corr + 0.5 * fmt # 2) weighted sum of rewards (77.1) rg = r.view(P, G) # 3) group-relative advantage (77.2) adv = ((rg - rg.mean(1, keepdim=True)) / (rg.std(1, keepdim=True) + 1e-4)).view(-1) old = logp(policy, a, b, y) ref_lp = logp(ref, a, b, y) for _ in range(mu): # 4) mu clipped updates on the batch new = logp(policy, a, b, y) ratio = torch.exp(new - old) pg = torch.min(ratio * adv[:, None], ratio.clamp(1 - eps, 1 + eps) * adv[:, None]) kl = torch.exp(ref_lp - new) - (ref_lp - new) - 1 # k3 estimator (77.4), always >= 0 loss = -(pg - beta * kl).mean() # all completions have T tokens opt.zero_grad(); loss.backward(); opt.step() if step % 50 == 0: zero = (rg.std(1) == 0).float().mean() print(f"step {step:3d} correct {corr.mean():.3f} format {fmt.mean():.3f}" f" KL {kl.mean():.3f} zero-std groups {zero:.2f}")
Output (PyTorch 2.8, CPU):
step 0 correct 0.000 format 0.000 KL 0.000 zero-std groups 1.00 step 50 correct 0.059 format 0.961 KL 1.464 zero-std groups 0.53 step 100 correct 0.180 format 0.969 KL 1.587 zero-std groups 0.31 step 150 correct 0.531 format 0.926 KL 2.286 zero-std groups 0.28 step 200 correct 0.504 format 0.961 KL 2.378 zero-std groups 0.38 step 250 correct 0.707 format 0.984 KL 1.944 zero-std groups 0.41 step 300 correct 0.801 format 0.965 KL 1.902 zero-std groups 0.59 step 350 correct 0.812 format 0.953 KL 2.317 zero-std groups 0.28 step 400 correct 0.879 format 0.973 KL 2.250 zero-std groups 0.59
Read it like a training log. At step 0, the random policy almost never produces the layout, so every group is all zeros (zero-std 1.00) and nothing is learned yet; a few lucky groups give the first signal. By step 50 the format is learned (0.961) but correctness is still 0.059: the easy reward is learned first. Correctness then climbs to 0.879. The zero-std share goes up again late in training, because many prompts are now solved by all 8 answers: the model has outgrown these problems. The KL rises to about 2 and stays there; \(\beta = 0.04\) holds it, but loosely.
Step 6: train with TRL
Here is the full training script. The comments explain each choice; Step 9 lists the experiments that change them.
# train_grpo.py: GRPO on GSM8K with two verifiable rewards. Usage: python train_grpo.py import json, torch from datasets import load_dataset from trl import GRPOConfig, GRPOTrainer from rewards import SYSTEM, gold_of, correctness_reward, format_reward MODEL = "Qwen/Qwen2.5-0.5B-Instruct" def to_row(ex): # TRL reads "prompt"; "gold" is passed to the rewards return {"prompt": [{"role": "system", "content": SYSTEM}, {"role": "user", "content": ex["question"]}], "gold": gold_of(ex["answer"])} train = load_dataset("openai/gsm8k", "main")["train"].shuffle(seed=0) train = train.map(to_row, remove_columns=["question", "answer"]) args = GRPOConfig( output_dir="out/grpo", max_steps=250, seed=0, learning_rate=2e-6, lr_scheduler_type="constant_with_warmup", warmup_steps=10, per_device_train_batch_size=16, # 16 completions per micro-batch = 2 prompts x 8 gradient_accumulation_steps=4, # 64 completions = 8 prompts per optimizer step num_generations=8, # G: the group size max_prompt_length=320, max_completion_length=384, temperature=1.0, beta=0.04, # KL weight (TRL default is 0: then no KL and no reference model) loss_type="grpo", # per-answer length normalization, as in DeepSeekMath reward_weights=[1.0, 0.5], # r = correctness + 0.5 * format bf16=True, gradient_checkpointing=True, logging_steps=5, save_steps=50, report_to="none") trainer = GRPOTrainer(model=MODEL, reward_funcs=[correctness_reward, format_reward], args=args, train_dataset=train) trainer.train() trainer.save_model("out/grpo/final") json.dump(trainer.state.log_history, open("out/grpo/log.json", "w"), indent=1)
Four remarks on the config.
- Learning rate. RL updates for LLMs use small learning rates: DeepSeekMath used \(10^{-6}\), and \(10^{-6}\) is also TRL’s default [Certain]. We start at \(2 \times 10^{-6}\) for a short run on a small model; this is a starting point, not a tuned value. Too high, and the policy can collapse in a few steps (the reward jumps, then the answers become repetitive or empty).
- Batch arithmetic. The number of completions per optimizer step must be divisible by \(G\): here \(16 \times 4 = 64\), which is 8 prompts with 8 answers each. On several GPUs, multiply by the number of processes.
- Two TRL defaults we change on purpose. TRL’s default
betais 0 and its defaultloss_typeis"dapo"(a token-level average) in version 0.24; we checked both in its source code. We set \(\beta = 0.04\) to watch the KL, and"grpo"to match the original paper and to see the length effects described in session 70. - Faster generation. With
use_vllm=True, TRL generates with vLLM, which is usually much faster thangenerate[Likely]. It needs thevllmpackage and some GPU memory for it; the rest of the script does not change.
- input: the instruct model, GSM8K train and test,
rewards.py - unit-test the reward functions on hand-written answers (Figure 1) // a reward bug is a goal bug
- before: greedy answers on the test set; record strict and loose accuracy, format rate, length; estimate \(p\) per prompt
- choose \(G\), \(B\), \(L_\text{max}\) with (77.5) and (77.6) (Figure 3)
- train with GRPO; every few steps, look at the five curves of Step 7 and at a few sampled answers
- after: the same evaluation with the same prompt, the same limit and greedy decoding
- compare row by row: fixed, broken, unchanged; read samples from each group
- write down what changed: format, math, length, or the checker
Step 7: watch reward, length and KL
TRL logs many numbers every logging_steps. Five of them tell you most of what is going on. This small script prints them from the saved log:
# show_log.py: the curves to watch. Usage: python show_log.py out/grpo/log.json
import sys, json
log = [r for r in json.load(open(sys.argv[1])) if "reward" in r]
cols = ["reward", "rewards/correctness_reward/mean", "rewards/format_reward/mean",
"completions/mean_length", "completions/clipped_ratio", "kl", "frac_reward_zero_std"]
print("step " + " ".join(f"{c.split('/')[-2] if c.count('/') == 2 else c.split('/')[-1]:>14s}" for c in cols))
for r in log:
print(f"{r['step']:4d} " + " ".join(f"{r.get(c, float('nan')):14.3f}" for c in cols))
| Logged name | What it is | Healthy | Warning sign |
|---|---|---|---|
rewards/format_reward/mean | mean format reward of the batch | rises fast in the first tens of steps, then stays near 1 | stays near 0: the prompt does not show the layout well enough; no signal |
rewards/correctness_reward/mean | share of sampled answers that are right | rises slowly and noisily | jumps much faster than the test accuracy: check for a checker loophole |
completions/mean_length | mean completion length in tokens | may first drop (rambling stops), then change slowly | grows to the limit; or collapses to a few tokens |
completions/clipped_ratio | share of completions cut at max_completion_length | low and falling | rising: answers are cut, so they lose the answer block and get 0 |
kl | mean k3 KL to the reference (only when \(\beta \gt 0\)) | grows slowly from 0 | a sudden jump: the policy is drifting fast, often with a reward hack or a too-high learning rate |
frac_reward_zero_std | share of groups with all rewards equal | well below 1, slowly rising as problems get solved | near 1: no learning signal; problems too hard or too easy |
entropy | mean token entropy of the policy | slowly decreasing | falling to near 0: the policy has stopped exploring |
Also read samples during training, not only at the end. A simple way: inside correctness_reward, append the completions and their rewards to a file every 20 calls. Ten minutes of reading tells you more than any curve: you will see the first answers that use the tags, the first hedges if your checker is loose, and the moment the reasoning gets longer or shorter.
Step 8: evaluate and read samples, before and after
Evaluate the model before training and after, with exactly the same prompt, the same token limit and greedy decoding (no sampling), so the only difference is the weights. Use the full test set of 1,319 problems.
# eval_gsm8k.py: greedy answers on GSM8K test. Usage: python eval_gsm8k.py <model path> <out.json> [n] import sys, json, torch from datasets import load_dataset from transformers import AutoTokenizer, AutoModelForCausalLM from rewards import SYSTEM, gold_of, correct, layout, loose path, out = sys.argv[1], sys.argv[2] n = int(sys.argv[3]) if len(sys.argv) > 3 else 1319 dev = "cuda" if torch.cuda.is_available() else "cpu" tok = AutoTokenizer.from_pretrained(path, padding_side="left") # left padding for batched generation model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16 if dev == "cuda" else torch.float32).to(dev).eval() test = load_dataset("openai/gsm8k", "main")["test"].select(range(n)) MAX_NEW = 384 # same limit as training rows = [] for i in range(0, n, 32): qs, sols = test[i:i + 32]["question"], test[i:i + 32]["answer"] msgs = [[{"role": "system", "content": SYSTEM}, {"role": "user", "content": q}] for q in qs] enc = tok.apply_chat_template(msgs, add_generation_prompt=True, padding=True, return_tensors="pt", return_dict=True).to(dev) with torch.no_grad(): o = model.generate(**enc, max_new_tokens=MAX_NEW, do_sample=False) for j, q in enumerate(qs): ids = o[j, enc["input_ids"].shape[1]:] ntok = int((ids != tok.pad_token_id).sum()) txt = tok.decode(ids, skip_special_tokens=True) g = gold_of(sols[j]) rows.append({"i": i + j, "q": q, "gold": g, "out": txt, "tokens": ntok, "cut": ntok >= MAX_NEW, "correct": correct(txt, g), "format": layout(txt), "loose": loose(txt, g)}) mean = lambda k: sum(float(r[k]) for r in rows) / n res = {k: round(mean(k), 4) for k in ["correct", "format", "loose", "tokens", "cut"]} print(path, json.dumps(res)) json.dump({"summary": res, "rows": rows}, open(out, "w"), indent=1)
Run python eval_gsm8k.py Qwen/Qwen2.5-0.5B-Instruct before.json and python eval_gsm8k.py out/grpo/final after.json. The trainer saves the tokenizer with the model, so the same script loads both.
We ran eval_gsm8k.py on Qwen2.5-0.5B-Instruct for the first 64 test problems (n = 64), on a CPU in float32, with greedy decoding. Your full GPU run in bfloat16 can differ a little. The results:
- Strict accuracy 0.016 (1 of 64). Loose accuracy 0.359 (23 of 64, standard error 0.060 by (77.7)).
- Format reward 0.025 on average: 60 answers use none of the four tags; 4 use only
<answer>and</answer>(0.4 each); none uses the think tags or the full layout. - Mean length 215 tokens; 6 of 64 answers (0.094) are cut at 384 tokens, some of them in a loop that repeats the same phrase.
So the model can already do a third of the math, but it writes the answer in its own style: “So, Janet makes $18 every day at the farmers’ market.” The only strict success ends with Therefore, the answer is:\n<answer> 83 </answer>, which earns correctness 1 and format 0.4, a total reward of 1.2. Even with the example in the system prompt, the instruct model rarely follows the layout under greedy decoding. Two lessons follow. First, the strict column will rise a lot after training, and most of that rise can be format: compare it with the loose column. Second, at the start the learning signal comes from the partial format credit and from the few sampled answers that happen to use tags, which is why frac_reward_zero_std in the first steps is worth checking.
Is the change real? An accuracy \(\hat p\) on \(n\) problems has a standard error
With \(\hat p = 0.4\) and \(n = 1{,}319\), \(\mathrm{SE} = 0.0135\): a 95% interval of about \(\pm 2.6\) points. Because before and after are scored on the same problems, a paired count is stronger: how many problems were fixed (wrong before, right after) and how many were broken (right before, wrong after). The script below prints these counts and then shows random samples from each group.
# compare.py: read samples before and after. Usage: python compare.py before.json after.json
import sys, json, random
a = json.load(open(sys.argv[1]))["rows"]; b = json.load(open(sys.argv[2]))["rows"]
groups = {"fixed (wrong -> right)": [], "broken (right -> wrong)": [],
"right both": [], "wrong both": []}
for x, y in zip(a, b):
key = {(0, 1): "fixed (wrong -> right)", (1, 0): "broken (right -> wrong)",
(1, 1): "right both", (0, 0): "wrong both"}[(int(x["correct"]), int(y["correct"]))]
groups[key].append((x, y))
for k, v in groups.items():
print(f"{k:26s} {len(v):5d}")
n01, n10 = len(groups["fixed (wrong -> right)"]), len(groups["broken (right -> wrong)"])
print(f"net change: {(n01 - n10) / len(a):+.3f} (McNemar counts: {n01} fixed, {n10} broken)")
print(f"mean tokens {sum(r['tokens'] for r in a)/len(a):.0f} -> {sum(r['tokens'] for r in b)/len(b):.0f}")
random.seed(0)
for k in ["fixed (wrong -> right)", "broken (right -> wrong)", "wrong both"]:
for x, y in random.sample(groups[k], min(2, len(groups[k]))):
print("\n=====", k, "| gold", x["gold"], "\nQ:", x["q"], "\n--- before ---\n", x["out"], "\n--- after ---\n", y["out"])
What did the model learn? Read at least 20 “fixed” samples and 10 “broken” ones, and sort each fixed sample into one of these causes:
- Format only. The before answer had the right number but outside the tags, or with extra words in the box. The after answer is the same math in the right layout. This is a real gain for the strict metric, but not a gain in math. The loose column shows it: if loose accuracy did not move, most of the strict gain is format.
- Fewer cut answers. Before, the answer rambled past the limit and lost its answer block; after, it finishes. Check the “cut” column.
- Better reasoning. The before answer made an arithmetic or setup mistake; the after answer does the steps correctly, for example it now checks a sum or keeps track of units. This is the gain you want.
- Luck. The two answers are very similar and the after answer happens to be right. With greedy decoding a small change of weights can flip an answer either way; the “broken” group measures how often.
A short GRPO run on a small instruct model often improves mostly format and finishing, and only a little the math itself [Likely]. That is not a failure: it is what the reward asked for. The question for your notes is how much of each.
Step 9: experiments
Change one thing at a time, keep the seed, the data order and the number of steps, and fill one more row of the table for each.
- No KL:
beta=0.0. The reference model is not loaded, which saves memory. Does the test accuracy change? Does the length drift more? - Loss aggregation:
loss_type="dr_grpo"or"dapo". Compare the length curves with the"grpo"run. Session 70 predicts that per-answer normalization makes long wrong answers cheaper. - No format reward:
reward_weights=[1.0, 0.0]. Count the steps until the format rate passes 0.9, and compare with the default. - Group size:
num_generations=4or 16, with the same number of completions per step. Watchfrac_reward_zero_std. - The loose checker: use
looseas the correctness reward. Then evaluate with the strict checker and read samples. Do you find hedges, or numbers placed at the end of the text? - Std over the batch:
scale_rewards="batch"divides by the spread of the whole batch instead of each group, which removes the difficulty bias of (77.2) [Likely].
Where this is used
| Task | Verifiable reward | Notes |
|---|---|---|
| Math word problems (GSM8K, MATH, competitions) | exact final number or expression, plus a format check | The setting of this lab, DeepSeekMath and DeepSeek-R1-Zero [Certain] |
| Code generation | hidden unit tests pass | Run the code in a sandbox with a time limit; tests can be gamed if they are weak [Likely] |
| Structured output (JSON, SQL, tool calls) | parses, matches a schema, or the query returns the right rows | Format rewards are often enough to fix layout errors of small models [Likely] |
| Instruction constraints (“answer in 3 bullet points”) | a rule checks each constraint | Tülu 3 used such verifiable constraints in its RLVR stage [Likely] |
| Video classification and action recognition with a VLM | predicted label equals the gold label | Video-R1 and similar works apply GRPO to video question answering with rule rewards [Likely] |
| Key-moment detection and temporal grounding in video | IoU of the predicted time span with the gold span | A continuous reward between 0 and 1; plus a format reward for the time stamps [Likely] |
| Video-RAG and video retrieval answers | the cited clip id or time stamp is the right one | The checker needs a gold clip per question; hard to build without labels [Speculation] |
Common mistakes
- An untested reward function. A loophole (the last number anywhere, several answer blocks, “24 or 72”) becomes the goal. Unit-test the checker on hand-written good and bad answers first.
- No signal at the start. If the model never uses the layout or never gets a problem right, every group has zero spread and nothing is learned. Check
frac_reward_zero_stdin the first steps; show an example in the prompt, add partial format credit, or use easier problems. - A completion limit that is too short. Cut answers lose their answer block and get 0, so the policy learns to be short instead of right. Watch
completions/clipped_ratio. - Trusting the training reward. The batch reward is measured on the training prompts with sampling and with your own checker. Evaluate on held-out test problems with greedy decoding, and read the answers.
- A different evaluation before and after. Another prompt, another token limit or sampling instead of greedy decoding makes the comparison meaningless.
- A batch that does not divide by \(G\). The number of completions per step must be a multiple of
num_generations; otherwise TRL stops with an error at the start [Likely]. - A learning rate from SFT. \(2 \times 10^{-5}\) is normal for SFT but usually far too high for RL on the same model [Likely]; start near \(10^{-6}\).
- Assuming the default settings. TRL’s defaults (no KL, the DAPO loss, the reward scaling) differ from the DeepSeekMath paper and change between versions. Print your config and save it with the run.
Interview questions
What makes a reward “verifiable”, and why is it attractive for RL on LLMs?basic
Write the GRPO advantage for a group, and compute it for rewards 1, 0, 0, 0, 0, 0, 0, 0.core
Why add a format reward if you only care about correctness?core
Your batch reward rises steadily, but the test accuracy is flat. List the causes you would check.advanced
What does frac_reward_zero_std tell you, and what do you do when it is near 1?core
What is the KL term in GRPO, and what happens with \(\beta = 0\)?core
With TRL’s default of one update per batch, does the PPO clip ever act? Why?advanced
num_iterations=1, the probabilities used for the ratio come from the same weights that are being updated, so \(\rho = 1\) at the moment of the gradient, which is inside the clip range. The gradient of \(\min(\rho \hat A, \operatorname{clip}(\rho)\hat A)\) at \(\rho = 1\) is \(\hat A \nabla \log \pi\), the plain policy gradient. The clip only matters when the same batch is used for several updates (num_iterations above 1) or when generation runs on slightly stale weights.How would you plan the group size for problems where the model’s pass rate is about 10%?core
After training, the answers are much longer. Is that good?advanced
loss_type="dr_grpo", which removes the per-answer normalization.How do you know what the model learned from GRPO?core
Why use an instruct model rather than a base model for this lab?basic
Exercises
- Compute the advantages for the group rewards 1.5, 1.5, 1.5, 1.5, 0.5, 0.5, 0.5, 0.5. Then for 1.5 eight times.
- Score these answers with
correctandlayout(gold 18): (a)<think>16-3-4=9, 9*2=18</think><answer>18</answer>(b)<answer>18 dollars</answer>(c)<think>...</think> <answer>-18</answer>. Check them in Figure 1. - Your evaluation before training gives a pass rate of 0.05 on hard problems. What share of groups give a correctness signal at \(G = 8\) and at \(G = 64\)?
- How many tokens at most are generated by 400 steps with \(B = 16\), \(G = 8\), \(L_\text{max} = 512\)?
- Your accuracy goes from 0.30 to 0.33 on the 1,319 test problems. Is this clearly more than noise? What extra numbers would make the case stronger?
- In Figure 4, find the setting with the highest final batch reward and explain why it is the worst policy.
Answerscheck
Key takeaways
- A verifiable reward is a program: here, correctness (exactly one answer block holding exactly the gold number) plus a small format reward (77.1). Unit-test it before training, because the policy will learn its loopholes.
- GRPO compares answers within a group (77.2); groups where every reward is equal teach nothing. Plan \(G\) with (77.5) and the generation budget with (77.6).
- With one update per batch, the GRPO loss (77.3) is a plain policy gradient with a KL pull (77.4); \(\beta = 0.04\) lets you watch the KL, \(\beta = 0\) saves the reference model.
- Watch the reward per function, the length, the clipped ratio, the KL and
frac_reward_zero_std; a rising reward alone proves nothing (Figure 4, loose checker). - Evaluate before and after with the same prompt, limit and greedy decoding, use the standard error (77.7) and paired fixed/broken counts, and read samples to see whether the model learned format, finishing or math.
Key formulas
References for this session
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Hugging Face (code). TRL library
- Raschka, Build a Reasoning Model (From Scratch), chapter 6: Training reasoning models with RL
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective
- Hugging Face (code). TRL library
- Hugging Face (2026). TRL documentation: GRPO Trainer
- Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems (GSM8K)
- Qwen Team (2024). Qwen2.5 Technical Report
- Raschka, Build a Reasoning Model (From Scratch), chapter 6: Training reasoning models with RL
- Raschka, Build a Reasoning Model (From Scratch), chapter 7: Improving GRPO