Policy learning words: on-policy, off-policy, online, offline
Papers say “GRPO is on-policy”, “DPO is offline” and “we ran online DPO”. This session explains these words with one question each: who wrote the training data, and can we get new data while we train? We meet the cast of policies (current, behavior, old, reference), see with numbers why learning from someone else’s samples gets hard for long answers, and watch iterative DPO close part of the gap to online RL in a small simulation.
What you will learn
- The four policies in every post-training paper: the current policy, the behavior policy that wrote the data, the old policy of PPO, and the frozen reference policy, plus rollout and trajectory
- On-policy versus off-policy: whether the data came from the policy you are updating, and why the policy gradient needs fresh samples
- Importance sampling as the bridge, and why its weights explode for long answers: the effective sample size falls like exp(−Tσ²)
- Online versus offline: whether you can collect new samples and labels during training, and the 2 × 2 map that places SFT, RFT, PPO, GRPO, offline DPO and online DPO
- Why offline DPO on a fixed dataset cannot learn about answers the data never shows, and how online DPO and iterative DPO (Llama 3, Self-Rewarding) fix part of it
- How to read any new method in four questions, and the cost trade-off between fresh samples and reused data
Why this matters
Read three post-training papers and you will meet the same short words again and again: “GRPO is on-policy”, “DPO is an offline method”, “we use online DPO”, “PPO is slightly off-policy”, “the reference policy is the SFT model”, “we collect rollouts from the behavior policy”. Each word answers a simple question about the training data. Who wrote it? When? Can we get more of it while we train?
These questions matter because they decide cost and quality. Fresh samples from the current model are expensive: for every update you must generate long answers, and generation is the slow part of RL for LLMs (session 66). Reusing old or fixed data is cheap, but the model learns from answers it would no longer write, and the math has to correct for that, or it silently stops being correct. The choice between these two worlds is one of the main design decisions in modern post-training.
Real systems show the full range. DeepSeekMath sorts methods by where their data comes from: “RFT and DPO follow the offline style, while Online RFT and GRPO follow the online style”, and it reports that online sampling wins clearly in the later stage of training. [Certain] Llama 3 ran six rounds of SFT followed by DPO, and for DPO it used mainly “the most recent batches of preference data collected using the best performing models” from the previous rounds. [Certain] That is DPO made iterative: offline inside one round, online across rounds. Online DPO samples two answers from the current model at every step and asks an LLM judge which one is better. [Certain]
This session is a dictionary, like session 63, but for the data side of RL. It sits between session 70 (an on-policy method) and session 72. After it you should be able to place any new method on a 2 × 2 map in under a minute.
The cast: four policies, rollouts and trajectories
Recall from session 63: a policy \(\pi(y \mid x)\) gives a probability to every answer \(y\) for a prompt \(x\). For an LLM, it is the model itself, one token at a time. A rollout is the act of running a policy to produce an answer, and the result, the prompt plus the answer plus its score, is a trajectory. In LLM papers the two words are used almost the same way: “we collect 64 rollouts per prompt” means 64 sampled answers.
A post-training run usually has up to four different policies at once. They are often copies of the same network at different moments, which is why they are easy to mix up.
- Current policy \(\pi_\theta\): the model we are training right now. Its weights \(\theta\) change at every optimizer step. It is also called the target policy: the policy we want to evaluate and improve.
- Behavior policy \(\mu\): the policy that actually wrote the training data. It can be the current policy, an older copy of it, a different model (GPT-4, a human), or a mix of many models.
- Old policy \(\pi_{\theta_\text{old}}\): in PPO and GRPO, a frozen copy of \(\pi_\theta\) taken at the moment a batch was generated. It is the behavior policy of that batch. It is kept only to compute the ratio \(\pi_\theta / \pi_{\theta_\text{old}}\) (session 66).
- Reference policy \(\pi_\text{ref}\): a frozen model, usually the SFT model, that the training must not move too far from. It appears in a KL penalty (PPO, GRPO) or inside the loss (DPO). It never generates data in standard PPO or DPO.
The goal that ties them together is the KL-regularized objective of RLHF and RLVR (session 66, session 70):
- Read the symbols
- \(\mathcal{D}\)
- the distribution of prompts; prompts are fixed text, only the answers are sampled
- \(y \sim \pi_\theta(\cdot \mid x)\)
- the answer is sampled from the current policy: this is the part that makes the objective on-policy
- \(r(x, y)\)
- the score of an answer: a reward model, a verifier, or a judge
- \(\beta\)
- how strongly we pull back toward the reference; typical values are 0.01 to 0.1 [Likely]
- \(\pi_\text{ref}\)
- the frozen anchor, usually the SFT model
Note the important detail: the expectation is over answers from \(\pi_\theta\) itself. When \(\theta\) changes, the distribution of answers that the objective cares about changes too. Figure 1 walks through one training iteration and shows which policy plays which role at each moment.
How to read Figure 1
Follow the “data age” number. Right after the rollout the data is brand new: age 0, behavior policy = current policy, and the importance ratio is exactly 1. After a few mini-batch steps the current policy has moved, so the same data is now a few steps old: that is the small off-policy part of PPO that clipping controls. In the last step (offline DPO), the behavior policy is some other model and the age only grows. The reference box never writes data in any of these methods; it only appears inside the KL or the loss.
On-policy: learn from your own fresh samples
A method is on-policy when every update uses data written by the current policy: \(\mu = \pi_\theta\). Spinning Up puts it this way: policy optimization “is almost always performed on-policy, which means that each update only uses data collected while acting according to the most recent version of the policy.” [Certain]
Why is this needed? Look at the policy gradient of the first term of (71.1), from session 64:
The right side is an average over answers from \(\pi_\theta\). If we fill it with answers from a different policy, we get the average under the wrong distribution: a biased gradient. The bias is not small in general. Answers that the current policy now likes a lot but the old policy rarely wrote are missing from the data, so the gradient never sees them.
REINFORCE and GRPO with one update per batch are on-policy in the strict sense. DeepSeekMath states: “The policy model only has a single update following each exploration stage.” [Certain] Then \(\pi_\theta = \pi_{\theta_\text{old}}\) at the moment of the update, the ratio is 1, and the gradient is exactly (71.2). PPO with several mini-batches per batch is on-policy in spirit: each batch is fresh when it is generated, but after the first mini-batch step it is a few steps old. The clipped ratio corrects for that small gap and stops the update when the gap gets too large (session 66).
The price of on-policy learning is sample cost: each sample is used for one update (or a few), then thrown away. Spinning Up notes that Q-learning methods, which are off-policy, “can reuse data more effectively than policy optimization techniques”. [Certain] For LLMs, a rollout of a long reasoning answer can take far longer than the backward pass that learns from it. [Likely]
Off-policy: learn from someone else’s samples
A method is off-policy when the data comes from a behavior policy different from the current one: \(\mu \ne \pi_\theta\). Spinning Up describes Q-learning this way: “each update can use data collected at any point during training, regardless of how the agent was choosing to explore the environment.” [Certain] The data may come from an older copy of the model (a replay buffer, or a late batch in asynchronous training), or from a completely different model.
Importance sampling: the bridge
If you must average under \(\pi\) but you only have samples from \(\mu\), you can weight each sample by how much more likely it is under \(\pi\) than under \(\mu\). This is importance sampling:
- Read the symbols
- \(\mu\)
- the behavior policy that wrote the samples
- \(\pi\)
- the target policy we want the average for (the current policy)
- \(w(y)\)
- the importance weight: above 1 if \(\pi\) likes the answer more than \(\mu\) did, below 1 if less
- \(f(y)\)
- anything we want to average: a reward, or a gradient term as in (71.2)
The identity needs one condition: \(\mu(y) \gt 0\) wherever \(\pi(y) \gt 0\). If the behavior policy never writes an answer, no weight can tell you anything about it. This is called coverage, and it is the deep problem of offline methods below.
A prompt has two possible answers. Answer A gets reward 0 and answer B gets reward 1. The behavior policy wrote A with probability 0.95 and B with 0.05. The current policy likes both equally: \(\pi(A) = \pi(B) = 0.5\), so its true expected reward is 0.5.
The plain average of the logged rewards estimates \(\E_\mu[r] = 0.05\): badly wrong. With weights \(w(A) = 0.5/0.95 = 0.526\) and \(w(B) = 0.5/0.05 = 10\), the weighted average is \(0.05 \times 10 \times 1 = 0.5\). Correct on average.
But look at the variance. One weighted sample \(w\,r\) is 10 with probability 0.05 and 0 otherwise, so \(\E[(wr)^2] = 0.05 \times 100 = 5\) and the variance is \(5 - 0.25 = 4.75\). One on-policy sample (a 0 or 1 coin with mean 0.5) has variance 0.25. Each off-policy sample is worth about \(4.75 / 0.25 = 19\) times less: you need about 19 times more samples for the same accuracy.
Why weights explode for long answers
For a language model, an answer is a sequence of \(T\) tokens. Its probability is a product of token probabilities, so its importance weight is a product of \(T\) token ratios:
Even if each token ratio is close to 1, a product of hundreds of them is not. Example: token ratios 1.1, 0.9 and 1.2 give \(w = 1.188\), a log-weight of 0.172. Over 1,000 tokens such small changes add up to weights of \(10^{\pm 5}\) and more. A few samples then carry almost all of the weight, and the rest count for nothing. We measure this with the effective sample size (ESS):
If all weights are equal, ESS \(= N\). If one weight dominates, ESS \(\approx 1\). For example, weights \((2, 1, 0.5, 0.5)\) give ESS \(= 16 / 5.5 = 2.91\) out of 4. A simple model shows how fast it falls with length. Suppose each token log-ratio \(\delta_t\) is an independent normal number with spread \(\sigma\) and mean \(-\sigma^2/2\) (the mean that keeps \(\E_\mu[w] = 1\), as it must be). Then the log-weight is normal with variance \(T\sigma^2\), and
- Read the symbols
- \(T\)
- the answer length in tokens
- \(\sigma\)
- how much the two policies disagree per token (the spread of the token log-ratio)
- \(\KL(\mu \,\|\, \pi)\)
- the sequence-level KL between the policies; in this model it is \(T\sigma^2/2\), so it grows linearly with length
With \(\sigma = 0.1\) (per-token KL of only 0.005 nats), an answer of 100 tokens keeps \(e^{-1} = 36.8\%\) of its samples. At 1,000 tokens only \(e^{-10} = 4.5 \times 10^{-5}\) is left: out of 10,000 samples, less than one is effective. This is the exact reason why LLM methods never use the full sequence weight (71.4). PPO and GRPO use per-token ratios and clip them, which accepts a small bias to avoid the explosion (session 66). And it is why “just reweight the old data” is not a free way to make any method off-policy. Try it in Figure 2.
How to read Figure 2
Start at \(\sigma = 0.1\), \(T = 100\): the histogram is a hill around 0 and about a third of the samples count. Move \(T\) to 1,000: the hill becomes very wide, a handful of answers on the far right carry nearly all the weight, and the theory says ESS/N is about \(4.5 \times 10^{-5}\). The measured value is then too optimistic, because the rare answers with huge weights were never drawn: with heavy-tailed weights, the sample itself hides how bad things are. Set \(\sigma = 0\) (the same policy) and every weight is 1: that is on-policy data. The figure samples the sum of the \(T\) token log-ratios directly; it has exactly the normal distribution of the model behind (71.6).
Reusing one batch: watch the ratios drift
Figure 2 drew the weights from a formula. Figure 3 makes them by training. It uses a small model that answers four sums (“2+3=”, “4+3=”, “6+2=”, “3+3=”) with two tokens: a number, then either the closing tag “</a>” or a period. A program scores the answer: 2 for the right number with a closed tag, 1 for a wrong number with a closed tag, 0 without the tag. We sample one batch of 8 answers per prompt and then train on this same batch again and again, one gradient step per epoch, the way PPO and GRPO reuse a batch for several epochs. The advantages are the group-normalized rewards of session 70.
In epoch 1 the policy that wrote the data and the policy being trained are the same, so every ratio is 1: on-policy. After one step they differ, and the batch is now off-policy data. The figure tracks four things after each epoch: the true gain in expected reward (computed exactly from the probabilities), the gain that the batch itself claims through the importance-weighted average (71.3), the KL between the old and the new policy, and the effective sample size (71.5). Switch between the PPO clip and plain importance weights to see why the clip exists.
- \(\pi_{\theta_\text{old}} \leftarrow \pi_\theta\); sample \(G\) answers per prompt // on-policy now
- group advantages \(\hat A\); store \(\log \pi_{\theta_\text{old}}\)
- for epoch \(= 1 \dots K\) on the same answers:
- \(\rho_t = \pi_\theta / \pi_{\theta_\text{old}}\) per token; \(w = \prod_t \rho_t\) // (71.4)
- clip: \(\min(\rho \hat A, \operatorname{clip}(\rho)\hat A)\); no clip: \(\rho \hat A\)
- \(\theta \leftarrow \theta + \alpha g\) (one full-batch step)
- measure true gain, estimate (71.3), KL, ESS (71.5)
- batch used up: collect a fresh one
How to read Figure 3
Choose Fast and press Play with the clip (seed 1). After 5 epochs about 69% of the tokens are clipped, and from then on the policy barely moves: after 40 epochs the KL from the old policy is 0.063, the ESS/N is 0.79 and the largest weight is 1.96. The batch says the gain is 0.270; the truth is 0.167. Now choose “No clip” and play again. The weights spread every epoch; after 10 epochs the largest is 11.6 and the ESS/N is 0.29; after 40 epochs the KL is 4.38 nats, the batch claims a gain of 6.17 (impossible: the reward cannot exceed 2) while the true gain is 0.737. Look at the policy panel: the policy is now nearly certain of answers it saw in this one batch, such as “2</a>” for “3+3=”, because the batch held no right answer for that prompt. Press “Collect a fresh batch”: the weights jump back to 1, because fresh data is on-policy again.
Two lessons come out of Figure 3. First, data reuse is cheap but not free: each extra epoch makes the batch more off-policy, and the batch’s own estimate of progress becomes more optimistic than the truth. Second, the clip is what keeps reuse safe. It does not fix the weights; it stops training on tokens whose ratio has already moved past \(1 \pm \epsilon\) in the direction the advantage wants. So after a few epochs most of the batch stops contributing, which is exactly when a fresh batch is needed. This is why PPO uses a few epochs per batch and why many GRPO runs use one.
Online and offline: can you get new data?
The second pair of words is about a different question. A method is online if, during training, it can collect new samples and new scores: generate answers and get them rewarded or labeled. It is offline if it learns only from a dataset that was collected before training began. Levine et al. define offline RL as “reinforcement learning algorithms that utilize previously collected data, without additional online data collection.” [Certain]
The two questions are related but not the same:
- On-policy implies online. To always have data from the current policy, you must keep generating it.
- Offline implies off-policy, except at the very first step. Once the first update is made, the fixed data was written by some other policy.
- Online does not imply on-policy. An online method can still learn from data that is a little old: a replay buffer, several PPO epochs, or asynchronous training where generation runs one step behind learning. Asynchronous RLHF (Noukhovitch et al.) trains on older samples while new ones are generated, and found online DPO to be “most robust to off-policy data”. [Certain]
So there are three useful boxes, not four: on-policy and online, off-policy and online, off-policy and offline. A fourth label, semi-online or iterative, is used for methods that are offline inside a round but collect new data between rounds. Figure 4 places the common methods.
How to read Figure 4
The bottom-left cell stays empty: data that is fixed before training cannot keep up with a policy that changes. Notice that the same name can move between cells. DPO on a fixed dataset is bottom-right; the same loss with pairs sampled from the current model at each step (online DPO) is top-left. The loss did not change; only the data pipeline did. That is why these words describe a training setup, not a loss function.
DeepSeekMath ran exactly this comparison on its math data. RFT (rejection-sampling fine-tuning: sample answers from the SFT model once, keep the correct ones, fine-tune on them) is offline. Online RFT does the same with samples from the current policy. “Online RFT is comparable to RFT in the early stage of training” but “gains an absolute advantage in the later stage”; the authors explain that later on, the policy’s answers differ more from the SFT model’s answers, so fresh samples matter more. [Certain]
Online and offline side by side
Figure 5 runs the same toy as Figure 3 in two ways, with the same seed and the same update rule (group advantages, one epoch of clipped mini-batch steps per iteration). The online run samples a fresh group of 8 answers per prompt from its current policy at every iteration. The offline run gets one dataset D of 8 answers per prompt, written by the reference policy before training, and trains on it at every iteration. Because both runs use the same seed, D is exactly the online run’s first batch: the two runs start from identical data and differ only in whether they may ask for more.
The offline ratio is \(\pi_\theta / \pi_\text{ref}\), because the reference wrote D. You can choose whether the offline run clips this ratio, like PPO does against its old policy, or uses plain importance weights. Neither choice fixes the real problem. With the clip, the offline policy is held within about \(\pm 20\%\) of the data-writing policy and barely learns. Without it, the policy moves far, but only toward what D happens to contain.
- for iteration \(= 1 \dots 60\):
- online: sample 8 new answers per prompt from \(\pi_\theta\); offline: take the fixed \(D\)
- \(\hat A\) from each group’s rewards // (70.1)
- one epoch of clipped steps; \(\rho = \pi_\theta / \mu\), \(\mu = \pi_{\theta_\text{old}}\) online, \(\pi_\text{ref}\) offline
- log accuracy, KL to \(\pi_\text{ref}\), ESS of \(D\) // (71.5)
How to read Figure 5
Look at D for seed 1 first: “2+3=” and “3+3=” have no right answer in it. Press “Run to the end”. The online run reaches accuracy 0.972 by iteration 10 and 0.998 by iteration 60. The offline run with plain weights ends at 0.249 with a KL of 3.57 nats and an ESS/N of 0.14: it is very sure of “9</a>” for “2+3=” and “2</a>” for “3+3=”, the best answers its data offers. It even misses “4+3=”, where D does contain “7</a>”: D also holds the wrong “7.” and two “5</a>”, which together support 5 about as much as 7, and the offline run cannot ask for more samples to settle it. Switch to the clipped offline run: it stays close to the reference (KL 0.35) and ends at 0.158. Try other seeds: the offline result always depends on what D happens to contain.
Offline DPO: what a fixed dataset cannot tell you
DPO (session 68) optimizes a loss on a fixed set of preference pairs \(\mathcal{P} = \{(x, y_w, y_l)\}\), where \(y_w\) is the preferred (“winning”) answer and \(y_l\) the rejected one:
- Read the symbols
- \(\mathcal{P}\)
- the preference dataset; in offline DPO it is collected once, before training
- \(y_w, y_l\)
- the chosen and the rejected answers; they were written by the behavior policy \(\mu\) of the dataset, not by \(\pi_\theta\)
- \(\pi_\text{ref}\)
- the frozen reference; it appears inside the loss, but it does not generate anything
- \(\sigma\)
- the logistic sigmoid; \(\beta\) controls how far from \(\pi_\text{ref}\) the solution can go
No sample is drawn from \(\pi_\theta\) anywhere in (71.7). So DPO on a fixed dataset is offline and off-policy. Three different models can be involved: the dataset’s behavior policy (often many models at once), the reference (the SFT model), and the current policy. The DPO paper itself notices the gap: “Since the preference datasets are sampled using \(\pi^\text{SFT}\), we initialize \(\pi_\text{ref} = \pi^\text{SFT}\) whenever available”, and if not, they first fine-tune on the preferred answers to “mitigate the distribution shift”. [Certain] The derivation of DPO assumes the answers come from the reference; public datasets often break this assumption.
What goes wrong? The loss only talks about answers in the dataset. It says “raise \(y_w\) relative to \(y_l\)”. It says nothing about the many other answers that the policy may start to write as it moves. If the policy starts to put probability on a new kind of answer that no pair covers, nothing in the loss will push it down. Xu et al. argue from theory and experiments that DPO “may have fundamental limitations” of this kind and report that a well-tuned PPO beat it across their testbeds. [Certain] Tajwar et al. report that methods which use on-policy samples, or push down the probability of bad answers, “outperform offline and maximum likelihood objectives”. [Certain] Not every study agrees on the size of the gap, and offline DPO is still very widely used because it is cheap and stable. [Likely]
Online and iterative DPO: mixing the two worlds
The fix is to keep the DPO loss and change the data pipeline. In online DPO (Guo et al., “online AI feedback”, OAIF), each training step samples “two responses from the current model” for each prompt, an LLM annotator chooses the better one, and one DPO step is taken on these fresh pairs. [Certain] The objective becomes
- Read the symbols
- \(\operatorname{sg}[\pi_\theta]\)
- “stop gradient”: the answers are sampled from the current weights, but we do not differentiate through the sampling
- \(y_w, y_l\)
- \(y_1\) and \(y_2\) after the judge has said which one is better
- \(h_\theta\)
- the DPO margin inside the sigmoid of (71.7)
This is on-policy and online. It needs a judge that can label new pairs at training speed: an LLM judge, a reward model, or a verifier. Human labels are too slow for every step.
Iterative DPO is the middle ground. Training runs in rounds \(k = 0, 1, 2, \dots\). In each round you generate answers with the current model, label pairs with a reward model or judge, and then run ordinary offline DPO on that round’s data:
- Read the symbols
- \(\theta_k\)
- the weights at the start of round \(k\); they write the data of that round (the behavior policy)
- \(\mathcal{P}_k\)
- the new preference pairs of round \(k\)
- \(\pi_{\text{ref},k}\)
- the reference of round \(k\): either kept at the SFT model, or reset to \(\pi_{\theta_k}\) each round (an option, not a rule)
Inside a round, the data gets older with every step, as in offline DPO. Across rounds, it is refreshed, as in online RL. With one round, this is offline DPO on data from the starting model. With one step per round, it is online DPO. Xiong et al. propose iterative DPO for the online setting and describe offline methods as lacking “strategical exploration”. [Certain] Self-Rewarding Language Models run three iterations where the model also acts as its own judge. [Certain] Llama 3’s six rounds use the same pattern at scale, with a reward model choosing pairs. [Certain] Tülu 3 builds its DPO data with an “on-policy data curation pipeline” that includes answers from its own SFT model, and reports that on-policy data improves DPO. [Certain]
Figure 6 runs this on a toy problem where everything can be computed exactly. One prompt has 10 possible answers. Answer \(k\) has true reward \(k/9\), so answer 9 is best. The reference policy prefers the bad answers: its logits are \(-0.6k\), so it writes answer 0 with probability 0.452 and answer 9 with only 0.002, and its expected reward is 0.132. A Bradley-Terry judge labels pairs. The total budget is the same for every setting: 480 sampled pairs and 300 gradient steps, split evenly over the rounds (rounded down, so 8 rounds get 8 × 37 = 296 steps). Only the number of rounds changes: 1 round is offline DPO, 16 rounds is close to online DPO.
How to read Figure 6
With 1 round (offline), most pairs compare answers 0, 1 and 2, because that is what the reference writes. The loss learns their order well but sees the good answers rarely: only about 13% of the pairs contain an answer with reward above 0.5. With more rounds, the policy moves up first, then samples better answers, then learns about them: the share of informative pairs rises to about 22%. Notice also the KL to the reference: iterative training moves further away. Raise \(\beta\) with a fixed reference and the gain shrinks, because the anchor holds the policy near answers it already knew. Tick the reset box: each round’s reference is the last policy, so \(\beta\) limits only the step per round. This is a toy. It shows the mechanism, not the size of the effect on real LLMs.
Reference policy: expected reward 0.132. Offline DPO (1 round, 480 pairs, 300 steps): expected reward 0.290, KL to the reference 0.803 nats; 13.3% of the pairs contain an answer with reward above 0.5. Iterative DPO with 8 rounds (60 pairs and 37 steps per round): expected reward 0.363, KL 1.025; 22.1% of the pairs contain such an answer. With 16 rounds: 0.363. Same label budget, almost the same number of gradient steps (296 instead of 300), and the gain over the reference is \((0.363 - 0.132) / (0.290 - 0.132) = 1.46\) times larger (46% more), only because the data followed the policy.
Iterative and online DPO cost more than offline DPO: you need a generation engine and a judge in the loop, as in PPO. But they keep two DPO advantages: no critic model and a simple, stable loss. [Likely] This is why many recent recipes use them.
The algorithms
- input: prompts \(\mathcal{D}\); policy \(\pi_\theta\); frozen \(\pi_\text{ref}\); scorer \(r\); updates per batch \(M\)
- for iteration \(k = 1, 2, \dots\):
- \(\theta_\text{old} \leftarrow \theta\) // snapshot: the behavior policy of this batch
- sample prompts \(x\); rollouts \(y \sim \pi_{\theta_\text{old}}(\cdot \mid x)\) // fresh data, age 0
- score \(r(x, y)\); store \(\log \pi_{\theta_\text{old}}(y_t)\) and \(\log \pi_\text{ref}(y_t)\) per token
- compute advantages (critic + GAE for PPO, group normalization for GRPO)
- for \(m = 1 \dots M\): // \(m = 1\): exactly on-policy; \(m > 1\): data is \(m - 1\) steps old
- per-token ratio \(\pi_\theta / \pi_{\theta_\text{old}}\), clipped; KL to \(\pi_\text{ref}\); gradient step
- throw the batch away // on-policy: data is not reused next iteration
- input: prompts \(\mathcal{D}\); SFT model \(\pi_\text{SFT}\); judge (reward model, verifier or LLM); rounds \(K\); \(\beta\)
- \(\pi_{\theta_0} \leftarrow \pi_\text{SFT}\); \(\pi_\text{ref} \leftarrow \pi_\text{SFT}\)
- for round \(k = 0 \dots K-1\):
- for each prompt: sample \(n\) answers from \(\pi_{\theta_k}\) // behavior policy = current policy at round start
- judge them; pick a pair (for example best vs worst) \(\rightarrow \mathcal{P}_k\)
- (optional) \(\pi_\text{ref} \leftarrow \pi_{\theta_k}\); (optional) mix in some pairs from earlier rounds
- train with the DPO loss (71.7) on \(\mathcal{P}_k\) for a few epochs \(\rightarrow \theta_{k+1}\) // offline inside the round
- evaluate; stop if quality stops rising or the KL gets too large
- note: \(K = 1\) is offline DPO; one gradient step per round with fresh pairs is online DPO (71.8)
The PyTorch code below checks formula (71.6) numerically and then runs the toy of Figure 6 with torch: one call per number of rounds. It runs in a few seconds on a CPU. The random numbers differ from the JavaScript in the figure, so the results differ slightly, but the trend is the same.
import math, torch import torch.nn.functional as F torch.manual_seed(0) # 1) Importance weights and effective sample size, formulas (71.4) to (71.6) def ess_fraction(logw): w = torch.softmax(logw, dim=0) # normalized weights, stable in log space return (1.0 / (w * w).sum() / logw.numel()).item() N, sigma = 4000, 0.1 for T in [1, 10, 100, 1000]: eps = torch.randn(N, T) * sigma - sigma**2 / 2 # per-token log-ratios log(pi/mu) logw = eps.sum(dim=1) # sequence weight = product of token ratios print(f"T={T:5d} ESS/N measured={ess_fraction(logw):.3f} theory={math.exp(-T*sigma**2):.2g}") # 2) A toy "prompt" with K possible answers: answer k has true reward k/(K-1). # The reference (SFT) policy prefers the bad answers. K = 10 reward = torch.arange(K) / (K - 1) z_ref = -0.6 * torch.arange(K, dtype=torch.float) # reference logits def collect_pairs(logits, n, gen): """Sample n answer pairs from a policy and label them with a Bradley-Terry judge.""" p = torch.softmax(logits, 0) a = torch.multinomial(p, n, replacement=True, generator=gen) b = torch.multinomial(p, n, replacement=True, generator=gen) keep = a != b a, b = a[keep], b[keep] p_a_wins = torch.sigmoid(5.0 * (reward[a] - reward[b])) a_wins = torch.rand(len(a), generator=gen) < p_a_wins return torch.where(a_wins, a, b), torch.where(a_wins, b, a) # (chosen, rejected) def dpo_loss(z, z_ref_now, yw, yl, beta): logp, logp_ref = F.log_softmax(z, 0), F.log_softmax(z_ref_now, 0) margin = beta * ((logp[yw] - logp_ref[yw]) - (logp[yl] - logp_ref[yl])) # formula (71.7) return -F.logsigmoid(margin).mean() def train(rounds, beta=0.1, total_pairs=480, total_steps=300, lr=0.1, update_ref=False, seed=1): gen = torch.Generator().manual_seed(seed) z = z_ref.clone().requires_grad_(True) ref = z_ref.clone() opt = torch.optim.SGD([z], lr=lr / beta) for _ in range(rounds): yw, yl = collect_pairs(z.detach(), total_pairs // rounds, gen) # behavior policy = current policy if update_ref: ref = z.detach().clone() # new anchor each round for _ in range(total_steps // rounds): opt.zero_grad() dpo_loss(z, ref, yw, yl, beta).backward() opt.step() p = torch.softmax(z.detach(), 0) kl = (p * (p.log() - F.log_softmax(z_ref, 0))).sum() return (p * reward).sum().item(), kl.item() print("reference expected reward:", round((torch.softmax(z_ref, 0) * reward).sum().item(), 3)) for rounds in [1, 2, 4, 8, 16]: # rounds = 1 is offline DPO; more rounds = closer to online DPO res = [train(rounds, seed=s) for s in range(1, 9)] er = sum(r[0] for r in res) / 8; kl = sum(r[1] for r in res) / 8 print(f"rounds={rounds:2d} expected reward={er:.3f} KL to reference={kl:.3f}")
Output with PyTorch 2.8 on a CPU: ESS/N measured 0.990, 0.903, 0.362 and 0.008 for \(T\) = 1, 10, 100 and 1,000 (theory 0.99, 0.90, 0.37 and \(4.5 \times 10^{-5}\)); the measured value at \(T = 1000\) is far too optimistic, as Figure 2 explains. Then the reference expected reward 0.132, and for 1, 2, 4, 8 and 16 rounds: 0.288, 0.326, 0.338, 0.350 and 0.351.
How to read any new method: four questions
| Question | Word it decides | Example answers |
|---|---|---|
| Who wrote the data of this update? | on-policy / off-policy | GRPO: the current weights. Offline DPO: other models, long ago. |
| Can the method get new samples and labels during training? | online / offline | PPO: yes, every iteration. SFT: no. |
| How old is the data, in optimizer steps? | “how off-policy” | GRPO with one update: 0. PPO with 4 mini-batches: 0 to 3. Async: + 1 iteration. |
| Where does the reference policy appear? | none (separate question) | PPO / GRPO: KL penalty. DPO: inside the loss. SimPO: nowhere. |
The third question matters because “off-policy” is a matter of degree. PPO with four mini-batch steps per batch is a tiny bit off-policy; a replay buffer of the last 100 iterations is very off-policy; a public preference dataset written by other models is completely off-policy. The ESS formula (71.6) gives the intuition: what matters is the KL between the policy that wrote the data and the current one.
Where this is used
| System or task | Data setup | Why |
|---|---|---|
| InstructGPT, PPO-RLHF (session 66) | Online, on-policy in spirit: fresh rollouts each iteration, a few mini-batch steps per batch | The policy gradient needs current samples; clipping handles the small lag |
| DeepSeekMath / DeepSeek-R1 GRPO (session 70) | Online, on-policy; DeepSeekMath does one update per exploration stage | Verifiable rewards are cheap to compute for any new answer |
| Zephyr-style DPO on a public preference set (session 68) | Offline, off-policy: pairs written by other models before training [Likely] | Cheapest: no generation and no judge during training |
| Llama 3 post-training | Iterative: six rounds of SFT + DPO, new preference data from the best recent models each round | Keeps DPO cheap per round while the data follows the policy |
| Online DPO with an LLM judge (OAIF) | Online, on-policy: two fresh answers per prompt per step | Feedback on the model’s own current mistakes |
| Asynchronous RLHF | Online, slightly off-policy: train on the previous batch while the next is generated | Generation and training run in parallel; about 40% faster on an 8B chatbot task at matched quality [Certain] |
| Video LMMs: DPO for video question answering (Zhang et al., 2024) | Preference pairs on the model’s own answers, scored by a language model that reads detailed video captions | Human labels for long videos are slow; a text judge makes fresh labels cheap |
| Video captioning and key-moment descriptions with a metric reward | On-policy RL: sample a caption, score it (for example CIDEr), update, as in self-critical sequence training [Likely] | The metric can score any new caption, so online RL is possible |
| Video recommendation and thumbnail or highlight choice from logs | Offline, off-policy: logged user clicks, reweighted by the logging policy’s probabilities (inverse propensity scoring) [Likely] | You cannot show every user a new policy; importance weights (71.3) correct for the logging policy |
Common mistakes
- Mixing up the reference policy and the old policy. \(\pi_\text{ref}\) is fixed for the whole run (or a round) and anchors the KL. \(\pi_{\theta_\text{old}}\) is refreshed every iteration and only defines the ratio. Using the reference as the denominator of the PPO ratio makes the clip fire on almost every token after a short time.
- Calling DPO “on-policy” because it has \(\pi_\theta\) in the loss. Having the current policy inside the loss is not the same as sampling from it. Offline DPO never samples.
- Calling every online method on-policy. Several PPO epochs, replay buffers and asynchronous generation are online but partly off-policy. Report the data age.
- Reweighting old sequences with the full product of ratios. The sequence weight (71.4) has an effective sample size of about \(e^{-T\sigma^2}\); for long answers it is close to zero. Use per-token ratios with clipping, or refresh the data.
- Trusting the measured ESS on heavy-tailed weights. When weights are huge but rare, the sample may not contain the huge ones, and the measured ESS looks fine (Figure 2 at \(T = 1000\)).
- Running DPO with a reference that did not write the data without noticing. If the pairs came from other models, the reference’s log-probabilities of those answers can be very low and the implied rewards strange; first SFT on the chosen answers, as the DPO paper suggests, or generate on-policy pairs.
- Iterating DPO without watching the KL. Each round moves further from the start, especially with a reset reference. Track quality and KL per round, and stop when quality stops rising.
- Mismatched sampling and training probabilities. If the generation engine samples with a different temperature, precision or top-p than the one used to compute \(\log \pi_{\theta_\text{old}}\) in training, the “on-policy” data is in fact slightly off-policy. [Likely]
Interview questions
Define on-policy and off-policy in one sentence each.basic
What is the difference between online and offline, and how does it relate to on/off-policy?basic
Name the four policies that appear in RLHF papers and what each one does.core
Is PPO on-policy or off-policy?core
Why is DPO called offline? Is it off-policy?core
Why does the policy gradient need on-policy samples?core
Why not make any method off-policy by importance-weighting old answers?advanced
What is online DPO and what does it need that offline DPO does not?core
What is iterative DPO? How did Llama 3 use it?core
In iterative DPO, should the reference be reset to the latest policy each round?advanced
Your RL run generates with vLLM and trains with FSDP. Is it really on-policy?advanced
Exercises
- A behavior policy writes answer A with probability 0.6 and B with 0.4. The current policy has \(\pi(A) = 0.9\), \(\pi(B) = 0.1\). Rewards are \(r(A) = 1\), \(r(B) = 0\). Compute both importance weights and check that \(\E_\mu[w r] = \E_\pi[r]\).
- Compute the ESS (71.5) of the weights \((3.6, 0.2, 0.1, 0.1)\).
- With \(\sigma = 0.03\) per token, what share of samples is effective for answers of 1,000 tokens, using (71.6)? What is the sequence KL?
- Classify each setup on the 2 × 2 map: (a) SFT on human-written answers; (b) GRPO with 4 mini-batch updates per batch; (c) DPO on pairs generated once by the SFT model; (d) the same as (c), but regenerated every 500 steps.
- In Figure 6 at \(\beta = 0.1\) with a fixed reference, read the expected reward for 1 and for 8 rounds. How much larger is the gain over the reference?
- In PPO with \(\epsilon = 0.2\), a token had probability 0.30 at rollout time and 0.40 after two mini-batch steps. What is its ratio, and does it fall outside the clip range?
Answerscheck
Key takeaways
- Four policies: the current policy (trained), the behavior policy (wrote the data), the old policy (PPO/GRPO snapshot for the ratio), and the reference (frozen anchor in the KL or the DPO loss). A rollout is one sampled answer; with its score it is a trajectory.
- On-policy means the data came from the current policy; the policy gradient (71.2) needs it. GRPO with one update per batch is exactly on-policy; PPO is on-policy in spirit and slightly off-policy inside an iteration.
- Reusing one batch for many epochs turns it into off-policy data: the weights spread, the ESS falls and the batch overstates its own gain (Figure 3). The clip stops this; fresh data resets it. Offline training is limited by what its dataset contains, whatever the update rule (Figure 5).
- Importance sampling (71.3) bridges the gap, but whole-answer weights (71.4) explode: ESS/N \(\approx e^{-T\sigma^2} = e^{-2\KL}\) (71.6). That is why LLM methods use per-token clipped ratios and fresh data.
- Online means new samples and labels during training; offline means a fixed dataset. On-policy implies online; offline implies off-policy.
- Offline DPO (71.7) is cheap but cannot learn about answers its data never shows. Online DPO (71.8) and iterative DPO (71.9, Llama 3, Self-Rewarding) keep the loss and refresh the data; in the toy of Figure 6 this gives 46% more reward gain for the same budget.
Key formulas
References for this session
- OpenAI (Achiam) (2018). Spinning Up in Deep RL
- Lambert (2025). Reinforcement Learning from Human Feedback (book)
- Kaufmann et al. (2023). A Survey of Reinforcement Learning from Human Feedback
- OpenAI (Achiam) (2018). Spinning Up in Deep RL
- Lambert (2025). Reinforcement Learning from Human Feedback (book)
- Kaufmann et al. (2023). A Survey of Reinforcement Learning from Human Feedback
- Schulman et al. (2017). Proximal Policy Optimization Algorithms
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
- Ouyang et al. (2022). Training language models to follow instructions with human feedback
- Guo et al. (2024). Direct Language Model Alignment from Online AI Feedback
- Xiong et al. (2023). Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
- Yuan et al. (2024). Self-Rewarding Language Models
- Tajwar et al. (2024). Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
- Xu et al. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
- Noukhovitch et al. (2024). Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
- Levine, Kumar, Tucker and Fu (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Zhang et al. (2024). Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward