RL infrastructure
An RL step for an LLM is a small distributed system: one engine generates answers, another trains on them, and the new weights must travel back. This session shows why generation is the bottleneck, how weights are synced between trainer and generator on shared or separate GPUs, what asynchronous RL gains and what off-policy drift costs, and what TRL, verl and OpenRLHF are each good for.
What you will learn
- The four phases of an RL step (rollout, scoring, training, weight sync) and how their times add up
- Why decoding is memory-bound and why the long tail of answer lengths leaves GPUs idle, with numbers for a 7B model on an H100
- Why RL libraries use an inference engine such as vLLM or SGLang, and what continuous batching and prefix caching buy
- Weight sync: its cost in bytes and seconds, and colocated (hybrid engine) versus separate GPU placement
- Synchronous versus asynchronous RL, staleness as a delayed feedback loop, and the corrections: behaviour log-probs, decoupled PPO, truncated importance sampling
- Engine mismatch: why vLLM and the trainer give different log-probs for the same weights, and how to check it
- TRL, verl and OpenRLHF: design, engines, backends, and when to choose each
Why this matters
The last sessions were about what to optimize: PPO (session 66), GRPO (session 70), reward models and AI feedback (session 74). This session is about how to run it fast. An RL step for an LLM is not one big matrix multiply. It is a small distributed system: one part generates answers, one part scores them, one part trains on them, and the new weights must travel back to the generator. If you build it naively, most of your GPUs wait most of the time.
The authors of OpenRLHF write that the inference (generation) phase “often accounts for over 90% of the total RLHF (or RLVR) runtime”. [Certain] So the main engineering tools of this session all attack generation time or idle time:
- A fast inference engine for the rollouts, such as vLLM or SGLang, instead of the training framework’s slow
generate(). - Weight sync: after each update, the new weights must be copied from the trainer into the engine. Where the two live (same GPUs or separate GPUs) decides how this copy works and what it costs.
- Synchronous versus asynchronous RL: generate the next batch while training on the current one. This removes idle time, but the data is now a little off-policy (generated by an older policy), and the loss must handle that.
- Libraries: TRL, verl and OpenRLHF package these choices in different ways. Knowing which does what saves weeks.
One RL step as a dataflow
Take one GRPO or PPO step from session 66 and look at it as a system. There are three kinds of work:
- Rollout (generation): sample answers \(y \sim \pi_\theta(\cdot \mid x)\) for a batch of prompts. For GRPO, each prompt is sampled \(G\) times. This is inference: no gradients, token by token, with a KV cache (session 80).
- Scoring: compute rewards (a verifier such as unit tests or exact match, or a reward model forward pass), plus the extra log-probs the loss needs: the reference model for the KL term, and the critic values for PPO.
- Training: forward and backward passes with the RL loss, then an optimizer step. This is training: big, parallel, sharded with FSDP, ZeRO or Megatron (session 45, session 48).
Then a fourth step that does not exist in supervised training: weight sync, which copies the new weights into the generator. We call one set of weights a version; the generator must know which version it is using.
The two big jobs want very different software. Training wants large parallel matrix multiplies, sharded optimizer states and activation checkpointing. Generation wants a paged KV cache, continuous batching, CUDA graphs and fast sampling kernels (session 134, session 135). No single engine is best at both, so real systems run two engines with two copies of the policy weights and move weights between them.
For a synchronous step, the times simply add up:
- Read the symbols
- \(T_\text{gen}\)
- time to generate all answers of the batch; it ends when the longest answer ends
- \(T_\text{score}\)
- time for rewards, reference log-probs and (for PPO) critic values
- \(T_\text{train}\)
- forward, backward and optimizer step(s) on the batch
- \(T_\text{sync}\)
- time to copy the new weights into the generator (or to switch the same GPUs from training to generation)
How to read Figure 1
Press Next to walk through the phases, or Play. Watch the “Busy now” list: in phases 1 and 2 only the rollout GPUs work; in phases 4 and 5 only the training GPUs work. If these are different GPUs, each group is idle for a large part of the step. This is the problem that colocation and asynchronous RL (later sections) solve in two different ways.
Why rollout is the bottleneck
There are three reasons, and each one has a number behind it.
Reason 1: decoding is memory-bound
Training processes all tokens of a sequence in parallel; one weight read from GPU memory serves thousands of tokens. Decoding produces one token per sequence per step, and every step must read all the weights from memory again (session 82). With \(N\) parameters in bf16 (2 bytes each) and a batch of \(b\) sequences, one decode step does about \(2Nb\) FLOPs and reads at least \(2N\) bytes. Its arithmetic intensity (FLOPs per byte moved) is therefore about
- Read the symbols
- \(N\)
- number of parameters of the policy
- \(b\)
- number of sequences decoded together (the decode batch)
- \(M_\text{KV}\)
- bytes of KV cache read in this step; it grows with batch size and context length
A GPU is compute-bound only when the intensity is above its ridge point, peak FLOP/s divided by memory bandwidth. For an H100 SXM (989 TFLOP/s dense bf16, half of the 1,979 “with sparsity” on the spec sheet, and 3.35 TB/s) the ridge point is about 295 FLOP/byte. [Certain] So decoding needs a batch of hundreds of sequences just to stop wasting compute, and the KV cache limits how large that batch can be.
Take \(N = 7 \times 10^9\), so the weights are \(14\) GB. One decode step must read them: \(14 / 3{,}350 = 4.18\) ms at least. With \(b = 1\) that is at most 239 tokens per second, using 0.3% of the GPU’s FLOPs. With \(b = 64\) and a tiny KV cache, the FLOP use rises to 21.7%; with \(b = 256\) to 86.7%.
Now add the KV cache. Qwen2.5-7B has 28 layers, 4 key-value heads and head size 128 (its config.json), so one token of KV cache is \(2 \times 28 \times 4 \times 128 \times 2 = 57{,}344\) bytes. [Certain] A batch of 256 sequences at an average context of 1,500 tokens holds 22.0 GB of KV cache. One step now reads \(14 + 22.0 = 36.0\) GB and takes at least 10.75 ms: about 23,800 tokens per second, 33.7% of peak FLOPs. At an average context of 8,000 tokens (long reasoning traces) the KV cache would be 117 GB, more than the GPU’s 80 GB, so the batch must shrink, and the throughput falls with it.
For comparison, training at 40% MFU (session 51) costs \(6N\) FLOPs per token, so one H100 trains on about \(0.4 \times 989 \times 10^{12} / (6 \times 7 \times 10^9) = 9{,}420\) tokens per second. These are best-case bounds; real engines reach less. [Likely]
The point is not that decoding is always slower per token than training. It is that decoding speed depends on batch size and memory, which an RL system must manage actively, and that it is very easy to get a small fraction of this bound with a naive loop.
Reason 2: the long tail of answer lengths
The answers in one batch have very different lengths. Reasoning models in particular produce some answers that are 10 times longer than the median (session 72). A synchronous step cannot train until the last answer is done. While the long answers finish, the slots of the short ones are empty. For a group of \(S\) answers decoded together with static batching (start together, end together), the share of slot time that does useful work is
- Read the symbols
- \(L_i\)
- length of answer \(i\) in tokens
- \(S\)
- number of decode slots (sequences that run at the same time)
- \(\max_i L_i\)
- the longest answer in the group: everyone waits for it
Continuous batching, the default in vLLM and SGLang, starts a new sequence in a slot as soon as the old one finishes (session 135). This fills the gaps in the middle of the batch. But it cannot fill the end: when the queue of prompts is empty, the last long answers still run alone. The total time is at least the average work per slot, \(\sum L_i / S\), and at least the longest answer.
How to read Figure 2
At the default (16 slots, \(\sigma = 0.8\), cap 4,096) static batching keeps only about a quarter of the slots busy: 10,937 steps for work that needs 2,713.5 at best. Continuous batching halves the time to 5,480 steps, but the bottom panel still ends with a long grey area: one answer hit the cap at 4,096 tokens and runs alone at the end. Lower the cap to 1,024: the tail shrinks and slot use rises, but more answers are cut (red), which changes the RL data (cut answers usually get a zero or a penalty). Set \(\sigma = 0.2\): equal lengths, and both methods are close to the bound.
This is why RL systems use the tricks that follow. Some are pure engineering: generate with large batches across many prompts, use continuous batching, and use prefix caching: all \(G\) answers of a GRPO group share the same prompt, so its KV cache can be computed once and shared (SGLang’s RadixAttention and vLLM’s automatic prefix caching do this). [Likely] Others change the RL algorithm a little: cap the length, stop overly long or repeating answers early, or keep generating the long tail while the trainer already works, which is asynchronous RL.
Reason 3: the training framework is a poor generator
Calling model.generate() inside a training framework uses static batching, no paged KV cache and often no CUDA graphs. Under FSDP or ZeRO-3 it is worse: the weights are sharded, so every decode step must all-gather every layer’s weights from the other GPUs, for one token per sequence. [Likely] That is why every serious RL library hands generation to an inference engine. TRL’s documentation says it directly: generating with the model’s own generate is the bottleneck, and vLLM is used because it serves completions far faster. [Certain]
Weight sync and GPU placement
Using an inference engine creates a new job. After every optimizer step, the engine’s copy of the weights is out of date. Before the next rollout, the new weights must be copied in. In bf16 the amount of data is \(2N\) bytes, so a lower bound on the sync time is
- Read the symbols
- \(2N\)
- bytes of the policy weights in bf16; with LoRA only the adapter (a few MB) must move
- \(\text{BW}_\text{link}\)
- bandwidth of the path the weights travel: NVLink inside a node, InfiniBand between nodes, or no copy at all when engine and trainer share GPU memory
Real transfers are slower than this bound, because the trainer’s weights are sharded in one layout and the engine needs another (see below). [Likely]
A 7B policy is 14 GB in bf16. Over one InfiniBand link at 400 Gb/s (50 GB/s): \(14/50 = 0.28\) s. Over H100 NVLink at 450 GB/s per direction (the 900 GB/s on the spec sheet counts both directions [Likely]): \(14/450 = 0.031\) s. Over PCIe Gen4 x16 at about 32 GB/s: 0.44 s.
A 70B policy is 140 GB: 2.8 s over one 400 Gb/s link, 0.35 s if eight such links work in parallel (one per GPU), 4.4 s over PCIe Gen4. A 32B policy over one link: 1.28 s.
Compared with a step of a minute or more, the network sync is small. Saving a checkpoint to disk and reloading it in a separate inference server is the slow way: at a few GB/s of disk bandwidth, a 70B model takes tens of seconds each way. [Likely] Kimi k1.5, which switches the same GPUs between Megatron (training) and vLLM (inference), reports under one minute from training to inference and about ten seconds back. [Certain]
Two placements
Colocated (also called a “hybrid engine”): the trainer and the inference engine use the same GPUs, one after the other. During rollout, the trainer’s optimizer states and gradients are moved to CPU memory or freed, and the engine’s KV cache takes the space. During training, the engine is put to sleep and its KV cache is freed. The weight “sync” is a local copy, or a reshard: training might use FSDP across 8 GPUs, while generation wants each 7B model whole on one GPU or tensor-parallel across 2. verl’s 3D-HybridEngine does exactly this resharding of the actor model between training and generation, with “zero memory redundancy” and reduced communication, according to the HybridFlow paper. [Certain] TRL’s default vllm_mode="colocate" runs vLLM inside the trainer process on the same GPUs. [Certain]
Separate (disaggregated): some GPUs run only the engine, others run only the trainer. TRL’s vllm_mode="server" starts a vLLM server on dedicated GPUs; after each optimizer step the trainer streams the updated weights to it over NCCL. [Certain] OpenRLHF uses Ray, a Python framework for running tasks and long-lived workers (“actors”) on a cluster, to place vLLM engines, the actor trainer and the other models on their own GPUs. [Certain]
| Colocated (hybrid engine) | Separate GPUs | |
|---|---|---|
| Weight sync | local copy or reshard in GPU memory; no network | NCCL broadcast over NVLink or InfiniBand, formula (75.4) |
| Memory | engine and trainer share memory; needs offload and sleep modes | each side has all of its GPUs’ memory |
| Idle time (synchronous) | none between phases, but the long tail wastes all GPUs | each group waits while the other works |
| Asynchronous training | not possible: the phases share hardware | natural: generate batch \(t+1\) while training on batch \(t\) |
| Tuning | one number of GPUs for both jobs | must balance generation GPUs against training GPUs |
Synchronous versus asynchronous RL
With separate GPUs, the obvious fix for idle time is a pipeline: while the trainer trains on batch \(t\), the engine already generates batch \(t+1\) with the previous weights. With \(g\) generation GPUs out of \(G\), and generation and training work \(W_\text{gen}\) and \(W_\text{train}\) in GPU-seconds, the step times are
- Read the symbols
- \(G\), \(g\)
- total GPUs, and the number used for generation
- \(W_\text{gen}\), \(W_\text{train}\)
- GPU-seconds of generation and training work in one step, assuming perfect scaling
- \(\max(\cdot,\cdot)\)
- in the pipeline, the slower side sets the pace; the best split makes both sides equal
The colocated step is \(W_\text{gen}/G + W_\text{train}/G + T_\text{switch}\). With perfect scaling and no long tail, colocation is very hard to beat: all GPUs are always busy. Two effects change this in practice. First, the long tail of Figure 2: in a synchronous step, all generation GPUs wait for the last answer, while an asynchronous engine can fill that time with answers for the next batch. AReaL names exactly this: generation must wait for the longest sequence before training can start, which leaves devices idle. [Certain] Second, spreading generation over all GPUs makes the decode batch per GPU smaller, which makes decoding even more memory-bound (75.2); AReaL also names this. [Certain] Figure 3 models the first effect with a “tail waste” share that only synchronous generation pays.
How to read Figure 3
At the default (80% generation, 6 + 2 GPUs, 2 s sync, 30% tail waste) the steps take 69.14 s colocated (72.3% GPU use), 118.19 s separate synchronous (42.3%) and 55.33 s asynchronous (90.4%). Set tail waste to 0: colocated becomes the fastest (52 s, 96.2%), because it never idles. Move the split to 4 + 4: the asynchronous pipeline is now limited by generation and loses its advantage. The best split puts \(80\% \times 8 = 6.4\) GPUs on generation. Separate synchronous is never the best: it pays both the idle time and the sync.
The price of asynchrony: off-policy data
In the pipeline, batch \(t+1\) was generated by weights of version \(t\) or older, but the trainer is at version \(t+1\). The data is off-policy: the policy that generated it (the behaviour policy \(\pi_\text{behav}\)) is not the policy being trained (session 71). We call the version gap the staleness \(k\). Policy-gradient formulas assume \(k = 0\). What happens if we ignore the gap?
A small model shows the danger clearly. Let the policy be a Gaussian \(a \sim \mathcal{N}(\mu, \sigma^2)\) over one number, with reward \(-(a - c)^2\), so the best \(\mu\) is \(c\). The policy-gradient estimate with a batch-mean baseline, computed with the current \(\mu_t\) but on samples from the stale \(\mu_{t-k}\), has the expected value \(\operatorname{Cov}(R, a)/\sigma^2 = -2(\mu_{t-k} - c)\). The update becomes a delayed feedback loop:
- Read the symbols
- \(\mu_t\)
- the policy mean at step \(t\); the trainer’s current weights
- \(\mu_{t-k}\)
- the mean of the behaviour policy that generated the batch, \(k\) versions old
- \(\eta\)
- step size (learning rate)
- \(c\)
- the best action; the gradient points towards it
The step is computed from where the policy was, not where it is. It is like steering a car by looking at a photo taken \(k\) seconds ago: at slow speed this works, at high speed you overshoot and swerve. For this kind of linear delay equation, \(x_{t+1} = x_t - a\, x_{t-k}\), there is a classical result (Levin and May, 1976 [Likely]): it is stable exactly when \(0 \lt a \lt 2\sin\frac{\pi}{2(2k+1)}\). With \(a = 2\eta\), the largest safe step size is
We checked this numerically with the roots of the characteristic polynomial \(z^{k+1} - z^k + a\): \(\eta_\text{max} = 1\) for \(k = 0\), 0.5 for \(k = 1\), 0.309 for \(k = 2\), 0.174 for \(k = 4\) and 0.092 for \(k = 8\). Staleness shrinks the safe learning rate, roughly in proportion to \(1/k\) for large \(k\). This toy is not an LLM, but the mechanism (a gradient computed from old behaviour) is the same. [Likely]
Corrections
The standard fix is to treat the stale batch as off-policy data and use importance sampling, as in session 66. The PPO ratio must use the log-probs of the policy that actually generated the tokens, which the engine returns with each sample. With large staleness the ratio becomes noisy. Hilton, Cobbe and Schulman (2021) proposed to decouple the two roles of \(\pi_{\theta_\text{old}}\) in PPO: the proximal policy \(\pi_\text{prox}\) that anchors the trust region, and the behaviour policy that corrects for off-policy sampling. [Certain] AReaL uses this form, with \(\pi_\text{prox}\) set to the weights just before each update: [Certain]
- Read the symbols
- \(\pi_\text{behav}\)
- the (stale) weights that sampled the tokens; their log-probs come from the rollout engine
- \(\pi_\text{prox}\)
- a recent policy that defines the trust region; in AReaL, the weights just before the update
- \(u_t\)
- the PPO ratio against the proximal policy; the clip keeps each update small
- \(\pi_\text{prox}/\pi_\text{behav}\)
- the importance weight that corrects for the staleness; it gets no gradient
AReaL also bounds the staleness: generation requests are throttled so that no sample is more than \(\eta\) versions old (they use \(\eta = 4\) for code and \(\eta = 8\) for math), and found that moderate staleness (\(\eta \le 8\)) had minimal impact on final performance, while unbounded staleness was worse even with the decoupled objective. [Certain] (Their \(\eta\) is a staleness bound, not our step size.) Noukhovitch et al. (2024) ran RLHF with a one-step-off pipeline and report about 40% faster training on a LLaMA 3.1 8B chatbot task and about 70% faster on GSM8k with a 1B model, matching the synchronous results; they also found online DPO to be the most robust of the methods they tried to off-policy data. [Certain]
How to read Figure 4
At the default (\(k = 4\), \(\eta = 0.2\), no correction) the blue curve overshoots and swings with growing size, because 0.2 is above the limit 0.1736 of (75.7); the dashed grey curve with fresh data converges. Lower \(\eta\) to 0.15: the stale run now converges too, more slowly and with an overshoot. Choose importance sampling: the swing disappears, because the weights turn the stale samples into an estimate for the current policy, but the effective sample size drops to about 5 of 64 in the first steps, so the gradient is noisy. Truncated IS keeps the effective sample size near 29 while still removing most of the swing. Set \(k = 0\): all three modes are identical.
Off-policy even when synchronous: engine mismatch
There is a second, quieter source of off-policy data. The rollout engine and the trainer compute the “same” model with different kernels, different numerical precision and different order of floating-point sums. So even with identical weights, \(\pi_\text{engine}(y_t) \ne \pi_\text{trainer}(y_t)\). Yao et al. (2025) showed that in frameworks such as verl, which use vLLM for rollouts and FSDP for training, this gap “implicitly turns the on-policy RL to be off-policy”, and that for some tokens the two disagree strongly. [Certain] Their fix is truncated importance sampling (TIS): multiply each token’s loss by a capped weight
- Read the symbols
- \(\pi_\text{trainer}\)
- probability of the sampled token under the trainer’s computation of the old weights (no gradient)
- \(\pi_\text{engine}\)
- probability the engine actually sampled from, returned with the rollout
- \(C\)
- the cap; it trades a small bias for bounded variance; values around 2 appear in verl’s examples [Likely]
For this to work, the engine must return the log-probs it really sampled from, after temperature and other logit processing. This is why TRL asks vLLM to return “processed” log-probs, which it says is what the importance sampling correction expects. [Certain] A simple health check in any RL system: at staleness 0, compute \(\log \pi_\text{trainer} - \log \pi_\text{engine}\) on the rollout tokens and log its mean and maximum. The code below does this.
Partial rollouts
Another way to cut the tail is to not wait for it. Kimi k1.5 gives each rollout a fixed output token budget per iteration; an answer that is not finished is stored in a replay buffer and continued in the next iteration, so one long answer is made of segments from several iterations. [Certain] AReaL goes further with interruptible rollouts: when new weights arrive, the engine stops all ongoing generations, loads the new version, recomputes the KV caches and continues the unfinished answers. One answer can therefore contain tokens from different policy versions. [Certain] This works because the loss is per token and each token keeps the behaviour log-prob it was sampled with.
The algorithms
- input: \(G\) GPUs holding the sharded trainer state; inference engine asleep; prompt batch \(\{x_i\}\); version \(v\)
- offload optimizer state and gradients (or free them); wake the engine; reshard weights version \(v\) into the engine layout // \(T_\text{switch}\)
- generate \(G\) answers per prompt with continuous batching; keep tokens and behaviour log-probs // \(T_\text{gen}\), the slow part
- put the engine to sleep (free KV cache); reload the trainer state
- score: rewards from verifiers or a reward model; reference log-probs; critic values for PPO // \(T_\text{score}\)
- advantages (GAE or group-relative); recompute \(\log \pi_{\theta_\text{old}}\) in the trainer if TIS (75.9) is used
- for each mini-batch: PPO/GRPO loss, backward, optimizer step // \(T_\text{train}\)
- \(v \leftarrow v + 1\); log reward, KL, clip fraction, mean and max of \(\log\pi_\text{trainer} - \log\pi_\text{engine}\)
- input: engine workers on \(g\) GPUs; trainer on \(G - g\) GPUs; staleness bound \(k_\text{max}\); a queue between them
- engine loop: while training runs:
- wait while the queue holds data that would be more than \(k_\text{max}\) versions old when it is used // throttle
- generate a batch with the current engine weights, version \(v_e\); push (tokens, behaviour log-probs, \(v_e\)) to the queue
- if new weights are announced: load them (NCCL broadcast), \(v_e \leftarrow\) new version // optional: interrupt and resume unfinished answers
- trainer loop: for \(v = 1, 2, \dots\):
- pop a batch; its staleness is \(v - 1 - v_e\); score it, compute advantages
- loss with the ratio against the behaviour log-probs, or the decoupled objective (75.8)
- backward, optimizer step; broadcast the new weights to the engines // \(T_\text{sync}\), (75.4)
- log the staleness, the clip fraction and the effective sample size of the batch
The PyTorch code below runs Algorithm 2 on a toy problem, on a CPU, in a few seconds. The “engine” is a separate copy of the policy in bf16 (as an inference engine would have), with its own sampling loop. It returns behaviour log-probs and the weight version with each batch. The trainer computes log-probs in fp32 and uses a GRPO-style group advantage with a PPO clip. The staleness \(k\) is how many batches the engine runs ahead. Everything is deterministic (fixed seeds, single thread of control).
import copy, collections, torch, torch.nn as nn, torch.nn.functional as F V, T, P, G = 8, 6, 16, 4 # vocab, response length, prompts per step, answers per prompt GOOD = 3 # the verifier rewards token 3 class TinyLM(nn.Module): # next-token logits from (previous token, position) def __init__(s): super().__init__() s.tok, s.pos = nn.Embedding(V + 1, 32), nn.Embedding(T, 32) s.mlp = nn.Sequential(nn.Linear(32, 64), nn.Tanh(), nn.Linear(64, V)) def forward(s, prev): return s.mlp(s.tok(prev) + s.pos.weight[None, :prev.size(1)]) def prev_of(y): # shift right; V is the "start" token return torch.cat([torch.full((y.size(0), 1), V), y[:, :-1]], 1) def logp_of(model, y): # trainer side: log pi(y_t | y_<t) in fp32 lp = F.log_softmax(model(prev_of(y)).float(), -1) return lp.gather(-1, y[..., None]).squeeze(-1) @torch.no_grad() def sample(model, n, gen): # token-by-token sampling; returns tokens and their log-probs y, lps = torch.zeros(n, 0, dtype=torch.long), [] for t in range(T): prev = prev_of(torch.cat([y, torch.zeros(n, 1, dtype=torch.long)], 1)) logits = model(prev)[:, t].float() nxt = torch.multinomial(F.softmax(logits, -1), 1, generator=gen).squeeze(1) lps.append(F.log_softmax(logits, -1).gather(-1, nxt[:, None]).squeeze(1)) y = torch.cat([y, nxt[:, None]], 1) return y, torch.stack(lps, 1) class Engine: # stands in for vLLM or SGLang: a separate bf16 copy def __init__(s, policy): s.model, s.version = copy.deepcopy(policy).to(torch.bfloat16).eval(), 0 def sync(s, policy, version): # weight sync: trainer -> engine (cast to bf16) s.model.load_state_dict(policy.state_dict()); s.version = version def rollout(s, n, gen): y, lp = sample(s.model, n, gen) return y, lp, s.version # tokens, behaviour log-probs, weight version def run(k, steps=60): # k = how many batches the engine runs ahead (staleness) torch.manual_seed(0); gen, ev = torch.Generator().manual_seed(0), torch.Generator().manual_seed(1) policy = TinyLM(); opt = torch.optim.Adam(policy.parameters(), lr=3e-3) engine, queue, clip = Engine(policy), collections.deque(), [] for step in range(steps): while len(queue) <= k: # producer: keep k batches ready ahead of the trainer queue.append(engine.rollout(P * G, gen)) y, beh_lp, ver = queue.popleft() s = (y == GOOD).float().mean(1).view(P, G) # verifiable reward adv = ((s - s.mean(1, keepdim=True)) / (s.std(1, keepdim=True) + 1e-4)).view(-1, 1) new_lp = logp_of(policy, y) ratio = torch.exp(new_lp - beh_lp) # against the BEHAVIOUR policy loss = -torch.min(ratio * adv, ratio.clamp(0.8, 1.2) * adv).mean() opt.zero_grad(); loss.backward(); opt.step() engine.sync(policy, step + 1) # push the new weights clip.append(((ratio.detach() - 1).abs() > 0.2).float().mean().item()) if k == 0 and step == 0: d = (new_lp.detach() - beh_lp).abs() # same weights, two engines print(f"same version: |logp_trainer - logp_engine| mean {d.mean():.5f} max {d.max():.5f}") if step % 20 == 19: yv, _ = sample(policy, 1024, ev) # evaluate the CURRENT policy print(f"k={k} step {step+1} reward {(yv == GOOD).float().mean():.3f}" f" lag {step - ver} clip frac {sum(clip[-20:]) / 20:.3f}") for k in (0, 1, 4): run(k) # Output (CPU, torch 2.8): # same version: |logp_trainer - logp_engine| mean 0.00097 max 0.00435 # k=0 step 20 reward 0.961 lag 0 clip frac 0.000 # k=0 step 40 reward 0.999 lag 0 clip frac 0.000 # k=0 step 60 reward 1.000 lag 0 clip frac 0.000 # k=1 step 20 reward 0.935 lag 1 clip frac 0.040 # k=1 step 40 reward 0.998 lag 1 clip frac 0.010 # k=1 step 60 reward 1.000 lag 1 clip frac 0.000 # k=4 step 20 reward 0.538 lag 4 clip frac 0.276 # k=4 step 40 reward 0.892 lag 4 clip frac 0.280 # k=4 step 60 reward 0.981 lag 4 clip frac 0.064
Three things to read from this output. First, even at the same weight version, the bf16 engine and the fp32 trainer disagree: the mean gap in log-prob is 0.00097 and the largest 0.00435. That is the engine mismatch of (75.9), small here because the toy model is tiny. Second, with one step of staleness the run is almost as good as synchronous (0.935 versus 0.961 at step 20): the pipeline is nearly free. Third, with \(k = 4\) the clip fraction jumps to about 0.28 and learning per step is much slower (0.538 at step 20): many samples fall outside the trust region of the behaviour policy and give no gradient. In a real system, \(k = 1\) to a few is the usual trade, because the wall-clock gain from overlap is large and the per-step cost is small. [Likely]
Libraries: TRL, verl, OpenRLHF
All three implement the loop above with an inference engine for rollouts. They differ in scale, structure and defaults.
| TRL (Hugging Face) | verl (HybridFlow, ByteDance Seed) | OpenRLHF | |
|---|---|---|---|
| Design | Trainer classes on top of Transformers; one script, launched with accelerate or torchrun [Likely] | a hybrid controller: one central Python program describes the dataflow, while workers run the distributed computation [Certain] | Ray schedules separate workers for vLLM engines, actor training and the other models [Certain] |
| Rollout engine | vLLM, colocated (default) or as a server on its own GPUs [Certain] | vLLM, SGLang, TensorRT-LLM [Certain] | vLLM [Certain] |
| Training backend | Accelerate with DeepSpeed or FSDP, PEFT/LoRA [Likely] | FSDP, Megatron-LM, and others [Certain] | DeepSpeed ZeRO, with automatic tensor parallelism and ring-attention sequence parallelism [Certain] |
| Weight sync | NCCL stream to the vLLM server after each optimizer step, or in-process when colocated [Certain] | 3D-HybridEngine reshards the actor between training and generation layouts [Certain] | model slices sent from the training engines to vLLM [Certain] |
| Algorithms | SFT, DPO, GRPO, RLOO, online DPO, reward modelling, and more [Likely] | PPO, GRPO, DAPO, and many recipes [Certain] | PPO, GRPO-style methods, REINFORCE variants, DPO [Likely] |
| Async | no asynchronous pipeline in the docs we read [Likely] | one-step-off-policy and fully asynchronous trainer recipes [Certain] | asynchronous training mode [Certain] |
| Good for | one node, small to mid-size models, LoRA, quick experiments, labs like session 76 and session 77 | large clusters, very large or MoE models (Megatron), many algorithms, research at scale | multi-node PPO with a separate critic and reward model; clear Ray-based separation of roles |
Published speed-ups, each against the baselines chosen in its own paper: HybridFlow reports 1.53× to 20.57× higher throughput than earlier RLHF systems [Certain]; OpenRLHF reports 1.22× to 1.68× speed-ups over other frameworks across model sizes [Certain]; AReaL reports up to 2.77× faster training than synchronous systems with the same number of GPUs [Certain]. Earlier, DeepSpeed-Chat (2023) combined training and inference optimizations in one RLHF system and popularized the colocated “hybrid engine” idea. [Likely] These numbers are not comparable with each other: setups, models and baselines differ.
How to choose, as a rule of thumb [Likely]: start with TRL on one node, with vLLM colocated, and LoRA if memory is tight. Move to verl or OpenRLHF when the model or the cluster grows, when you need PPO with a separate critic at scale, when you need Megatron for very large models, or when long reasoning rollouts make asynchronous training worth its complexity.
Where this is used
| Setting | The infrastructure problem | Typical choice |
|---|---|---|
| Reasoning models trained with verifiable rewards (session 70) | very long answers, huge length spread, the tail of Figure 2 | large GRPO groups with prefix caching, length caps, partial or interruptible rollouts, asynchronous training with a staleness bound |
| Classic PPO-RLHF chat alignment | four models in memory (session 66); reward model and critic forward passes | separate placement (OpenRLHF, verl) or colocation with offloading |
| Code agents with unit tests | the reward is a program run; sandboxes on CPU workers can be slower than generation | a pool of reward workers in parallel with the trainer; asynchronous scoring [Likely] |
| Multi-turn agents with tools | each episode waits for tool calls; GPUs idle between turns | server-based asynchronous rollouts (verl lists “server-based asynchronous rollout” for agentic RL [Certain]) |
| Small models and LoRA on one node | memory, not throughput | TRL with colocated vLLM; with LoRA only the adapter needs to be synced |
| VLM alignment with images | image tokens make prompts long; the engine must run the vision encoder too | an engine with multimodal support; TRL sends images to a separate vLLM endpoint for multimodal prompts [Certain] |
| Video-LLMs: video QA, key-moment detection, temporal grounding | thousands of visual tokens per prompt, so KV cache memory limits the decode batch (75.2) even with short answers | prefix caching of the shared video prompt across the \(G\) answers; fewer, larger generation GPUs; rewards such as time-span overlap (session 205) [Likely] |
| Video captioning for retrieval | the reward is a retrieval score of the caption against the clip, which needs an embedding model on GPU | the retrieval model as a separate reward worker, scoring in parallel with generation [Speculation] |
Common mistakes
- Generating with the training framework.
generate()under FSDP or ZeRO-3 all-gathers the weights at every decode step and uses static batching. Use an inference engine, even on one GPU. - Forgetting the weight sync, or syncing the wrong thing. If the engine keeps the old weights, the run is silently off-policy by more and more versions. Log the engine’s weight version with every batch and assert the staleness you expect. With LoRA, merge or load the adapter in the engine, and check the tied embeddings and renamed layers.
- Recomputing the “old” log-probs with the current weights in an asynchronous run. Then the ratio is 1 at the start and the staleness is invisible to the loss. Use the behaviour log-probs from the engine (or the decoupled objective 75.8).
- Trusting the engine’s log-probs blindly. They may be computed before temperature scaling, in a different precision, or with different kernels. Check the mean and maximum of \(\log\pi_\text{trainer} - \log\pi_\text{engine}\) at staleness 0; use TIS (75.9) when the gap is large.
- Different sampling settings in the engine and the loss. If the engine samples at temperature 0.7 or with top-p, the behaviour policy is that modified distribution, not the raw softmax. The ratio must use the same distribution. [Likely]
- Unbounded staleness. A fast engine with a slow trainer builds a long queue of old data. Throttle generation so no sample is more than a few versions old.
- Ignoring the long tail. A few answers at the length cap can double the step time. Log the length distribution and the number of cut answers per step, not only the mean length.
- Out of memory when switching phases in a colocated setup. The engine’s KV cache reservation (for vLLM, a fraction of GPU memory) plus the trainer’s state must fit, or one of them must be offloaded. Lower the engine’s memory fraction or use its sleep mode. [Likely]
Interview questions
Why is rollout generation usually the bottleneck in RLHF and RLVR?basic
Why not just call model.generate() in the trainer?basic
What is weight sync, and how long does it take for a 70B model?core
Colocated or separate GPUs for the inference engine: what are the trade-offs?core
What is asynchronous RL for LLMs, and what does it cost?core
Explain the decoupled PPO objective used by AReaL.advanced
Why are vLLM’s log-probs different from the trainer’s for the same weights, and what do you do about it?advanced
How do partial or interruptible rollouts help with the long tail?advanced
In a GRPO setup with 8 GPUs, generation is 80% of the work. How would you split them for an asynchronous pipeline?core
Which library would you pick for (a) a LoRA GRPO run on one 8-GPU node, (b) PPO with a 70B policy and separate critic on 64 GPUs?core
GRPOTrainer with use_vllm=True in colocate mode, or server mode with some GPUs for vLLM; LoRA keeps memory low and only the adapter must be synced. (b) verl or OpenRLHF: both handle multi-node placement of actor, critic, reference and reward models with an inference engine; verl offers a Megatron backend for very large models and resharding with its 3D-HybridEngine; OpenRLHF places roles on GPUs with Ray and trains with DeepSpeed ZeRO. Either way, plan the memory as in session 66 (four models) and measure where the step time goes before tuning.Exercises
- Four answers are decoded together with static batching. Their lengths are 200, 300, 400 and 1,500 tokens. What share of slot time does useful work, by (75.3)?
- A 32B policy in bf16 is synced over one 400 Gb/s link. What is the lower bound on the sync time? How does it change with LoRA adapters of 100 MB?
- Generation takes 300 GPU-seconds and training 100 GPU-seconds per step on 8 GPUs, with a 2 s sync. Compute the step time for separate synchronous with 6 + 2 GPUs, and for asynchronous with 6 + 2.
- Using (75.7), what is the largest stable step size in the Gaussian toy for staleness \(k = 3\)? Is \(\eta = 0.25\) stable?
- The engine sampled a token with probability 0.10, and the trainer computes 0.30 for the same weights. What is the TIS weight with \(C = 2\)? And if the trainer computes 0.15?
- Why does a GRPO group of \(G = 16\) answers per prompt benefit from prefix caching more than PPO with one answer per prompt?
Answerscheck
Key takeaways
- An RL step is a distributed dataflow: rollout, scoring, training, and weight sync (75.1). Generation usually dominates; OpenRLHF reports over 90% of runtime.
- Decoding is memory-bound (75.2) and waits for the longest answer (75.3). Use an inference engine (vLLM, SGLang) with continuous batching and prefix caching, never the trainer’s
generate(). - Weight sync moves \(2N\) bytes per update (75.4): well under a second for 7B on InfiniBand, seconds for 70B; much slower through disk.
- Colocated (hybrid engine) keeps all GPUs busy and syncs locally; separate GPUs allow asynchronous pipelining (75.5) but must be balanced.
- Asynchronous RL trains on data that is \(k\) versions old. Ignoring this is a delayed feedback loop that lowers the safe learning rate (75.6), (75.7); use behaviour log-probs, the decoupled objective (75.8) and a staleness bound.
- Even synchronous RL is slightly off-policy because engine and trainer compute different probabilities; measure the gap and use truncated importance sampling (75.9).
- TRL: simplest, one node, colocated or server vLLM. verl: hybrid controller, FSDP or Megatron, vLLM or SGLang, resharding engine, async recipes. OpenRLHF: Ray, vLLM and DeepSpeed, roles on separate GPUs.
Key formulas
References for this session
- Hugging Face (code). TRL library
- Sheng et al. (2024). HybridFlow: A Flexible and Efficient RLHF Framework (verl)
- Hu et al. (2024). OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
- Hugging Face (code). TRL library
- Sheng et al. (2024). HybridFlow: A Flexible and Efficient RLHF Framework (verl)
- Hu et al. (2024). OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
- vLLM project (docs). vLLM documentation
- Schulman et al. (2017). Proximal Policy Optimization Algorithms
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Noukhovitch et al. (2024). Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
- Fu et al. (2025). AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- Kimi Team (2025). Kimi k1.5: Scaling Reinforcement Learning with LLMs
- Yao et al. (2023). DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales
- Yao, Liu et al. (2025). Your Efficient RL Framework Secretly Brings You Off-Policy RL Training
- Hilton, Cobbe and Schulman (2021). Batch size-invariance for policy optimization