LLM & VLM courseBlog
75Track 4: Post-training and RL · session 23 of 25

RL infrastructure

An RL step for an LLM is a small distributed system: one engine generates answers, another trains on them, and the new weights must travel back. This session shows why generation is the bottleneck, how weights are synced between trainer and generator on shared or separate GPUs, what asynchronous RL gains and what off-policy drift costs, and what TRL, verl and OpenRLHF are each good for.

Time: 3 to 4 hours (reading 2 to 2.5 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 66 and 70; helpful: the KV cache, ZeRO and FSDP

What you will learn

  • The four phases of an RL step (rollout, scoring, training, weight sync) and how their times add up
  • Why decoding is memory-bound and why the long tail of answer lengths leaves GPUs idle, with numbers for a 7B model on an H100
  • Why RL libraries use an inference engine such as vLLM or SGLang, and what continuous batching and prefix caching buy
  • Weight sync: its cost in bytes and seconds, and colocated (hybrid engine) versus separate GPU placement
  • Synchronous versus asynchronous RL, staleness as a delayed feedback loop, and the corrections: behaviour log-probs, decoupled PPO, truncated importance sampling
  • Engine mismatch: why vLLM and the trainer give different log-probs for the same weights, and how to check it
  • TRL, verl and OpenRLHF: design, engines, backends, and when to choose each

Why this matters

The last sessions were about what to optimize: PPO (session 66), GRPO (session 70), reward models and AI feedback (session 74). This session is about how to run it fast. An RL step for an LLM is not one big matrix multiply. It is a small distributed system: one part generates answers, one part scores them, one part trains on them, and the new weights must travel back to the generator. If you build it naively, most of your GPUs wait most of the time.

The authors of OpenRLHF write that the inference (generation) phase “often accounts for over 90% of the total RLHF (or RLVR) runtime”. [Certain] So the main engineering tools of this session all attack generation time or idle time:

  • A fast inference engine for the rollouts, such as vLLM or SGLang, instead of the training framework’s slow generate().
  • Weight sync: after each update, the new weights must be copied from the trainer into the engine. Where the two live (same GPUs or separate GPUs) decides how this copy works and what it costs.
  • Synchronous versus asynchronous RL: generate the next batch while training on the current one. This removes idle time, but the data is now a little off-policy (generated by an older policy), and the loss must handle that.
  • Libraries: TRL, verl and OpenRLHF package these choices in different ways. Knowing which does what saves weeks.

One RL step as a dataflow

Take one GRPO or PPO step from session 66 and look at it as a system. There are three kinds of work:

  1. Rollout (generation): sample answers \(y \sim \pi_\theta(\cdot \mid x)\) for a batch of prompts. For GRPO, each prompt is sampled \(G\) times. This is inference: no gradients, token by token, with a KV cache (session 80).
  2. Scoring: compute rewards (a verifier such as unit tests or exact match, or a reward model forward pass), plus the extra log-probs the loss needs: the reference model for the KL term, and the critic values for PPO.
  3. Training: forward and backward passes with the RL loss, then an optimizer step. This is training: big, parallel, sharded with FSDP, ZeRO or Megatron (session 45, session 48).

Then a fourth step that does not exist in supervised training: weight sync, which copies the new weights into the generator. We call one set of weights a version; the generator must know which version it is using.

The two big jobs want very different software. Training wants large parallel matrix multiplies, sharded optimizer states and activation checkpointing. Generation wants a paged KV cache, continuous batching, CUDA graphs and fast sampling kernels (session 134, session 135). No single engine is best at both, so real systems run two engines with two copies of the policy weights and move weights between them.

For a synchronous step, the times simply add up:

\[ T_\text{step} = T_\text{gen} + T_\text{score} + T_\text{train} + T_\text{sync} \tag{75.1} \]
Read the symbols
\(T_\text{gen}\)
time to generate all answers of the batch; it ends when the longest answer ends
\(T_\text{score}\)
time for rewards, reference log-probs and (for PPO) critic values
\(T_\text{train}\)
forward, backward and optimizer step(s) on the batch
\(T_\text{sync}\)
time to copy the new weights into the generator (or to switch the same GPUs from training to generation)
Figure 1. One synchronous RL step as a dataflow with six phases. The rollout engine and the trainer each hold a copy of the policy. The dashed arrow is the weight sync that closes the loop. The panel on the left shows which resource is busy in each phase; in a synchronous loop only one of them works at a time.
How to read Figure 1

Press Next to walk through the phases, or Play. Watch the “Busy now” list: in phases 1 and 2 only the rollout GPUs work; in phases 4 and 5 only the training GPUs work. If these are different GPUs, each group is idle for a large part of the step. This is the problem that colocation and asynchronous RL (later sections) solve in two different ways.

Why rollout is the bottleneck

There are three reasons, and each one has a number behind it.

Reason 1: decoding is memory-bound

Training processes all tokens of a sequence in parallel; one weight read from GPU memory serves thousands of tokens. Decoding produces one token per sequence per step, and every step must read all the weights from memory again (session 82). With \(N\) parameters in bf16 (2 bytes each) and a batch of \(b\) sequences, one decode step does about \(2Nb\) FLOPs and reads at least \(2N\) bytes. Its arithmetic intensity (FLOPs per byte moved) is therefore about

\[ I_\text{decode} \approx \frac{2 N b}{2 N + M_\text{KV}} \ \le\ b \quad \text{FLOP/byte} \tag{75.2} \]
Read the symbols
\(N\)
number of parameters of the policy
\(b\)
number of sequences decoded together (the decode batch)
\(M_\text{KV}\)
bytes of KV cache read in this step; it grows with batch size and context length

A GPU is compute-bound only when the intensity is above its ridge point, peak FLOP/s divided by memory bandwidth. For an H100 SXM (989 TFLOP/s dense bf16, half of the 1,979 “with sparsity” on the spec sheet, and 3.35 TB/s) the ridge point is about 295 FLOP/byte. [Certain] So decoding needs a batch of hundreds of sequences just to stop wasting compute, and the KV cache limits how large that batch can be.

Worked example: decode speed of a 7B policy on one H100

Take \(N = 7 \times 10^9\), so the weights are \(14\) GB. One decode step must read them: \(14 / 3{,}350 = 4.18\) ms at least. With \(b = 1\) that is at most 239 tokens per second, using 0.3% of the GPU’s FLOPs. With \(b = 64\) and a tiny KV cache, the FLOP use rises to 21.7%; with \(b = 256\) to 86.7%.

Now add the KV cache. Qwen2.5-7B has 28 layers, 4 key-value heads and head size 128 (its config.json), so one token of KV cache is \(2 \times 28 \times 4 \times 128 \times 2 = 57{,}344\) bytes. [Certain] A batch of 256 sequences at an average context of 1,500 tokens holds 22.0 GB of KV cache. One step now reads \(14 + 22.0 = 36.0\) GB and takes at least 10.75 ms: about 23,800 tokens per second, 33.7% of peak FLOPs. At an average context of 8,000 tokens (long reasoning traces) the KV cache would be 117 GB, more than the GPU’s 80 GB, so the batch must shrink, and the throughput falls with it.

For comparison, training at 40% MFU (session 51) costs \(6N\) FLOPs per token, so one H100 trains on about \(0.4 \times 989 \times 10^{12} / (6 \times 7 \times 10^9) = 9{,}420\) tokens per second. These are best-case bounds; real engines reach less. [Likely]

The point is not that decoding is always slower per token than training. It is that decoding speed depends on batch size and memory, which an RL system must manage actively, and that it is very easy to get a small fraction of this bound with a naive loop.

Reason 2: the long tail of answer lengths

The answers in one batch have very different lengths. Reasoning models in particular produce some answers that are 10 times longer than the median (session 72). A synchronous step cannot train until the last answer is done. While the long answers finish, the slots of the short ones are empty. For a group of \(S\) answers decoded together with static batching (start together, end together), the share of slot time that does useful work is

\[ U_\text{static} = \frac{\sum_{i=1}^{S} L_i}{S \cdot \max_i L_i} \tag{75.3} \]
Read the symbols
\(L_i\)
length of answer \(i\) in tokens
\(S\)
number of decode slots (sequences that run at the same time)
\(\max_i L_i\)
the longest answer in the group: everyone waits for it

Continuous batching, the default in vLLM and SGLang, starts a new sequence in a slot as soon as the old one finishes (session 135). This fills the gaps in the middle of the batch. But it cannot fill the end: when the queue of prompts is empty, the last long answers still run alone. The total time is at least the average work per slot, \(\sum L_i / S\), and at least the longest answer.

answer, static batchesanswer, continuous batchinganswer cut at the length capidle slot
Figure 2. 64 answers with seeded random lengths: log-normal with median 400 tokens and spread \(\sigma\), cut at the cap. Each row is a decode slot. Top: static batches of \(S\) answers, each batch waits for its longest answer, formula (75.3). Bottom: continuous batching, where a free slot takes the next answer at once. The green dashed line is the lower bound \(\sum L_i / S\). Time is counted in decode steps and ignores that a step gets slower with more active slots.
How to read Figure 2

At the default (16 slots, \(\sigma = 0.8\), cap 4,096) static batching keeps only about a quarter of the slots busy: 10,937 steps for work that needs 2,713.5 at best. Continuous batching halves the time to 5,480 steps, but the bottom panel still ends with a long grey area: one answer hit the cap at 4,096 tokens and runs alone at the end. Lower the cap to 1,024: the tail shrinks and slot use rises, but more answers are cut (red), which changes the RL data (cut answers usually get a zero or a penalty). Set \(\sigma = 0.2\): equal lengths, and both methods are close to the bound.

This is why RL systems use the tricks that follow. Some are pure engineering: generate with large batches across many prompts, use continuous batching, and use prefix caching: all \(G\) answers of a GRPO group share the same prompt, so its KV cache can be computed once and shared (SGLang’s RadixAttention and vLLM’s automatic prefix caching do this). [Likely] Others change the RL algorithm a little: cap the length, stop overly long or repeating answers early, or keep generating the long tail while the trainer already works, which is asynchronous RL.

Reason 3: the training framework is a poor generator

Calling model.generate() inside a training framework uses static batching, no paged KV cache and often no CUDA graphs. Under FSDP or ZeRO-3 it is worse: the weights are sharded, so every decode step must all-gather every layer’s weights from the other GPUs, for one token per sequence. [Likely] That is why every serious RL library hands generation to an inference engine. TRL’s documentation says it directly: generating with the model’s own generate is the bottleneck, and vLLM is used because it serves completions far faster. [Certain]

Weight sync and GPU placement

Using an inference engine creates a new job. After every optimizer step, the engine’s copy of the weights is out of date. Before the next rollout, the new weights must be copied in. In bf16 the amount of data is \(2N\) bytes, so a lower bound on the sync time is

\[ T_\text{sync} \ \ge\ \frac{2N}{\text{BW}_\text{link}} \tag{75.4} \]
Read the symbols
\(2N\)
bytes of the policy weights in bf16; with LoRA only the adapter (a few MB) must move
\(\text{BW}_\text{link}\)
bandwidth of the path the weights travel: NVLink inside a node, InfiniBand between nodes, or no copy at all when engine and trainer share GPU memory

Real transfers are slower than this bound, because the trainer’s weights are sharded in one layout and the engine needs another (see below). [Likely]

Worked example: how long does a weight sync take?

A 7B policy is 14 GB in bf16. Over one InfiniBand link at 400 Gb/s (50 GB/s): \(14/50 = 0.28\) s. Over H100 NVLink at 450 GB/s per direction (the 900 GB/s on the spec sheet counts both directions [Likely]): \(14/450 = 0.031\) s. Over PCIe Gen4 x16 at about 32 GB/s: 0.44 s.

A 70B policy is 140 GB: 2.8 s over one 400 Gb/s link, 0.35 s if eight such links work in parallel (one per GPU), 4.4 s over PCIe Gen4. A 32B policy over one link: 1.28 s.

Compared with a step of a minute or more, the network sync is small. Saving a checkpoint to disk and reloading it in a separate inference server is the slow way: at a few GB/s of disk bandwidth, a 70B model takes tens of seconds each way. [Likely] Kimi k1.5, which switches the same GPUs between Megatron (training) and vLLM (inference), reports under one minute from training to inference and about ten seconds back. [Certain]

Two placements

Colocated (also called a “hybrid engine”): the trainer and the inference engine use the same GPUs, one after the other. During rollout, the trainer’s optimizer states and gradients are moved to CPU memory or freed, and the engine’s KV cache takes the space. During training, the engine is put to sleep and its KV cache is freed. The weight “sync” is a local copy, or a reshard: training might use FSDP across 8 GPUs, while generation wants each 7B model whole on one GPU or tensor-parallel across 2. verl’s 3D-HybridEngine does exactly this resharding of the actor model between training and generation, with “zero memory redundancy” and reduced communication, according to the HybridFlow paper. [Certain] TRL’s default vllm_mode="colocate" runs vLLM inside the trainer process on the same GPUs. [Certain]

Separate (disaggregated): some GPUs run only the engine, others run only the trainer. TRL’s vllm_mode="server" starts a vLLM server on dedicated GPUs; after each optimizer step the trainer streams the updated weights to it over NCCL. [Certain] OpenRLHF uses Ray, a Python framework for running tasks and long-lived workers (“actors”) on a cluster, to place vLLM engines, the actor trainer and the other models on their own GPUs. [Certain]

Colocated (hybrid engine)Separate GPUs
Weight synclocal copy or reshard in GPU memory; no networkNCCL broadcast over NVLink or InfiniBand, formula (75.4)
Memoryengine and trainer share memory; needs offload and sleep modeseach side has all of its GPUs’ memory
Idle time (synchronous)none between phases, but the long tail wastes all GPUseach group waits while the other works
Asynchronous trainingnot possible: the phases share hardwarenatural: generate batch \(t+1\) while training on batch \(t\)
Tuningone number of GPUs for both jobsmust balance generation GPUs against training GPUs

Synchronous versus asynchronous RL

With separate GPUs, the obvious fix for idle time is a pipeline: while the trainer trains on batch \(t\), the engine already generates batch \(t+1\) with the previous weights. With \(g\) generation GPUs out of \(G\), and generation and training work \(W_\text{gen}\) and \(W_\text{train}\) in GPU-seconds, the step times are

\[ T_\text{sync-sep} = \frac{W_\text{gen}}{g} + \frac{W_\text{train}}{G - g} + T_\text{sync}, \qquad T_\text{async} = \max\!\Big(\frac{W_\text{gen}}{g},\ \frac{W_\text{train}}{G - g}\Big) + T_\text{sync} \tag{75.5} \]
Read the symbols
\(G\), \(g\)
total GPUs, and the number used for generation
\(W_\text{gen}\), \(W_\text{train}\)
GPU-seconds of generation and training work in one step, assuming perfect scaling
\(\max(\cdot,\cdot)\)
in the pipeline, the slower side sets the pace; the best split makes both sides equal

The colocated step is \(W_\text{gen}/G + W_\text{train}/G + T_\text{switch}\). With perfect scaling and no long tail, colocation is very hard to beat: all GPUs are always busy. Two effects change this in practice. First, the long tail of Figure 2: in a synchronous step, all generation GPUs wait for the last answer, while an asynchronous engine can fill that time with answers for the next batch. AReaL names exactly this: generation must wait for the longest sequence before training can start, which leaves devices idle. [Certain] Second, spreading generation over all GPUs makes the decode batch per GPU smaller, which makes decoding even more memory-bound (75.2); AReaL also names this. [Certain] Figure 3 models the first effect with a “tail waste” share that only synchronous generation pays.

generationtrainingweight sync or switch
Figure 3. A toy model of 8 GPUs and 400 GPU-seconds of work per step, using formula (75.5). Work scales perfectly with the number of GPUs. “Tail waste” is the share of generation time lost waiting for the longest answers; it is paid by colocated and separate synchronous generation, but not by the asynchronous engine, which keeps generating for the next batch. The asynchronous row trains on batch \(t\) while generating batch \(t+1\), so its data is one version old (lag 1). The numbers are illustrations, not measurements.
How to read Figure 3

At the default (80% generation, 6 + 2 GPUs, 2 s sync, 30% tail waste) the steps take 69.14 s colocated (72.3% GPU use), 118.19 s separate synchronous (42.3%) and 55.33 s asynchronous (90.4%). Set tail waste to 0: colocated becomes the fastest (52 s, 96.2%), because it never idles. Move the split to 4 + 4: the asynchronous pipeline is now limited by generation and loses its advantage. The best split puts \(80\% \times 8 = 6.4\) GPUs on generation. Separate synchronous is never the best: it pays both the idle time and the sync.

The price of asynchrony: off-policy data

In the pipeline, batch \(t+1\) was generated by weights of version \(t\) or older, but the trainer is at version \(t+1\). The data is off-policy: the policy that generated it (the behaviour policy \(\pi_\text{behav}\)) is not the policy being trained (session 71). We call the version gap the staleness \(k\). Policy-gradient formulas assume \(k = 0\). What happens if we ignore the gap?

A small model shows the danger clearly. Let the policy be a Gaussian \(a \sim \mathcal{N}(\mu, \sigma^2)\) over one number, with reward \(-(a - c)^2\), so the best \(\mu\) is \(c\). The policy-gradient estimate with a batch-mean baseline, computed with the current \(\mu_t\) but on samples from the stale \(\mu_{t-k}\), has the expected value \(\operatorname{Cov}(R, a)/\sigma^2 = -2(\mu_{t-k} - c)\). The update becomes a delayed feedback loop:

\[ \mu_{t+1} = \mu_t - 2\eta\,(\mu_{t-k} - c) \tag{75.6} \]
Read the symbols
\(\mu_t\)
the policy mean at step \(t\); the trainer’s current weights
\(\mu_{t-k}\)
the mean of the behaviour policy that generated the batch, \(k\) versions old
\(\eta\)
step size (learning rate)
\(c\)
the best action; the gradient points towards it

The step is computed from where the policy was, not where it is. It is like steering a car by looking at a photo taken \(k\) seconds ago: at slow speed this works, at high speed you overshoot and swerve. For this kind of linear delay equation, \(x_{t+1} = x_t - a\, x_{t-k}\), there is a classical result (Levin and May, 1976 [Likely]): it is stable exactly when \(0 \lt a \lt 2\sin\frac{\pi}{2(2k+1)}\). With \(a = 2\eta\), the largest safe step size is

\[ \eta \ \lt\ \eta_\text{max}(k) = \sin\!\frac{\pi}{2(2k+1)} \tag{75.7} \]

We checked this numerically with the roots of the characteristic polynomial \(z^{k+1} - z^k + a\): \(\eta_\text{max} = 1\) for \(k = 0\), 0.5 for \(k = 1\), 0.309 for \(k = 2\), 0.174 for \(k = 4\) and 0.092 for \(k = 8\). Staleness shrinks the safe learning rate, roughly in proportion to \(1/k\) for large \(k\). This toy is not an LLM, but the mechanism (a gradient computed from old behaviour) is the same. [Likely]

Corrections

The standard fix is to treat the stale batch as off-policy data and use importance sampling, as in session 66. The PPO ratio must use the log-probs of the policy that actually generated the tokens, which the engine returns with each sample. With large staleness the ratio becomes noisy. Hilton, Cobbe and Schulman (2021) proposed to decouple the two roles of \(\pi_{\theta_\text{old}}\) in PPO: the proximal policy \(\pi_\text{prox}\) that anchors the trust region, and the behaviour policy that corrects for off-policy sampling. [Certain] AReaL uses this form, with \(\pi_\text{prox}\) set to the weights just before each update: [Certain]

\[ J(\theta) = \E_{a_t \sim \pi_\text{behav}}\!\left[ \sum_t \frac{\pi_\text{prox}(a_t \mid s_t)}{\pi_\text{behav}(a_t \mid s_t)}\, \min\!\Big( u_t\, \hat A_t,\ \operatorname{clip}(u_t, 1-\epsilon, 1+\epsilon)\, \hat A_t \Big) \right], \quad u_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_\text{prox}(a_t \mid s_t)} \tag{75.8} \]
Read the symbols
\(\pi_\text{behav}\)
the (stale) weights that sampled the tokens; their log-probs come from the rollout engine
\(\pi_\text{prox}\)
a recent policy that defines the trust region; in AReaL, the weights just before the update
\(u_t\)
the PPO ratio against the proximal policy; the clip keeps each update small
\(\pi_\text{prox}/\pi_\text{behav}\)
the importance weight that corrects for the staleness; it gets no gradient

AReaL also bounds the staleness: generation requests are throttled so that no sample is more than \(\eta\) versions old (they use \(\eta = 4\) for code and \(\eta = 8\) for math), and found that moderate staleness (\(\eta \le 8\)) had minimal impact on final performance, while unbounded staleness was worse even with the decoupled objective. [Certain] (Their \(\eta\) is a staleness bound, not our step size.) Noukhovitch et al. (2024) ran RLHF with a one-step-off pipeline and report about 40% faster training on a LLaMA 3.1 8B chatbot task and about 70% faster on GSM8k with a 1B model, matching the synchronous results; they also found online DPO to be the most robust of the methods they tried to off-policy data. [Certain]

policy mean \(\mu_t\), chosen settingsame step size with fresh data (\(k = 0\)), dashedeffective sample size
Figure 4. Live training of the Gaussian policy of formula (75.6): \(\sigma = 1\), best action \(c = 2\), start at \(\mu = 0\), 64 seeded samples per step drawn from the policy \(k\) versions ago, advantage = reward minus the batch mean. “No correction” uses the samples as if they were fresh; “importance sampling” weights each sample by \(\pi_{\mu_t}(a)/\pi_{\mu_{t-k}}(a)\); “truncated IS” caps that weight at 2. Values outside \([-2, 6]\) are drawn at the edge. The bottom strip shows the effective sample size \((\sum w)^2 / \sum w^2\) of each batch.
How to read Figure 4

At the default (\(k = 4\), \(\eta = 0.2\), no correction) the blue curve overshoots and swings with growing size, because 0.2 is above the limit 0.1736 of (75.7); the dashed grey curve with fresh data converges. Lower \(\eta\) to 0.15: the stale run now converges too, more slowly and with an overshoot. Choose importance sampling: the swing disappears, because the weights turn the stale samples into an estimate for the current policy, but the effective sample size drops to about 5 of 64 in the first steps, so the gradient is noisy. Truncated IS keeps the effective sample size near 29 while still removing most of the swing. Set \(k = 0\): all three modes are identical.

Off-policy even when synchronous: engine mismatch

There is a second, quieter source of off-policy data. The rollout engine and the trainer compute the “same” model with different kernels, different numerical precision and different order of floating-point sums. So even with identical weights, \(\pi_\text{engine}(y_t) \ne \pi_\text{trainer}(y_t)\). Yao et al. (2025) showed that in frameworks such as verl, which use vLLM for rollouts and FSDP for training, this gap “implicitly turns the on-policy RL to be off-policy”, and that for some tokens the two disagree strongly. [Certain] Their fix is truncated importance sampling (TIS): multiply each token’s loss by a capped weight

\[ w_t = \min\!\left( \frac{\pi_\text{trainer}(y_t \mid s_t)}{\pi_\text{engine}(y_t \mid s_t)},\ C \right) \tag{75.9} \]
Read the symbols
\(\pi_\text{trainer}\)
probability of the sampled token under the trainer’s computation of the old weights (no gradient)
\(\pi_\text{engine}\)
probability the engine actually sampled from, returned with the rollout
\(C\)
the cap; it trades a small bias for bounded variance; values around 2 appear in verl’s examples [Likely]

For this to work, the engine must return the log-probs it really sampled from, after temperature and other logit processing. This is why TRL asks vLLM to return “processed” log-probs, which it says is what the importance sampling correction expects. [Certain] A simple health check in any RL system: at staleness 0, compute \(\log \pi_\text{trainer} - \log \pi_\text{engine}\) on the rollout tokens and log its mean and maximum. The code below does this.

Partial rollouts

Another way to cut the tail is to not wait for it. Kimi k1.5 gives each rollout a fixed output token budget per iteration; an answer that is not finished is stored in a replay buffer and continued in the next iteration, so one long answer is made of segments from several iterations. [Certain] AReaL goes further with interruptible rollouts: when new weights arrive, the engine stops all ongoing generations, loads the new version, recomputes the KV caches and continues the unfinished answers. One answer can therefore contain tokens from different policy versions. [Certain] This works because the loss is per token and each token keeps the behaviour log-prob it was sampled with.

The algorithms

Algorithm 1: synchronous, colocated RL step (hybrid engine)
  1. input: \(G\) GPUs holding the sharded trainer state; inference engine asleep; prompt batch \(\{x_i\}\); version \(v\)
  2. offload optimizer state and gradients (or free them); wake the engine; reshard weights version \(v\) into the engine layout // \(T_\text{switch}\)
  3. generate \(G\) answers per prompt with continuous batching; keep tokens and behaviour log-probs // \(T_\text{gen}\), the slow part
  4. put the engine to sleep (free KV cache); reload the trainer state
  5. score: rewards from verifiers or a reward model; reference log-probs; critic values for PPO // \(T_\text{score}\)
  6. advantages (GAE or group-relative); recompute \(\log \pi_{\theta_\text{old}}\) in the trainer if TIS (75.9) is used
  7. for each mini-batch: PPO/GRPO loss, backward, optimizer step // \(T_\text{train}\)
  8. \(v \leftarrow v + 1\); log reward, KL, clip fraction, mean and max of \(\log\pi_\text{trainer} - \log\pi_\text{engine}\)
Algorithm 2: asynchronous RL with bounded staleness (separate GPUs)
  1. input: engine workers on \(g\) GPUs; trainer on \(G - g\) GPUs; staleness bound \(k_\text{max}\); a queue between them
  2. engine loop: while training runs:
  3.   wait while the queue holds data that would be more than \(k_\text{max}\) versions old when it is used // throttle
  4.   generate a batch with the current engine weights, version \(v_e\); push (tokens, behaviour log-probs, \(v_e\)) to the queue
  5.   if new weights are announced: load them (NCCL broadcast), \(v_e \leftarrow\) new version // optional: interrupt and resume unfinished answers
  6. trainer loop: for \(v = 1, 2, \dots\):
  7.   pop a batch; its staleness is \(v - 1 - v_e\); score it, compute advantages
  8.   loss with the ratio against the behaviour log-probs, or the decoupled objective (75.8)
  9.   backward, optimizer step; broadcast the new weights to the engines // \(T_\text{sync}\), (75.4)
  10.   log the staleness, the clip fraction and the effective sample size of the batch

The PyTorch code below runs Algorithm 2 on a toy problem, on a CPU, in a few seconds. The “engine” is a separate copy of the policy in bf16 (as an inference engine would have), with its own sampling loop. It returns behaviour log-probs and the weight version with each batch. The trainer computes log-probs in fp32 and uses a GRPO-style group advantage with a PPO clip. The staleness \(k\) is how many batches the engine runs ahead. Everything is deterministic (fixed seeds, single thread of control).

import copy, collections, torch, torch.nn as nn, torch.nn.functional as F
V, T, P, G = 8, 6, 16, 4         # vocab, response length, prompts per step, answers per prompt
GOOD = 3                          # the verifier rewards token 3

class TinyLM(nn.Module):          # next-token logits from (previous token, position)
    def __init__(s):
        super().__init__()
        s.tok, s.pos = nn.Embedding(V + 1, 32), nn.Embedding(T, 32)
        s.mlp = nn.Sequential(nn.Linear(32, 64), nn.Tanh(), nn.Linear(64, V))
    def forward(s, prev):
        return s.mlp(s.tok(prev) + s.pos.weight[None, :prev.size(1)])

def prev_of(y):                   # shift right; V is the "start" token
    return torch.cat([torch.full((y.size(0), 1), V), y[:, :-1]], 1)

def logp_of(model, y):            # trainer side: log pi(y_t | y_<t) in fp32
    lp = F.log_softmax(model(prev_of(y)).float(), -1)
    return lp.gather(-1, y[..., None]).squeeze(-1)

@torch.no_grad()
def sample(model, n, gen):        # token-by-token sampling; returns tokens and their log-probs
    y, lps = torch.zeros(n, 0, dtype=torch.long), []
    for t in range(T):
        prev = prev_of(torch.cat([y, torch.zeros(n, 1, dtype=torch.long)], 1))
        logits = model(prev)[:, t].float()
        nxt = torch.multinomial(F.softmax(logits, -1), 1, generator=gen).squeeze(1)
        lps.append(F.log_softmax(logits, -1).gather(-1, nxt[:, None]).squeeze(1))
        y = torch.cat([y, nxt[:, None]], 1)
    return y, torch.stack(lps, 1)

class Engine:                     # stands in for vLLM or SGLang: a separate bf16 copy
    def __init__(s, policy):
        s.model, s.version = copy.deepcopy(policy).to(torch.bfloat16).eval(), 0
    def sync(s, policy, version): # weight sync: trainer -> engine (cast to bf16)
        s.model.load_state_dict(policy.state_dict()); s.version = version
    def rollout(s, n, gen):
        y, lp = sample(s.model, n, gen)
        return y, lp, s.version   # tokens, behaviour log-probs, weight version

def run(k, steps=60):             # k = how many batches the engine runs ahead (staleness)
    torch.manual_seed(0); gen, ev = torch.Generator().manual_seed(0), torch.Generator().manual_seed(1)
    policy = TinyLM(); opt = torch.optim.Adam(policy.parameters(), lr=3e-3)
    engine, queue, clip = Engine(policy), collections.deque(), []
    for step in range(steps):
        while len(queue) <= k:    # producer: keep k batches ready ahead of the trainer
            queue.append(engine.rollout(P * G, gen))
        y, beh_lp, ver = queue.popleft()
        s = (y == GOOD).float().mean(1).view(P, G)                # verifiable reward
        adv = ((s - s.mean(1, keepdim=True)) / (s.std(1, keepdim=True) + 1e-4)).view(-1, 1)
        new_lp = logp_of(policy, y)
        ratio = torch.exp(new_lp - beh_lp)                        # against the BEHAVIOUR policy
        loss = -torch.min(ratio * adv, ratio.clamp(0.8, 1.2) * adv).mean()
        opt.zero_grad(); loss.backward(); opt.step()
        engine.sync(policy, step + 1)                             # push the new weights
        clip.append(((ratio.detach() - 1).abs() > 0.2).float().mean().item())
        if k == 0 and step == 0:
            d = (new_lp.detach() - beh_lp).abs()                  # same weights, two engines
            print(f"same version: |logp_trainer - logp_engine| mean {d.mean():.5f} max {d.max():.5f}")
        if step % 20 == 19:
            yv, _ = sample(policy, 1024, ev)                      # evaluate the CURRENT policy
            print(f"k={k} step {step+1}  reward {(yv == GOOD).float().mean():.3f}"
                  f"  lag {step - ver}  clip frac {sum(clip[-20:]) / 20:.3f}")

for k in (0, 1, 4):
    run(k)
# Output (CPU, torch 2.8):
# same version: |logp_trainer - logp_engine| mean 0.00097 max 0.00435
# k=0 step 20  reward 0.961  lag 0  clip frac 0.000
# k=0 step 40  reward 0.999  lag 0  clip frac 0.000
# k=0 step 60  reward 1.000  lag 0  clip frac 0.000
# k=1 step 20  reward 0.935  lag 1  clip frac 0.040
# k=1 step 40  reward 0.998  lag 1  clip frac 0.010
# k=1 step 60  reward 1.000  lag 1  clip frac 0.000
# k=4 step 20  reward 0.538  lag 4  clip frac 0.276
# k=4 step 40  reward 0.892  lag 4  clip frac 0.280
# k=4 step 60  reward 0.981  lag 4  clip frac 0.064

Three things to read from this output. First, even at the same weight version, the bf16 engine and the fp32 trainer disagree: the mean gap in log-prob is 0.00097 and the largest 0.00435. That is the engine mismatch of (75.9), small here because the toy model is tiny. Second, with one step of staleness the run is almost as good as synchronous (0.935 versus 0.961 at step 20): the pipeline is nearly free. Third, with \(k = 4\) the clip fraction jumps to about 0.28 and learning per step is much slower (0.538 at step 20): many samples fall outside the trust region of the behaviour policy and give no gradient. In a real system, \(k = 1\) to a few is the usual trade, because the wall-clock gain from overlap is large and the per-step cost is small. [Likely]

Libraries: TRL, verl, OpenRLHF

All three implement the loop above with an inference engine for rollouts. They differ in scale, structure and defaults.

TRL (Hugging Face)verl (HybridFlow, ByteDance Seed)OpenRLHF
DesignTrainer classes on top of Transformers; one script, launched with accelerate or torchrun [Likely]a hybrid controller: one central Python program describes the dataflow, while workers run the distributed computation [Certain]Ray schedules separate workers for vLLM engines, actor training and the other models [Certain]
Rollout enginevLLM, colocated (default) or as a server on its own GPUs [Certain]vLLM, SGLang, TensorRT-LLM [Certain]vLLM [Certain]
Training backendAccelerate with DeepSpeed or FSDP, PEFT/LoRA [Likely]FSDP, Megatron-LM, and others [Certain]DeepSpeed ZeRO, with automatic tensor parallelism and ring-attention sequence parallelism [Certain]
Weight syncNCCL stream to the vLLM server after each optimizer step, or in-process when colocated [Certain]3D-HybridEngine reshards the actor between training and generation layouts [Certain]model slices sent from the training engines to vLLM [Certain]
AlgorithmsSFT, DPO, GRPO, RLOO, online DPO, reward modelling, and more [Likely]PPO, GRPO, DAPO, and many recipes [Certain]PPO, GRPO-style methods, REINFORCE variants, DPO [Likely]
Asyncno asynchronous pipeline in the docs we read [Likely]one-step-off-policy and fully asynchronous trainer recipes [Certain]asynchronous training mode [Certain]
Good forone node, small to mid-size models, LoRA, quick experiments, labs like session 76 and session 77large clusters, very large or MoE models (Megatron), many algorithms, research at scalemulti-node PPO with a separate critic and reward model; clear Ray-based separation of roles

Published speed-ups, each against the baselines chosen in its own paper: HybridFlow reports 1.53× to 20.57× higher throughput than earlier RLHF systems [Certain]; OpenRLHF reports 1.22× to 1.68× speed-ups over other frameworks across model sizes [Certain]; AReaL reports up to 2.77× faster training than synchronous systems with the same number of GPUs [Certain]. Earlier, DeepSpeed-Chat (2023) combined training and inference optimizations in one RLHF system and popularized the colocated “hybrid engine” idea. [Likely] These numbers are not comparable with each other: setups, models and baselines differ.

How to choose, as a rule of thumb [Likely]: start with TRL on one node, with vLLM colocated, and LoRA if memory is tight. Move to verl or OpenRLHF when the model or the cluster grows, when you need PPO with a separate critic at scale, when you need Megatron for very large models, or when long reasoning rollouts make asynchronous training worth its complexity.

Where this is used

SettingThe infrastructure problemTypical choice
Reasoning models trained with verifiable rewards (session 70)very long answers, huge length spread, the tail of Figure 2large GRPO groups with prefix caching, length caps, partial or interruptible rollouts, asynchronous training with a staleness bound
Classic PPO-RLHF chat alignmentfour models in memory (session 66); reward model and critic forward passesseparate placement (OpenRLHF, verl) or colocation with offloading
Code agents with unit teststhe reward is a program run; sandboxes on CPU workers can be slower than generationa pool of reward workers in parallel with the trainer; asynchronous scoring [Likely]
Multi-turn agents with toolseach episode waits for tool calls; GPUs idle between turnsserver-based asynchronous rollouts (verl lists “server-based asynchronous rollout” for agentic RL [Certain])
Small models and LoRA on one nodememory, not throughputTRL with colocated vLLM; with LoRA only the adapter needs to be synced
VLM alignment with imagesimage tokens make prompts long; the engine must run the vision encoder tooan engine with multimodal support; TRL sends images to a separate vLLM endpoint for multimodal prompts [Certain]
Video-LLMs: video QA, key-moment detection, temporal groundingthousands of visual tokens per prompt, so KV cache memory limits the decode batch (75.2) even with short answersprefix caching of the shared video prompt across the \(G\) answers; fewer, larger generation GPUs; rewards such as time-span overlap (session 205) [Likely]
Video captioning for retrievalthe reward is a retrieval score of the caption against the clip, which needs an embedding model on GPUthe retrieval model as a separate reward worker, scoring in parallel with generation [Speculation]

Common mistakes

  • Generating with the training framework. generate() under FSDP or ZeRO-3 all-gathers the weights at every decode step and uses static batching. Use an inference engine, even on one GPU.
  • Forgetting the weight sync, or syncing the wrong thing. If the engine keeps the old weights, the run is silently off-policy by more and more versions. Log the engine’s weight version with every batch and assert the staleness you expect. With LoRA, merge or load the adapter in the engine, and check the tied embeddings and renamed layers.
  • Recomputing the “old” log-probs with the current weights in an asynchronous run. Then the ratio is 1 at the start and the staleness is invisible to the loss. Use the behaviour log-probs from the engine (or the decoupled objective 75.8).
  • Trusting the engine’s log-probs blindly. They may be computed before temperature scaling, in a different precision, or with different kernels. Check the mean and maximum of \(\log\pi_\text{trainer} - \log\pi_\text{engine}\) at staleness 0; use TIS (75.9) when the gap is large.
  • Different sampling settings in the engine and the loss. If the engine samples at temperature 0.7 or with top-p, the behaviour policy is that modified distribution, not the raw softmax. The ratio must use the same distribution. [Likely]
  • Unbounded staleness. A fast engine with a slow trainer builds a long queue of old data. Throttle generation so no sample is more than a few versions old.
  • Ignoring the long tail. A few answers at the length cap can double the step time. Log the length distribution and the number of cut answers per step, not only the mean length.
  • Out of memory when switching phases in a colocated setup. The engine’s KV cache reservation (for vLLM, a fraction of GPU memory) plus the trainer’s state must fit, or one of them must be offloaded. Lower the engine’s memory fraction or use its sleep mode. [Likely]

Interview questions

Why is rollout generation usually the bottleneck in RLHF and RLVR?basic
Generation is sequential: one token per sequence per step, and each step reads all weights and the KV cache from memory, so it is memory-bound (intensity about \(b\) FLOP/byte for a batch of \(b\), against a ridge point of about 295 on an H100). The batch is limited by KV-cache memory. Answers have very different lengths and a synchronous step waits for the longest one, so many slots are idle. Training, by contrast, processes all tokens in parallel at high MFU. OpenRLHF reports that generation often takes over 90% of the runtime.
Why not just call model.generate() in the trainer?basic
It uses static batching (all sequences wait for the longest), a contiguous KV cache that wastes memory and so limits the batch, and usually no CUDA graphs. Under FSDP or ZeRO-3 each decode step must all-gather sharded weights for a single token per sequence. Inference engines such as vLLM and SGLang add a paged KV cache, continuous batching, prefix caching (very useful for GRPO groups that share a prompt), fused kernels and CUDA graphs.
What is weight sync, and how long does it take for a 70B model?core
After each optimizer step, the inference engine’s copy of the policy must be replaced by the new weights. In bf16 that is \(2N\) bytes, 140 GB for 70B. The lower bound is \(2N / \text{BW}\): 2.8 s over one 400 Gb/s InfiniBand link, 0.35 s over eight links in parallel. In practice the trainer’s sharded layout (FSDP or TP×PP) differs from the engine’s layout, so the transfer includes gathering and resharding. Colocated systems avoid the network but must reshard in GPU memory. Going through a checkpoint on disk is much slower.
Colocated or separate GPUs for the inference engine: what are the trade-offs?core
Colocated (hybrid engine): every GPU does both jobs in turn, so there is no idle group and the sync is local; but trainer and engine share memory (offload or sleep modes needed), the switch costs time, and the phases cannot overlap. Separate: each side has full memory and its own parallel layout, and asynchronous pipelining becomes possible; but in a synchronous loop each side idles while the other works, the split must be balanced (about a share \(W_\text{gen}/(W_\text{gen}+W_\text{train})\) of GPUs on generation), and weights travel over the network. TRL defaults to colocated vLLM; OpenRLHF separates roles with Ray; verl supports flexible placement.
What is asynchronous RL for LLMs, and what does it cost?core
The engine generates batch \(t+1\) while the trainer trains on batch \(t\), so both GPU groups stay busy and the step time becomes \(\max(T_\text{gen}, T_\text{train}) + T_\text{sync}\) instead of the sum. The cost: the data is generated by a policy \(k \ge 1\) versions old, so it is off-policy. The gradient is computed from where the policy was, which adds bias and, as in a delayed feedback loop, lowers the safe learning rate (\(\eta_\text{max} = \sin(\pi/(2(2k+1)))\) in the Gaussian toy). Fixes: importance ratios against the behaviour log-probs, the decoupled PPO objective, and a bound on staleness.
Explain the decoupled PPO objective used by AReaL.advanced
Standard PPO uses one old policy for two jobs: the importance correction (samples came from it) and the trust region (do not move far from it). With stale samples these differ. The decoupled form (Hilton et al., 2021) multiplies the clipped PPO term, with ratio \(u = \pi_\theta / \pi_\text{prox}\), by the weight \(\pi_\text{prox} / \pi_\text{behav}\). The proximal policy is recent (AReaL: the weights just before the update), so the clip limits each update; the behaviour policy is whatever sampled the tokens, possibly several versions old, and its weight corrects the off-policy sampling. AReaL combines it with a staleness bound (\(\eta = 4\) or 8).
Why are vLLM’s log-probs different from the trainer’s for the same weights, and what do you do about it?advanced
Different kernels, precision (bf16 vs fp32 softmax, sometimes FP8 or quantized weights in the engine), different reduction orders, and sometimes log-probs taken before temperature or top-p processing. So the sampling distribution is not exactly the one the trainer differentiates, and “on-policy” RL is slightly off-policy. Measure \(\log\pi_\text{trainer} - \log\pi_\text{engine}\) at staleness 0. Ask the engine for processed log-probs (the ones it sampled from). Use truncated importance sampling: weight each token’s loss by \(\min(\pi_\text{trainer}/\pi_\text{engine}, C)\) (Yao et al., 2025).
How do partial or interruptible rollouts help with the long tail?advanced
A synchronous step waits for its longest answer, so the slots of short answers sit idle. Kimi k1.5 caps the tokens per rollout per iteration and continues unfinished answers in the next iteration from a replay buffer. AReaL interrupts generation when new weights arrive, loads them, recomputes the KV cache and continues. Both keep the GPUs busy with new work instead of waiting. The cost is that one answer contains tokens from several policy versions, so each token must keep its own behaviour log-prob for the importance ratio.
In a GRPO setup with 8 GPUs, generation is 80% of the work. How would you split them for an asynchronous pipeline?core
Balance the two sides of (75.5): \(W_\text{gen}/g = W_\text{train}/(G-g)\) gives \(g = 0.8 \times 8 = 6.4\), so 6 or 7 GPUs for generation. With 400 GPU-seconds per step and 6 + 2: generation 53.3 s, training 40 s, so the step is 53.3 s plus sync, and the training GPUs idle about 25% of the time. With 7 + 1, training takes 80 s and becomes the bottleneck. In practice also consider memory (the trainer must fit its sharded states on its GPUs) and parallel layout limits (tensor parallel sizes that divide the number of heads).
Which library would you pick for (a) a LoRA GRPO run on one 8-GPU node, (b) PPO with a 70B policy and separate critic on 64 GPUs?core
(a) TRL: GRPOTrainer with use_vllm=True in colocate mode, or server mode with some GPUs for vLLM; LoRA keeps memory low and only the adapter must be synced. (b) verl or OpenRLHF: both handle multi-node placement of actor, critic, reference and reward models with an inference engine; verl offers a Megatron backend for very large models and resharding with its 3D-HybridEngine; OpenRLHF places roles on GPUs with Ray and trains with DeepSpeed ZeRO. Either way, plan the memory as in session 66 (four models) and measure where the step time goes before tuning.

Exercises

  1. Four answers are decoded together with static batching. Their lengths are 200, 300, 400 and 1,500 tokens. What share of slot time does useful work, by (75.3)?
  2. A 32B policy in bf16 is synced over one 400 Gb/s link. What is the lower bound on the sync time? How does it change with LoRA adapters of 100 MB?
  3. Generation takes 300 GPU-seconds and training 100 GPU-seconds per step on 8 GPUs, with a 2 s sync. Compute the step time for separate synchronous with 6 + 2 GPUs, and for asynchronous with 6 + 2.
  4. Using (75.7), what is the largest stable step size in the Gaussian toy for staleness \(k = 3\)? Is \(\eta = 0.25\) stable?
  5. The engine sampled a token with probability 0.10, and the trainer computes 0.30 for the same weights. What is the TIS weight with \(C = 2\)? And if the trainer computes 0.15?
  6. Why does a GRPO group of \(G = 16\) answers per prompt benefit from prefix caching more than PPO with one answer per prompt?
Answerscheck
1. Useful work \(200 + 300 + 400 + 1500 = 2400\); slot time \(4 \times 1500 = 6000\); \(U = 0.40\). Most of the loss is the one long answer. 2. \(2 \times 32 = 64\) GB; \(64 / 50 = 1.28\) s. With LoRA only the adapter moves: \(0.1 / 50 = 0.002\) s, plus the cost of merging it into the engine’s weights or loading it as an adapter. 3. Synchronous: \(300/6 + 100/2 + 2 = 50 + 50 + 2 = 102\) s. Asynchronous: \(\max(50, 50) + 2 = 52\) s, about twice as fast, with data one version old. 4. \(\eta_\text{max}(3) = \sin(\pi/14) = 0.2225\); \(\eta = 0.25\) is above it, so the uncorrected delayed update is unstable. 5. The ratio is \(0.30/0.10 = 3\), capped at \(C = 2\), so \(w = 2\). With 0.15 the ratio is 1.5, below the cap, so \(w = 1.5\). 6. All 16 answers start from the same prompt, so its KV cache is computed once and shared by 16 sequences; with long prompts (system prompt, documents, images or video frames) this saves a large share of prefill compute and KV memory. PPO with one answer per prompt has nothing to share across sequences except a common system prompt.

Key takeaways

  • An RL step is a distributed dataflow: rollout, scoring, training, and weight sync (75.1). Generation usually dominates; OpenRLHF reports over 90% of runtime.
  • Decoding is memory-bound (75.2) and waits for the longest answer (75.3). Use an inference engine (vLLM, SGLang) with continuous batching and prefix caching, never the trainer’s generate().
  • Weight sync moves \(2N\) bytes per update (75.4): well under a second for 7B on InfiniBand, seconds for 70B; much slower through disk.
  • Colocated (hybrid engine) keeps all GPUs busy and syncs locally; separate GPUs allow asynchronous pipelining (75.5) but must be balanced.
  • Asynchronous RL trains on data that is \(k\) versions old. Ignoring this is a delayed feedback loop that lowers the safe learning rate (75.6), (75.7); use behaviour log-probs, the decoupled objective (75.8) and a staleness bound.
  • Even synchronous RL is slightly off-policy because engine and trainer compute different probabilities; measure the gap and use truncated importance sampling (75.9).
  • TRL: simplest, one node, colocated or server vLLM. verl: hybrid controller, FSDP or Megatron, vLLM or SGLang, resharding engine, async recipes. OpenRLHF: Ray, vLLM and DeepSpeed, roles on separate GPUs.

Key formulas

\[ T_\text{step} = T_\text{gen} + T_\text{score} + T_\text{train} + T_\text{sync}, \qquad I_\text{decode} \approx \frac{2Nb}{2N + M_\text{KV}}, \qquad U_\text{static} = \frac{\sum_i L_i}{S \max_i L_i} \]
\[ T_\text{sync} \ge \frac{2N}{\text{BW}_\text{link}}, \qquad T_\text{async} = \max\!\Big(\frac{W_\text{gen}}{g}, \frac{W_\text{train}}{G-g}\Big) + T_\text{sync} \]
\[ \mu_{t+1} = \mu_t - 2\eta(\mu_{t-k} - c), \quad \eta_\text{max} = \sin\frac{\pi}{2(2k+1)}, \qquad w_t = \min\!\Big(\frac{\pi_\text{trainer}}{\pi_\text{engine}}, C\Big) \]

References for this session

  1. Hugging Face (code). TRL library
  2. Sheng et al. (2024). HybridFlow: A Flexible and Efficient RLHF Framework (verl)
  3. Hu et al. (2024). OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
  4. Hugging Face (code). TRL library
  5. Sheng et al. (2024). HybridFlow: A Flexible and Efficient RLHF Framework (verl)
  6. Hu et al. (2024). OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
  7. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  8. Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
  9. vLLM project (docs). vLLM documentation
  10. Schulman et al. (2017). Proximal Policy Optimization Algorithms
  11. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
  12. Noukhovitch et al. (2024). Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
  13. Fu et al. (2025). AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
  14. Kimi Team (2025). Kimi k1.5: Scaling Reinforcement Learning with LLMs
  15. Yao et al. (2023). DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales
  16. Yao, Liu et al. (2025). Your Efficient RL Framework Secretly Brings You Off-Policy RL Training
  17. Hilton, Cobbe and Schulman (2021). Batch size-invariance for policy optimization