LLM & VLM courseBlog
65Track 4: Post-training and RL · session 13 of 25

Value functions, advantage and GAE

REINFORCE pushes every token of an answer up or down by the same noisy reward. This session adds a critic: a value function that predicts the expected reward from each state. With it we get the advantage of each token, the TD error, and Generalized Advantage Estimation (GAE), which uses two numbers, γ and λ, to trade bias for variance.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: BeginnerBefore this: Sessions 63 and 64

What you will learn

  • The return, the state value V(s) and the action value Q(s, a), and how the Bellman equations link them
  • The advantage A = Q − V, and why it lowers the variance of the policy gradient without adding bias
  • How a critic is learned with Monte Carlo targets or with temporal-difference (TD) targets, and what the TD error means
  • The bias–variance trade-off between one-step TD and full Monte Carlo, with numbers
  • Generalized Advantage Estimation: the formula, the backward recursion, and the roles of γ and λ
  • How GAE is computed per token in RLHF (reward at the EOS token, KL penalty on every token), and the decay problem for long answers
  • Actor–critic training, the cost of a separate critic for LLMs, and why GRPO removes it

Why this matters

REINFORCE works, but it is noisy. One sampled answer gets one reward, and every token in it is pushed up or down by that same number. To learn from fewer samples, we need a way to say “this token was better than usual” or “worse than usual”. That is what value functions and the advantage give us.

In session 64 the gradient was \(\E[\sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, G_t]\), where \(G_t\) is the reward that followed. Subtracting a baseline made it less noisy. This session makes the baseline smart: a second network, the critic, learns to predict the expected reward from every state. With it we can build the advantage of each action, and with Generalized Advantage Estimation (GAE) we can choose how much to trust the critic.

This is exactly what PPO does in classic RLHF. InstructGPT-style training keeps four models in memory: the policy, a frozen reference model, a reward model, and a value model (the critic). [Certain] The value model and GAE are the parts in this session. GRPO, used for DeepSeek-R1-style reasoning training, removes the value model and uses the average reward of a group of answers instead. [Certain] You can only understand why that trade is worth it once you know what the critic does.

Value functions: how good is a state?

Recall the words from session 63. At step \(t\) the agent is in a state \(s_t\), takes an action \(a_t\) from its policy \(\pi(a \mid s)\), gets a reward \(r_t\), and moves to \(s_{t+1}\). For an LLM, the state is the prompt plus the tokens written so far, the action is the next token, and the policy is the model itself.

The return from step \(t\) is the sum of all future rewards, each later one multiplied by a discount \(\gamma \in [0, 1]\):

\[ G_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \dots = \sum_{k=0}^{T-t-1} \gamma^k\, r_{t+k} \tag{65.1} \]

The return is random: it depends on which tokens are sampled later. A value function is its average. There are two kinds:

\[ V^\pi(s) = \E_\pi\big[\, G_t \mid s_t = s \,\big], \qquad Q^\pi(s, a) = \E_\pi\big[\, G_t \mid s_t = s,\ a_t = a \,\big] \tag{65.2} \]
Read the symbols
\(V^\pi(s)\)
state value: the expected return if you start in \(s\) and follow \(\pi\) from there
\(Q^\pi(s, a)\)
action value: the expected return if you start in \(s\), take action \(a\) first, and then follow \(\pi\)
\(\gamma\)
discount factor; \(\gamma = 0.99\) means a reward 100 steps away counts \(0.99^{100} = 0.37\) times as much as one now. For LLM responses, \(\gamma = 1\) is common [Certain]
\(T\)
the step at which the episode ends (for an LLM: the end-of-sequence token)

The two are connected. \(V\) is the average of \(Q\) over the actions the policy would pick, and \(Q\) is one reward plus the value of where you land. These are the Bellman equations:

\[ V^\pi(s) = \E_{a \sim \pi(\cdot \mid s)}\big[ Q^\pi(s, a) \big], \qquad Q^\pi(s, a) = \E\big[\, r_t + \gamma\, V^\pi(s_{t+1}) \mid s_t = s,\ a_t = a \,\big] \tag{65.3} \]

The second equation is the key idea of this whole session: a value can be estimated from one real reward plus the value of the next state. We do not need to wait for the end of the episode.

For an LLM, transitions are deterministic: if the state is “prompt + tokens so far” and the action is token \(a\), the next state is just “prompt + tokens + \(a\)”. All the randomness is in the policy and in the reward. [Certain]

The advantage: better or worse than usual?

The advantage of an action is how much better it is than the average action in that state:

\[ A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s), \qquad \E_{a \sim \pi(\cdot \mid s)}\big[ A^\pi(s, a) \big] = 0 \tag{65.4} \]

The second part follows from (65.3): the average of \(Q\) is \(V\), so the average advantage is zero. Positive advantage means “this token was better than what the model usually writes here”; negative means worse. Using it in the policy gradient gives:

\[ \nabla_\theta J(\theta) = \E_\pi\Big[ \sum_{t} \nabla_\theta \log \pi_\theta(a_t \mid s_t)\; A^\pi(s_t, a_t) \Big] \tag{65.5} \]

This is the same expected gradient as REINFORCE: subtracting \(V(s_t)\) changes nothing on average, because \(\E_a[\nabla_\theta \log \pi_\theta(a \mid s)] = 0\) for any fixed state. [Certain] But the variance is much lower. Example: if every answer to a prompt gets a reward between 0.8 and 0.9, REINFORCE pushes all of them up. With the advantage, the 0.9 answers go up and the 0.8 answers go down, which is the information we actually want.

The problem: we do not know \(Q^\pi\) or \(V^\pi\). We must estimate them. The rest of this session is about how.

Learning the critic: Monte Carlo or temporal difference

The critic is a network \(V_\phi(s)\) with weights \(\phi\) that tries to predict \(V^\pi(s)\). We train it by regression: pick a target for each visited state, then minimize the squared error. There are two classic choices of target.

  • Monte Carlo (MC): wait until the episode ends and use the real return \(G_t\). The target is correct on average (unbiased), but it is noisy, because it adds up the randomness of every later step.
  • Temporal difference (TD): use one real reward plus the critic’s own guess for the next state, \(r_t + \gamma V_\phi(s_{t+1})\). This is called bootstrapping: learning a guess from a guess. The target has only one step of randomness (low variance), but if \(V_\phi\) is wrong, the target is wrong too (biased).

The difference between the TD target and the current guess is the TD error:

\[ \delta_t = r_t + \gamma\, V(s_{t+1}) - V(s_t) \tag{65.6} \]
Read the symbols
\(\delta_t\)
the “surprise” at step \(t\): how much better things turned out after one step than the critic expected
\(V(s_{t+1})\)
the critic’s value of the next state; it is 0 if \(s_{t+1}\) is the end of the episode

The TD error is also a one-step advantage estimate. If the critic were perfect (\(V = V^\pi\)), then \(\E[\delta_t \mid s_t, a_t] = Q^\pi(s_t, a_t) - V^\pi(s_t) = A^\pi(s_t, a_t)\), by (65.3). The GAE paper states this exactly: \(\delta_t\) is an unbiased advantage estimate when \(V = V^\pi\), and biased otherwise. [Certain]

Algorithm 1: learning a value function with TD(0) and with Monte Carlo
  1. input: policy \(\pi\), step size \(\alpha\), initial guesses \(V(s)\) for all states, \(V(\text{end}) = 0\)
  2. for each episode do
  3.   for each step \(t\): take \(a_t \sim \pi(\cdot \mid s_t)\), observe \(r_t, s_{t+1}\)
  4.     TD(0): \(V(s_t) \leftarrow V(s_t) + \alpha\, \delta_t\) // formula (65.6), update at once
  5.   Monte Carlo: after the episode ends, for every visited \(s_t\): \(V(s_t) \leftarrow V(s_t) + \alpha\,(G_t - V(s_t))\) // formula (65.1)

Figure 1 runs both lines of Algorithm 1 side by side, one step at a time, on a corridor. A robot starts in cell 3 of seven cells and moves right with probability \(\pi(\text{right})\), otherwise left. Reaching the right end gives \(+1\), the left end \(-1\), and every other step gives 0. Both critics start at \(V = 0\) and see exactly the same episodes.

Algorithm in the figure: TD(0) and Monte Carlo on the same episodes
  1. \(V_{\text{TD}}(s) \leftarrow 0\), \(V_{\text{MC}}(s) \leftarrow 0\) for every state; \(V(\text{end}) = 0\)
  2. for each episode: \(s_0 \leftarrow 3\)
  3. take \(a_t \sim \pi\); observe \(r_t\) and \(s_{t+1}\)
  4. TD: \(\delta_t \leftarrow r_t + \gamma V_{\text{TD}}(s_{t+1}) - V_{\text{TD}}(s_t)\) // (65.6)
  5. TD: \(V_{\text{TD}}(s_t) \leftarrow V_{\text{TD}}(s_t) + \alpha\, \delta_t\) // at once
  6. repeat lines 3–5 until \(s_{t+1}\) is an end cell
  7. MC: for \(t = T-1\) down to 0: \(G_t \leftarrow r_t + \gamma\, G_{t+1}\) // (65.1)
  8. MC: \(V_{\text{MC}}(s_t) \leftarrow V_{\text{MC}}(s_t) + \alpha\,(G_t - V_{\text{MC}}(s_t))\) // only after the episode
\(V_{\text{TD}}\)\(V_{\text{MC}}\)true \(V^\pi\)
RMS error of the values (this run)
TD(0)Monte Carlo
The estimate of \(V(s{=}4)\)
TD(0)Monte Carlotrue value
Figure 1. TD(0) and Monte Carlo learning the value function of a corridor, live. The bars under the cells are the two critics’ current values; the dark ticks are the true values \(V^\pi\), found by solving the Bellman equations (65.3) for the five cells. TD changes one value after every step; Monte Carlo changes all visited values once, after the episode, walking backward.
How to read Figure 1

Step through the first episode at Slow. TD updates right away, but in the first episode its \(\delta_t\) is 0 at every step until the robot reaches an end: only the last cell learns anything. That information then spreads back one cell per visit. Monte Carlo waits, then pushes every visited cell toward the full return, which is \(+1\) or \(-1\). Now press Play at Fast. The orange MC bars jump up and down after each episode: every target is a whole random outcome. The blue TD bars move in small steps. With \(\alpha = 0.05\) and seeds 1–50, TD has the lower error after 100 episodes in all 50 runs (mean RMS error 0.082 against 0.263). Raise \(\alpha\) to 0.3: both get noisier, MC much more. Lower \(\gamma\): the true values shrink toward 0, because the end is several steps away.

Which one learns faster in general? Figure 2 averages 100 runs of the classic test from Sutton and Barto (Example 6.2): a random walk over five states A to E. Every episode starts in C and moves left or right with probability ½. Leaving on the right gives reward 1; leaving on the left gives 0. The true values are \(1/6, 2/6, \dots, 5/6\). [Certain] All guesses start at 0.5.

TD(0)Monte Carlotrue values
Figure 2. TD versus Monte Carlo on the five-state random walk. Top: the true values (grey) and the values each method has after 100 episodes in run 1. Bottom: the root-mean-square (RMS) error over the five states, averaged over 100 seeded runs, for 100 episodes. Both methods see exactly the same episodes.
How to read Figure 2

Lower is better. With the default step sizes, TD reaches a lower error than Monte Carlo after 100 episodes. Raise the MC step size: it learns faster at first but then stays noisy, because each MC target is the full random outcome (0 or 1). Raise the TD step size to 0.3: it also gets noisy. A step size that is too small makes both slow. This is the bias–variance trade-off you will see again in GAE: TD targets are less noisy, MC targets are not biased by bad guesses.

On this task, TD is faster than MC for the step sizes Sutton and Barto test. [Certain] This is not a theorem for all problems, but it is common in practice. [Likely] Deep RL critics use TD-style targets for this reason.

Bias and variance: from one step to many

There is a whole range between TD and MC. We can use \(k\) real rewards and then bootstrap with the critic. Write it with TD errors, and the middle terms cancel (each \(+\gamma V(s_{t+1})\) cancels the next \(-V(s_{t+1})\)):

\[ \hat A_t^{(k)} = \sum_{l=0}^{k-1} \gamma^l \delta_{t+l} = -V(s_t) + r_t + \gamma r_{t+1} + \dots + \gamma^{k-1} r_{t+k-1} + \gamma^k V(s_{t+k}) \tag{65.7} \]
  • \(k = 1\): \(\hat A^{(1)}_t = \delta_t\). One random reward, then trust the critic. Low variance, high bias if the critic is wrong.
  • \(k\) up to the end of the episode: \(\hat A^{(\infty)}_t = G_t - V(s_t)\). The critic is only a baseline. No bias, high variance.

Here “bias” means: on average, the estimate is not equal to the true advantage (up to the baseline). “Variance” means: from one sample to the next, the estimate jumps around a lot. A policy gradient with a biased advantage pushes in a slightly wrong direction. A policy gradient with a high-variance advantage pushes in the right direction on average but needs many more samples. A3C (Mnih et al., 2016) used a fixed \(k\) for this reason. [Certain] GAE removes the need to pick one \(k\).

Generalized Advantage Estimation

Instead of choosing one \(k\), GAE takes an exponentially weighted average of all of them, with weight \((1 - \lambda)\lambda^{k-1}\) for \(\hat A^{(k)}\). After the algebra, this becomes a simple sum of TD errors (Schulman et al., 2015, equation 16): [Certain]

\[ \hat A_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^{T-t-1} (\gamma \lambda)^l\, \delta_{t+l} = \delta_t + \gamma\lambda\, \delta_{t+1} + (\gamma\lambda)^2 \delta_{t+2} + \dots \tag{65.8} \]
Read the symbols
\(\lambda \in [0, 1]\)
how far into the future we look at real rewards before trusting the critic; it plays the same role as in TD(\(\lambda\))
\((\gamma\lambda)^l\)
the weight of the surprise \(l\) steps later; it shrinks geometrically
\(T - t - 1\)
the number of steps left in the episode (or in the collected segment)

You never compute the sum directly. Going from the last step backward, each advantage is the TD error plus a discounted copy of the next advantage:

\[ \hat A_t = \delta_t + \gamma \lambda\, \hat A_{t+1}, \qquad \hat A_{T} = 0 \tag{65.9} \]

This is one backward pass over the sequence, \(O(T)\) work. The two ends of the range are the estimators from the previous section:

\[ \lambda = 0:\ \ \hat A_t = \delta_t, \qquad\qquad \lambda = 1:\ \ \hat A_t = \sum_{l} \gamma^l r_{t+l} - V(s_t) = G_t - V(s_t) \tag{65.10} \]

The GAE paper says: \(\lambda = 0\) is biased unless \(V\) is exact but has much lower variance; \(\lambda = 1\) has no bias whatever \(V\) is, but high variance. [Certain] Values in between trade the two. A useful number is the effective horizon \(1/(1 - \gamma\lambda)\): with \(\gamma = 0.99\) and \(\lambda = 0.95\), \(\gamma\lambda = 0.9405\) and the horizon is about 16.8 steps.

The paper’s experiments found the best results at intermediate values: \(\lambda\) between 0.92 and 0.99 on cart-pole and between 0.96 and 0.99 on 3D bipedal walking. [Certain] PPO then used \(\gamma = 0.99\), \(\lambda = 0.95\) for MuJoCo, Roboschool and Atari. [Certain] Figure 3 lets you see what \(\lambda\) does to a sequence.

critic value \(V(s_t)\)TD error \(\delta_t\)GAE advantage \(\hat A_t\)
Figure 3. GAE on one response of \(T\) tokens. The only reward is 1.0 at the last token. The critic’s values (top) rise from about 0.5 to 0.8 with some error. Bottom: the TD errors \(\delta_t\) (grey dots) and the GAE advantages \(\hat A_t\) (orange bars) from formula (65.9).
How to read Figure 3

At \(\lambda = 0\), each token’s advantage is just its own TD error, which here is mostly the critic’s own wiggles: the final reward reaches only the last token. At \(\lambda = 1\), every token gets “final reward minus my value”, so the reward reaches all tokens, but each token’s advantage no longer depends on what happened right after it. The readout shows how much of the final reward reaches the first token directly: \((\gamma\lambda)^{T-1}\). Set \(T = 256\) with \(\lambda = 0.95\): almost nothing reaches the start. Long chain-of-thought answers have thousands of tokens, which is why some reasoning recipes set \(\lambda = 1\) for the critic.

Watching bias and variance trade off

We can make the trade-off exact with a small model. Take an episode of \(T\) steps. Every reward has random noise with standard deviation \(\sigma\), independent across steps. The critic is off by the same amount \(b\) in every state after the first (\(V(s_l) = V^\pi(s_l) + b\) for \(l \ge 1\), and 0 at the end). With \(\gamma = 1\), the error of \(\hat A_0\) compared with the true “\(Q\) minus baseline” has

\[ \text{bias} = b\,(1 - \lambda^{T-1}), \qquad \text{variance} = \sigma^2 \sum_{l=0}^{T-1} \lambda^{2l}, \qquad \text{MSE} = \text{bias}^2 + \text{variance} \tag{65.11} \]

Formula (65.11) follows from (65.8): the critic errors telescope away except for the part that \(\lambda\) cuts off, and each reward’s noise enters with weight \(\lambda^l\). We checked both formulas with a 20,000-sample simulation. Figure 4 plots them.

\(\text{bias}^2\)variance\(\text{MSE} = \text{bias}^2 + \text{variance}\)
Figure 4. Bias and variance of the GAE estimate of the first step’s advantage, as a function of \(\lambda\), in the toy model above (\(\gamma = 1\)). The vertical line is the chosen \(\lambda\); the ring marks the \(\lambda\) with the lowest mean squared error.
How to read Figure 4

With the defaults (\(T = 20\), \(b = 0.5\), \(\sigma = 0.1\)), the best \(\lambda\) is 0.98: better than both TD (\(\lambda = 0\)) and MC (\(\lambda = 1\)). Set \(b = 0\) (a perfect critic): the bias vanishes and \(\lambda = 0\) wins. Set \(\sigma = 0\) (no noise): variance vanishes and \(\lambda = 1\) wins. Real problems are in between, which is why tuned values like 0.95 are common. This model is a simplification; in a real task the critic errors differ per state and can partly cancel.

Worked example: GAE by hand for a five-token answer

Prompt: “What is the capital of France?” The policy writes five tokens: “The”, “ capital”, “ is”, “ Paris”, “.”. As in RLHF, each token gets a small KL penalty \(-\beta\,\text{kl}_t\) with \(\beta = 0.1\) and per-token KL estimates \((0.2, 0.1, 0.0, 0.5, 0.1)\), and the last token also gets the reward-model score 1.0. So the rewards are \(r = (-0.02, -0.01, 0, -0.05, 0.99)\).

The critic says \(V = (0.60, 0.70, 0.75, 0.85, 0.95)\), and \(V = 0\) after the last token. Use \(\gamma = 1\), \(\lambda = 0.95\).

TD errors (65.6): \(\delta_0 = -0.02 + 0.70 - 0.60 = 0.08\), \(\delta_1 = -0.01 + 0.75 - 0.70 = 0.04\), \(\delta_2 = 0 + 0.85 - 0.75 = 0.10\), \(\delta_3 = -0.05 + 0.95 - 0.85 = 0.05\), \(\delta_4 = 0.99 + 0 - 0.95 = 0.04\).

Backward recursion (65.9): \(\hat A_4 = 0.04\); \(\hat A_3 = 0.05 + 0.95 \times 0.04 = 0.088\); \(\hat A_2 = 0.10 + 0.95 \times 0.088 = 0.1836\); \(\hat A_1 = 0.04 + 0.95 \times 0.1836 = 0.2144\); \(\hat A_0 = 0.08 + 0.95 \times 0.2144 = 0.2837\).

Compare the ends: with \(\lambda = 0\) the advantages are the TD errors \((0.08, 0.04, 0.10, 0.05, 0.04)\); with \(\lambda = 1\) they are \(G_t - V(s_t) = (0.31, 0.23, 0.19, 0.09, 0.04)\). GAE with 0.95 lies between them. The critic targets are \(\hat A_t + V(s_t) = (0.8837, 0.9144, 0.9336, 0.938, 0.99)\). Figure 6 replays these steps.

Formula (65.11) is a model. Figure 5 measures the same trade-off on real episodes. It uses the corridor of Figure 1 with a policy that goes right 70% of the time. Each step’s reward also has a small random part with standard deviation \(\sigma\), the way the per-token KL term of the next section adds a small reward at every token. The critic is deliberately wrong: it knows only part of the true value, \(\hat V(s) = (1 - \kappa)\, V^\pi(s)\), like a critic early in training. For every episode the figure computes \(\hat A_0\) for 21 values of \(\lambda\) and compares it with the true “\(Q\) minus baseline” \(Q^\pi(s_0, a_0) - \hat V(s_0)\). Averaged over many episodes, the errors give the bias and the variance of each \(\lambda\).

Algorithm in the figure: GAE on one episode, then its error
  1. roll out \(\tau = (s_0, a_0, r_0, \dots, s_T)\) with \(\pi\); \(\hat V\) is fixed
  2. for \(t = 0..T-1\): \(\delta_t \leftarrow r_t + \gamma \hat V(s_{t+1}) - \hat V(s_t)\) // (65.6)
  3. \(\hat A_T \leftarrow 0\)
  4. for \(t = T-1\) down to 0: \(\hat A_t \leftarrow \delta_t + \gamma\lambda\, \hat A_{t+1}\) // (65.9)
  5. critic targets \(\hat R_t \leftarrow \hat A_t + \hat V(s_t)\) // (65.13)
  6. for all 21 values of \(\lambda\): error \(\leftarrow \hat A_0 - \big(Q^\pi(s_0, a_0) - \hat V(s_0)\big)\); update its mean and mean square
Advantages of this episode
\(\hat A_t\) at the chosen \(\lambda\)\(\delta_t\) (\(\lambda = 0\))\(G_t - \hat V(s_t)\) (\(\lambda = 1\))
Error of \(\hat A_0\), measured over all episodes so far
\(\text{bias}^2\)variance\(\text{MSE} = \text{bias}^2 + \text{variance}\)
Figure 5. GAE on real episodes of the corridor, with a critic that is too pessimistic. Top: one episode, its TD errors and its advantages, computed backward with the live numbers. Bottom: the \(\text{bias}^2\), variance and mean squared error of \(\hat A_0\) for every \(\lambda\) from 0 to 1 in steps of 0.05, measured from all episodes played so far (the ring marks the lowest MSE). Moving the \(\lambda\) slider does not restart the measurement, because all 21 values are tracked at once; the other sliders restart it.
How to read Figure 5

Step through one episode. Line 2 fills the \(\delta_t\) column from left to right; line 4 fills the \(\hat A_t\) column from right to left, each value reusing the one below it. In the bar chart, the grey dots (\(\lambda = 0\)) are small because the critic expects little, while the blue ticks (\(\lambda = 1\)) carry the full final \(\pm 1\) back to every step. Then press Turbo. With seed 1, after 2,000 episodes \(\lambda = 0\) has MSE 0.274, almost all of it \(\text{bias}^2\) (0.250): it trusts a critic that knows a quarter of the truth. \(\lambda = 1\) has no bias (its measured \(\text{bias}^2\) is near 0) but variance 0.187. The lowest MSE, 0.134, is at \(\lambda = 0.90\), close to the values used in practice. Now set \(\kappa = 0\) (a perfect critic): bias disappears and \(\lambda = 0\) wins. Set \(\kappa = 1\) (the critic always says 0): the best \(\lambda\) moves up to about 0.95. The best \(\lambda\) depends on how good the critic is, which is why it is a tuned hyperparameter.

GAE for LLM responses

In RLHF with PPO, one response is one episode, and each generated token is one step. The reward is mostly sparse: the reward model scores the whole response, and that score is placed on the last token (the end-of-sequence token). Every token also gets a KL penalty that keeps the policy close to the reference model:

\[ r_t = -\beta \left( \log \pi_\theta(a_t \mid s_t) - \log \pi_{\text{ref}}(a_t \mid s_t) \right) + \mathbb{1}[t = T-1]\; r_{\text{RM}}(x, y) \tag{65.12} \]
Read the symbols
\(\beta\)
strength of the KL penalty
\(\pi_{\text{ref}}\)
the frozen starting model (usually the SFT model)
\(\mathbb{1}[t = T-1]\)
1 for the last token of the response, 0 otherwise
\(r_{\text{RM}}(x, y)\)
the reward model’s score for prompt \(x\) and full response \(y\)

This is the setup of the open reproduction of OpenAI’s summarization RLHF by Huang et al. (2024): the score at the EOS token, per-token KL penalties elsewhere, \(\gamma = 1\), \(\lambda = 0.95\), and a value network initialized from the reward model. [Certain] InstructGPT also initialized its value function from the reward model. [Likely] The critic is usually an LLM of similar size with a scalar “value head” (a linear layer from the hidden state to one number) instead of the vocabulary head; it outputs one value per token position.

Figure 6. The worked example, step by step: per-token rewards from (65.12), the critic’s values, the TD errors (65.6), then the GAE advantages computed from right to left (65.9), and finally the critic’s targets.
How to read Figure 6

The rows fill in from top to bottom. The advantage row fills in from right to left: the last token’s advantage is known first, and each earlier one reuses the next. Notice “ is” gets a larger advantage than “ capital”: after “ is” the critic’s value jumped from 0.75 to 0.85, so the critic thinks “ is” moved the answer toward a good ending.

The decay problem for long answers

With \(\gamma = 1\) and \(\lambda = 0.95\), the final reward reaches a token \(l\) steps before the end with direct weight \(0.95^{l}\): 0.20 after 31 tokens, 0.006 after 99, and about \(6 \times 10^{-23}\) after 999. Early tokens of a long answer then learn almost only from the critic’s own TD errors. If the critic is poor at the start of training, those advantages are mostly noise or bias. Yuan et al. (2025) call this “reward signal decay”. Their fix (VC-PPO) is to pre-train the value model before PPO and to decouple GAE: \(\lambda_{\text{critic}} = 1.0\) for the critic’s targets, while the policy keeps \(\lambda_{\text{actor}} = 0.95\). [Certain]

Actor–critic: a policy plus a learned critic

An actor–critic method trains two things at once: the actor (the policy \(\pi_\theta\)) and the critic (the value function \(V_\phi\)). The critic learns to predict returns; the actor uses the critic’s advantages to decide which actions to make more likely. The critic is trained by regression on the GAE-based targets (also called \(\lambda\)-returns):

\[ \mathcal{L}_V(\phi) = \frac{1}{2}\, \E_t\Big[ \big( V_\phi(s_t) - \hat R_t \big)^2 \Big], \qquad \hat R_t = \hat A_t + V_{\phi_{\text{old}}}(s_t) \tag{65.13} \]

The full loss is the policy term, the value term, and often an entropy bonus \(\mathcal{H}\) that keeps the policy from becoming too certain too early:

\[ \mathcal{L}(\theta, \phi) = -\E_t\Big[ \log \pi_\theta(a_t \mid s_t)\, \hat A_t \Big] + c_v\, \mathcal{L}_V(\phi) - c_e\, \E_t\big[ \mathcal{H}(\pi_\theta(\cdot \mid s_t)) \big] \tag{65.14} \]
Read the symbols
\(\hat A_t\)
the GAE advantage, treated as a constant (no gradient flows into it)
\(\hat R_t\)
the critic’s target, computed once with the critic from before the update
\(c_v, c_e\)
weights of the value loss and the entropy bonus; PPO’s Atari setting used \(c_v = 1\), \(c_e = 0.01\) [Certain]

PPO keeps this structure but replaces the policy term with a clipped ratio objective, so it can take several gradient steps on the same batch (session 66). Everything about \(\hat A_t\) in this session carries over unchanged.

Figure 7. Actor–critic in LLM RLHF. The policy (actor) writes a response. The reward model scores it and the reference model gives the KL penalty, which together make the per-token rewards (65.12). The value model (critic) gives \(V(s_t)\) for every token. GAE turns rewards and values into advantages (for the actor’s loss) and targets (for the critic’s loss).
Algorithm 2: one actor–critic iteration with GAE for an LLM
  1. input: policy \(\pi_\theta\), critic \(V_\phi\), reference \(\pi_{\text{ref}}\), reward model, prompts, \(\gamma, \lambda, \beta\)
  2. sample responses \(y \sim \pi_\theta(\cdot \mid x)\) for a batch of prompts // the rollout
  3. score each response with the reward model; compute per-token KL against \(\pi_{\text{ref}}\)
  4. build per-token rewards \(r_t\) // formula (65.12)
  5. run the critic once: \(V_t = V_\phi(s_t)\) for every response token; set \(V = 0\) after the last token
  6. for \(t = T-1\) down to 0: \(\delta_t \leftarrow r_t + \gamma V_{t+1} - V_t\); \(\hat A_t \leftarrow \delta_t + \gamma\lambda \hat A_{t+1}\) // (65.6), (65.9)
  7. \(\hat R_t \leftarrow \hat A_t + V_t\) // critic targets, (65.13)
  8. normalize \(\hat A\) over the batch (subtract mean, divide by std), using only real tokens
  9. update \(\theta\) and \(\phi\) by minimizing (65.14) // PPO: clipped policy loss, several epochs

Figure 8 runs Algorithm 2 on the token toy of session 63 and session 64: prompt “2+3=”, vocabulary 5, 6, 7, “.”, ⏎, at most 4 tokens, reward 1 for an answer whose digits are exactly “5” and that ends with ⏎. The actor is a table of logits indexed by the previous token (a bigram “LLM”, 25 weights). The critic is a table with one value per previous token (5 weights), so, like a real value model, it cannot tell every state apart. The reference policy is the starting policy (uniform), and the per-token reward is (65.12). We use \(\gamma = 1\), plain gradient steps and no advantage normalization, so every number can be followed by hand.

Algorithm in the figure: actor–critic with GAE (Algorithm 2, small)
  1. sample \(N\) answers \(y \sim \pi_\theta(\cdot \mid x)\), keeping \(\log \pi_\theta(y_t \mid s_t)\)
  2. score each answer with the checker: \(R \in \{0, 1\}\)
  3. per-token rewards \(r_t \leftarrow -\beta(\log \pi_\theta - \log \pi_{\text{ref}})\), plus \(R\) at the last token // (65.12)
  4. critic: \(V_t \leftarrow V_\phi(s_t)\); \(V = 0\) after the last token
  5. for \(t = T-1\) down to 0: \(\delta_t \leftarrow r_t + \gamma V_{t+1} - V_t\); \(\hat A_t \leftarrow \delta_t + \gamma\lambda \hat A_{t+1}\) // (65.6), (65.9)
  6. \(\hat R_t \leftarrow \hat A_t + V_t\) // (65.13)
  7. one gradient step on (65.14) for \(\theta\) (step \(\eta_\pi\)) and for \(\phi\) (step \(\eta_v\)), averaged over all tokens
Fraction of correct answers
batchmoving average (20)exact \(J\)
Critic values \(V_\phi\)
\(x\)after 5after 6after 7after .
Actor \(\pi_\theta(\text{next} \mid \text{state})\) and critic \(V_\phi\)
Figure 8. A complete actor–critic with GAE, learning live on a tiny language model. Each row of the batch is one answer; after line 5, every token shows its own advantage \(\hat A_t\) (green: pushed up, red: pushed down). The table next to the batch follows one answer through every quantity of Algorithm 2. \(J\) is exact, computed by enumerating all answers.
How to read Figure 8

Step through the first batch slowly. Line 3 gives every token a reward of exactly 0 in the first batch, because the policy still equals the reference, so the log-ratio is 0; only the last token of a correct answer gets 1. The critic starts at 0, so in the first batch \(\delta_t\) is 0 everywhere except at the last token, and only \(\lambda\) carries the final reward back to the first token. Compare this with Figure 8 of session 64, where every token got the same number: here the tokens of one answer get different advantages once the critic has learned something. Later, a correct answer’s first token “5” gets a smaller advantage than at the start, because the critic already expects a good answer from the prompt. Press Play. The critic learns that “after 5” is a good state and “after 6” is worthless. Now set \(\lambda = 0\): the first token gets credit only through the critic, which knows nothing at first, so learning is slower (over seeds 1–6, \(J\) after 50 iterations averages 0.835 with \(\lambda = 0\) and 0.933 with \(\lambda = 0.95\)). Raise \(\beta\): the KL penalty pulls the policy back toward the uniform reference.

Shared or separate critic?

In small RL problems the actor and critic often share one network body with two heads. For LLMs, the critic is usually a separate model, often initialized from the reward model or the SFT model. [Likely] A separate critic of the same size doubles the trainable weights and the optimizer memory: with AdamW at 16 bytes per parameter, a 7B critic alone needs about 112 GB of training state (session 38). This cost is one of the main reasons GRPO removes the critic and uses the mean reward of several answers to the same prompt as the baseline (session 70). The price: GRPO gives every token of an answer the same advantage, so it has no per-token credit assignment from a critic. [Certain]

Code: GAE and a tiny actor–critic in PyTorch

The function below computes GAE for a batch of right-padded responses, with a mask so that padding never leaks into the advantages. It reproduces the worked example and checks the two special cases. Then a tiny actor–critic learns a 4-token “sentence” from a reward given only at the end. It runs on a CPU in a few seconds and prints the same numbers every time.

import torch
torch.manual_seed(0)

# 1) GAE for a batch of responses, right-padded. mask = 1 on response tokens.
def gae(rewards, values, mask, gamma=1.0, lam=0.95):
    B, T = rewards.shape
    adv = torch.zeros_like(rewards)
    last = torch.zeros(B)
    for t in reversed(range(T)):                       # formula (65.9), right to left
        if t + 1 < T:
            next_v = values[:, t + 1] * mask[:, t + 1]  # V = 0 after the last token
        else:
            next_v = torch.zeros(B)
        delta = rewards[:, t] + gamma * next_v - values[:, t]   # formula (65.6)
        last = (delta + gamma * lam * last) * mask[:, t]
        adv[:, t] = last
    returns = (adv + values) * mask                    # targets for the critic, (65.13)
    return adv, returns

# the worked example: 5 tokens, KL penalty on every token, reward-model score 1.0 at the end
kl = torch.tensor([[0.2, 0.1, 0.0, 0.5, 0.1]])
r = -0.1 * kl; r[0, -1] += 1.0
V = torch.tensor([[0.6, 0.7, 0.75, 0.85, 0.95]])
m = torch.ones(1, 5)
A, R = gae(r, V, m)
print(A.round(decimals=4), R.round(decimals=4))
# [[0.2837, 0.2144, 0.1836, 0.0880, 0.0400]]  [[0.8837, 0.9144, 0.9336, 0.9380, 0.9900]]
# special cases: lambda = 0 gives the TD errors, lambda = 1 gives return minus V
print(gae(r, V, m, lam=0.0)[0].round(decimals=4), gae(r, V, m, lam=1.0)[0].round(decimals=4))
# [[0.08, 0.04, 0.10, 0.05, 0.04]]  [[0.31, 0.23, 0.19, 0.09, 0.04]]

# padding check: the second, shorter response must not see its padding
r2 = torch.zeros(2, 5); r2[0, 4] = 1.0; r2[1, 2] = 1.0
m2 = torch.tensor([[1., 1, 1, 1, 1], [1, 1, 1, 0, 0]])
A2, _ = gae(r2, torch.full((2, 5), 0.5), m2)
print(A2.round(decimals=4))
# [[0.4073, 0.4287, 0.4512, 0.4750, 0.5000], [0.4512, 0.4750, 0.5000, 0, 0]]

# 2) actor-critic on a toy "sentence" task: 4 steps, 3 tokens, the reward comes only at the end
T, K, N = 4, 3, 64
target = torch.tensor([2, 0, 1, 2])
logits = torch.zeros(T, K, requires_grad=True)   # the actor: one softmax per position
values = torch.zeros(T, requires_grad=True)      # the critic: one number per position
opt = torch.optim.Adam([logits, values], lr=0.05)
for it in range(300):
    dist = torch.distributions.Categorical(logits=logits.expand(N, T, K))
    a = dist.sample()                                  # [N, T]: N sampled "sentences"
    rew = torch.zeros(N, T)
    rew[:, -1] = (a == target).float().mean(1)         # sparse reward at the last step
    v = values.expand(N, T)
    adv, ret = gae(rew, v.detach(), torch.ones(N, T), gamma=1.0, lam=0.95)
    adv = (adv - adv.mean()) / (adv.std() + 1e-8)      # advantage normalization
    pg_loss = -(dist.log_prob(a) * adv).mean()         # actor, formula (65.14)
    v_loss = 0.5 * ((v - ret) ** 2).mean()             # critic, formula (65.13)
    opt.zero_grad(); (pg_loss + v_loss).backward(); opt.step()
p = torch.softmax(logits, -1)[torch.arange(T), target]
print(p.detach().round(decimals=3), values.detach().round(decimals=3))
# tensor([0.999, 0.999, 0.999, 0.999]) tensor([1., 1., 1., 1.]): the right token everywhere,
# and the critic predicts the maximum reward 1.0 from every position

Two details matter in real code. The advantages are computed with values.detach() (or under torch.no_grad()), so the policy loss does not train the critic. And the padding mask is used both to stop the recursion at the end of each response and to exclude padding from the mean and standard deviation in normalization.

Where this is used

SettingWhat the critic and GAE doTypical choices
RLHF with PPO (InstructGPT-style chat models)Value model gives per-token \(V\); GAE turns the end-of-sequence score plus KL penalties into per-token advantages\(\gamma = 1\), \(\lambda = 0.95\), critic initialized from the reward model [Certain for Huang et al. 2024]
Long chain-of-thought RL with PPOPre-trained value model; decoupled \(\lambda\) for actor and critic\(\lambda_{\text{critic}} = 1.0\), \(\lambda_{\text{actor}} = 0.95\) (VC-PPO) [Certain]
GRPO, RLOO and similar critic-free methodsNo critic: the baseline is the mean reward of a group of sampled answersSaves a full model in memory; one advantage per answer [Certain]
Robot control, games (the original GAE and PPO setting)Dense rewards, long episodes; separate or shared critic network\(\gamma = 0.99\), \(\lambda = 0.95\) [Certain for PPO’s benchmarks]
VLM agents (web or GUI agents that click through screens)Multi-turn episodes where each turn is a step; a critic helps credit which turn helpedTurn-level or token-level GAE; active research area [Speculation]
Video key-moment detection or video summarization trained with RLThe policy picks frames or clips one by one; the reward (for example, summary quality) comes at the end, so a critic gives each choice its own advantageSparse end reward, \(\gamma\) near 1; critic-free baselines are also common [Likely]
Video-RAG or video QA agents that decide which segment to retrieve nextEach retrieval is an action; the answer’s correctness is the reward; the value estimates how promising the current evidence isShort episodes (a few retrievals), often \(\lambda\) near 1 [Speculation]

Common mistakes

  • Off-by-one in \(V(s_{t+1})\). The critic’s value at position \(t\) must be read from the hidden state before token \(a_t\) is chosen, and the value after the last token must be 0. Mixing these up shifts all advantages by one token.
  • Letting padding into the recursion. Without the mask, the recursion runs into padding tokens and adds their fake values and rewards to the last real tokens. Always stop at the end of each response.
  • Bootstrapping through a true end. When the episode really ended (EOS), \(V(s_T) = 0\). When it was only cut off by a length limit, a true end did not happen; some implementations bootstrap with \(V(s_T)\) there, others give a penalty. Know which one your code does.
  • Training the critic through the advantage. If \(\hat A_t\) is not detached, the policy loss sends gradients into the critic and both learn the wrong thing.
  • Forgetting to normalize advantages, or normalizing over padding. Raw advantages change scale as the reward model changes; most PPO code normalizes them per batch, over real tokens only.
  • Using \(\lambda \lt 1\) with a cold critic on very long answers. The reward barely reaches early tokens (\(0.95^{999} \approx 6\times10^{-23}\)); pre-train the critic or use \(\lambda = 1\) for its targets.
  • Thinking \(\gamma\) and \(\lambda\) are the same knob. Both shrink the weights by \(\gamma\lambda\) in (65.8), but \(\gamma\) also changes the objective itself (what return we want to maximize), while \(\lambda\) only changes how we estimate the advantage.

Interview questions

Define \(V^\pi(s)\), \(Q^\pi(s,a)\) and \(A^\pi(s,a)\). How are they related?basic
\(V^\pi(s)\) is the expected return from state \(s\) following \(\pi\). \(Q^\pi(s,a)\) is the expected return if you first take action \(a\) in \(s\), then follow \(\pi\). The advantage is \(A = Q - V\): how much better action \(a\) is than the policy’s average action. \(V(s) = \E_{a\sim\pi}[Q(s,a)]\), so the average advantage under \(\pi\) is zero, and \(Q(s,a) = \E[r + \gamma V(s')]\) (Bellman).
Why use the advantage instead of the raw return in the policy gradient?basic
Subtracting a state-dependent baseline does not change the expected gradient, because \(\E_a[\nabla \log \pi(a\mid s)] = 0\). But it greatly lowers the variance: actions are pushed up only if they did better than expected and down if worse. Using \(V(s)\) as the baseline gives the advantage. With raw returns, all actions are pushed up whenever rewards are positive, and only the relative size carries signal, which needs many more samples.
What is the TD error, and why is it an advantage estimate?core
\(\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)\): the one-step surprise. If \(V = V^\pi\), then \(\E[\delta_t \mid s_t, a_t] = Q^\pi(s_t,a_t) - V^\pi(s_t) = A^\pi(s_t, a_t)\) by the Bellman equation. So it is an unbiased advantage estimate with a perfect critic, and biased with an imperfect one. It has low variance because it contains only one random reward and one transition.
Explain the bias–variance trade-off between TD and Monte Carlo.core
Monte Carlo uses the full sampled return: unbiased, but its variance includes the randomness of every later step. TD bootstraps from the critic after one step: low variance, but biased whenever the critic is wrong. \(n\)-step returns sit in between. GAE averages all \(n\)-step estimates with weights controlled by \(\lambda\): \(\lambda = 0\) is TD, \(\lambda = 1\) is Monte Carlo minus a baseline.
Write the GAE formula and its recursive form. What does \(\lambda\) control?core
\(\hat A_t = \sum_l (\gamma\lambda)^l \delta_{t+l}\), computed backward as \(\hat A_t = \delta_t + \gamma\lambda\hat A_{t+1}\), with \(\hat A = 0\) after the end. \(\lambda\) controls how many real future rewards enter before trusting the critic: low \(\lambda\) = more bias, less variance; high \(\lambda\) = less bias, more variance. The effective horizon is about \(1/(1-\gamma\lambda)\), for example 16.8 steps for \(\gamma = 0.99\), \(\lambda = 0.95\).
What is the difference between \(\gamma\) and \(\lambda\)?core
\(\gamma\) is part of the problem definition: it defines the discounted return we optimize, and with \(\gamma \lt 1\) far-away rewards truly matter less. \(\lambda\) is only part of the estimator: it changes how we estimate the advantage of the \(\gamma\)-discounted objective, trading bias for variance. With a perfect critic, any \(\lambda\) gives an unbiased estimate; \(\gamma\) still changes what “good” means. In RLHF, \(\gamma = 1\) is common, so only \(\lambda\) shortens the horizon.
How is GAE computed for an LLM response in RLHF? Where does the reward go?core
Each response token is a step. The reward model’s score is placed on the last (EOS) token; every token gets \(-\beta(\log\pi - \log\pi_{\text{ref}})\) as a KL penalty. The value model, a copy of an LLM with a scalar head, gives \(V\) at every position, and \(V = 0\) after the last token. Then compute \(\delta_t\) and run the backward recursion per response, masking padding. Typical choices: \(\gamma = 1\), \(\lambda = 0.95\). The critic is trained toward \(\hat A_t + V_t\).
Why might PPO with GAE fail on long chain-of-thought, and how can you fix it?advanced
With \(\lambda = 0.95\), the final reward reaches a token \(l\) steps earlier with weight \(0.95^l\), essentially zero after a few hundred tokens. Early tokens then learn only from the critic, and a freshly initialized critic (for example from a reward model trained on full answers) is biased on partial answers. VC-PPO (Yuan et al., 2025) pre-trains the value model on rollouts before PPO and uses \(\lambda = 1\) for the critic’s targets while keeping \(\lambda = 0.95\) for the actor. Alternatively, drop the critic (GRPO) and use group-mean baselines.
What does GRPO give up by removing the critic?advanced
It saves a full-size value model (weights, gradients, optimizer state, and its forward and backward passes) and avoids critic-training problems. It loses per-token credit assignment: all tokens of one answer share the same advantage, (reward − group mean) / group std. It also needs several samples per prompt to form the baseline, which adds rollout cost. For tasks with one verifiable outcome per answer, this trade has worked well in practice (DeepSeekMath, DeepSeek-R1).
Why is the critic’s target \(\hat A_t + V_{\text{old}}(s_t)\), and what is a \(\lambda\)-return?advanced
\(\hat A_t + V(s_t) = \sum_l (\gamma\lambda)^l\delta_{t+l} + V(s_t)\) is the \(\lambda\)-return: an exponentially weighted average of the \(n\)-step returns. At \(\lambda = 1\) it is the Monte Carlo return; at \(\lambda = 0\) it is the TD(0) target \(r_t + \gamma V(s_{t+1})\). It is computed once with the critic before the update (\(V_{\text{old}}\)) and kept fixed, so the regression target does not move while the critic trains on it. PPO implementations often also clip the value update relative to \(V_{\text{old}}\).
Does the value head need to be as large as the policy?advanced
Not necessarily. The value function must understand the partial response to predict its eventual reward, so in practice it is often an LLM of similar size (often initialized from the reward model). Smaller critics save memory but may be biased, which hurts low-\(\lambda\) estimates more. Sharing the policy’s body with a value head is cheaper, but the value loss can interfere with the policy’s representations; most large RLHF systems use separate models. [Likely]

Exercises

  1. Three tokens, \(\gamma = 1\), rewards \((0, 0, 1)\), critic values \((0.5, 0.6, 0.9)\), \(V = 0\) after the end. Compute \(\delta_t\) and GAE with \(\lambda = 0.5\).
  2. For the same data, compute the advantages with \(\lambda = 0\) and \(\lambda = 1\). Which tokens change most?
  3. Show that if the critic is \(V(s) = c\) (the same constant for every state, but 0 at the end) and \(\gamma = \lambda = 1\), GAE gives \(G_t - c\).
  4. What is the effective horizon \(1/(1 - \gamma\lambda)\) for \(\gamma = 1\), \(\lambda = 0.9\)? How much direct weight does the final reward get at a token 50 steps before the end?
  5. In Figure 2, find a pair of step sizes for which Monte Carlo beats TD after 100 episodes, if any. What does this tell you about comparing algorithms?
  6. A 7B policy and a 7B critic are trained with AdamW in mixed precision. How much training-state memory (weights, gradients, optimizer) do the two need together, before activations?
  7. In Figure 5, set the critic error \(\kappa = 0\). Why does \(\lambda = 0\) now have no bias, and why is its variance still above 0 when \(\sigma \gt 0\)?
Answerscheck
1. \(\delta = (0 + 0.6 - 0.5,\ 0 + 0.9 - 0.6,\ 1 + 0 - 0.9) = (0.1, 0.3, 0.1)\). Backward: \(\hat A_2 = 0.1\), \(\hat A_1 = 0.3 + 0.5\times0.1 = 0.35\), \(\hat A_0 = 0.1 + 0.5\times0.35 = 0.275\). 2. \(\lambda = 0\): \((0.1, 0.3, 0.1)\). \(\lambda = 1\): \(G_t - V = (0.5, 0.4, 0.1)\). The first token changes most (0.1 to 0.5), because it is farthest from the reward. 3. \(\delta_t = r_t + c - c = r_t\) for \(t \lt T-1\) and \(\delta_{T-1} = r_{T-1} - c\); the sum from \(t\) is \(\sum_{l} r_{t+l} - c = G_t - c\). 4. Horizon \(1/(1-0.9) = 10\) steps; weight \(0.9^{50} = 0.0052\). 5. Yes. TD with \(\alpha = 0.01\) ends at RMS 0.128, while MC with \(\alpha = 0.03\) ends at 0.095. TD with \(\alpha = 0.3\) (0.106) also loses to MC with \(\alpha = 0.02\) (0.083). Comparisons depend on step sizes and on how long you train, so compare each method at its own best setting. 6. \(14\times10^9\) parameters \(\times\) 16 bytes \(= 224\) GB. 7. With \(\kappa = 0\) the critic is exact, so \(\E[\delta_0 \mid s_0, a_0] = Q^\pi(s_0, a_0) - V^\pi(s_0)\) by (65.3): the one-step estimate is unbiased. Its only randomness is the noise in \(r_0\) (the next state is fixed once \(a_0\) is chosen), so its variance is \(\sigma^2\), much less than the variance of the full return used at \(\lambda = 1\). That is why \(\lambda = 0\) wins when the critic is perfect.

Key takeaways

  • \(V^\pi(s)\) is the expected return from a state, \(Q^\pi(s,a)\) from a state and an action (65.2); they are linked by the Bellman equations (65.3).
  • The advantage \(A = Q - V\) (65.4) has zero mean under the policy; using it in the policy gradient (65.5) keeps the gradient unbiased and lowers its variance.
  • The TD error \(\delta_t\) (65.6) is a low-variance, possibly biased, one-step advantage; the Monte Carlo return is unbiased but noisy (65.7).
  • GAE (65.8) sums TD errors with weights \((\gamma\lambda)^l\), computed backward in one pass (65.9); \(\lambda = 0\) is TD and \(\lambda = 1\) is Monte Carlo (65.10).
  • In RLHF, the reward model score sits on the last token and KL penalties on every token (65.12); typical \(\gamma = 1\), \(\lambda = 0.95\). For very long answers, the reward decays before reaching early tokens.
  • Actor–critic trains a policy and a critic together (65.13, 65.14). The critic costs a full model of memory, which is why GRPO replaces it with group baselines.

Key formulas

\[ A = Q - V, \quad \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t), \quad \hat A_t = \delta_t + \gamma\lambda\, \hat A_{t+1}, \quad \hat R_t = \hat A_t + V(s_t) \]

References for this session

  1. Schulman et al. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation
  2. Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., chapter 6: Temporal-Difference Learning
  3. OpenAI (Achiam) (2018). Spinning Up in Deep RL
  4. Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., chapter 12
  5. Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., chapter 13: Policy Gradient Methods
  6. Schulman et al. (2017). Proximal Policy Optimization Algorithms
  7. Mnih et al. (2016). Asynchronous Methods for Deep Reinforcement Learning
  8. Huang et al. (2024). The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
  9. Yuan et al. (2025). What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
  10. Ouyang et al. (2022). Training language models to follow instructions with human feedback
  11. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
  12. Lambert (2025). Reinforcement Learning from Human Feedback (book)