Value functions, advantage and GAE
REINFORCE pushes every token of an answer up or down by the same noisy reward. This session adds a critic: a value function that predicts the expected reward from each state. With it we get the advantage of each token, the TD error, and Generalized Advantage Estimation (GAE), which uses two numbers, γ and λ, to trade bias for variance.
What you will learn
- The return, the state value V(s) and the action value Q(s, a), and how the Bellman equations link them
- The advantage A = Q − V, and why it lowers the variance of the policy gradient without adding bias
- How a critic is learned with Monte Carlo targets or with temporal-difference (TD) targets, and what the TD error means
- The bias–variance trade-off between one-step TD and full Monte Carlo, with numbers
- Generalized Advantage Estimation: the formula, the backward recursion, and the roles of γ and λ
- How GAE is computed per token in RLHF (reward at the EOS token, KL penalty on every token), and the decay problem for long answers
- Actor–critic training, the cost of a separate critic for LLMs, and why GRPO removes it
Why this matters
REINFORCE works, but it is noisy. One sampled answer gets one reward, and every token in it is pushed up or down by that same number. To learn from fewer samples, we need a way to say “this token was better than usual” or “worse than usual”. That is what value functions and the advantage give us.
In session 64 the gradient was \(\E[\sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, G_t]\), where \(G_t\) is the reward that followed. Subtracting a baseline made it less noisy. This session makes the baseline smart: a second network, the critic, learns to predict the expected reward from every state. With it we can build the advantage of each action, and with Generalized Advantage Estimation (GAE) we can choose how much to trust the critic.
This is exactly what PPO does in classic RLHF. InstructGPT-style training keeps four models in memory: the policy, a frozen reference model, a reward model, and a value model (the critic). [Certain] The value model and GAE are the parts in this session. GRPO, used for DeepSeek-R1-style reasoning training, removes the value model and uses the average reward of a group of answers instead. [Certain] You can only understand why that trade is worth it once you know what the critic does.
Value functions: how good is a state?
Recall the words from session 63. At step \(t\) the agent is in a state \(s_t\), takes an action \(a_t\) from its policy \(\pi(a \mid s)\), gets a reward \(r_t\), and moves to \(s_{t+1}\). For an LLM, the state is the prompt plus the tokens written so far, the action is the next token, and the policy is the model itself.
The return from step \(t\) is the sum of all future rewards, each later one multiplied by a discount \(\gamma \in [0, 1]\):
The return is random: it depends on which tokens are sampled later. A value function is its average. There are two kinds:
- Read the symbols
- \(V^\pi(s)\)
- state value: the expected return if you start in \(s\) and follow \(\pi\) from there
- \(Q^\pi(s, a)\)
- action value: the expected return if you start in \(s\), take action \(a\) first, and then follow \(\pi\)
- \(\gamma\)
- discount factor; \(\gamma = 0.99\) means a reward 100 steps away counts \(0.99^{100} = 0.37\) times as much as one now. For LLM responses, \(\gamma = 1\) is common [Certain]
- \(T\)
- the step at which the episode ends (for an LLM: the end-of-sequence token)
The two are connected. \(V\) is the average of \(Q\) over the actions the policy would pick, and \(Q\) is one reward plus the value of where you land. These are the Bellman equations:
The second equation is the key idea of this whole session: a value can be estimated from one real reward plus the value of the next state. We do not need to wait for the end of the episode.
For an LLM, transitions are deterministic: if the state is “prompt + tokens so far” and the action is token \(a\), the next state is just “prompt + tokens + \(a\)”. All the randomness is in the policy and in the reward. [Certain]
The advantage: better or worse than usual?
The advantage of an action is how much better it is than the average action in that state:
The second part follows from (65.3): the average of \(Q\) is \(V\), so the average advantage is zero. Positive advantage means “this token was better than what the model usually writes here”; negative means worse. Using it in the policy gradient gives:
This is the same expected gradient as REINFORCE: subtracting \(V(s_t)\) changes nothing on average, because \(\E_a[\nabla_\theta \log \pi_\theta(a \mid s)] = 0\) for any fixed state. [Certain] But the variance is much lower. Example: if every answer to a prompt gets a reward between 0.8 and 0.9, REINFORCE pushes all of them up. With the advantage, the 0.9 answers go up and the 0.8 answers go down, which is the information we actually want.
The problem: we do not know \(Q^\pi\) or \(V^\pi\). We must estimate them. The rest of this session is about how.
Learning the critic: Monte Carlo or temporal difference
The critic is a network \(V_\phi(s)\) with weights \(\phi\) that tries to predict \(V^\pi(s)\). We train it by regression: pick a target for each visited state, then minimize the squared error. There are two classic choices of target.
- Monte Carlo (MC): wait until the episode ends and use the real return \(G_t\). The target is correct on average (unbiased), but it is noisy, because it adds up the randomness of every later step.
- Temporal difference (TD): use one real reward plus the critic’s own guess for the next state, \(r_t + \gamma V_\phi(s_{t+1})\). This is called bootstrapping: learning a guess from a guess. The target has only one step of randomness (low variance), but if \(V_\phi\) is wrong, the target is wrong too (biased).
The difference between the TD target and the current guess is the TD error:
- Read the symbols
- \(\delta_t\)
- the “surprise” at step \(t\): how much better things turned out after one step than the critic expected
- \(V(s_{t+1})\)
- the critic’s value of the next state; it is 0 if \(s_{t+1}\) is the end of the episode
The TD error is also a one-step advantage estimate. If the critic were perfect (\(V = V^\pi\)), then \(\E[\delta_t \mid s_t, a_t] = Q^\pi(s_t, a_t) - V^\pi(s_t) = A^\pi(s_t, a_t)\), by (65.3). The GAE paper states this exactly: \(\delta_t\) is an unbiased advantage estimate when \(V = V^\pi\), and biased otherwise. [Certain]
- input: policy \(\pi\), step size \(\alpha\), initial guesses \(V(s)\) for all states, \(V(\text{end}) = 0\)
- for each episode do
- for each step \(t\): take \(a_t \sim \pi(\cdot \mid s_t)\), observe \(r_t, s_{t+1}\)
- TD(0): \(V(s_t) \leftarrow V(s_t) + \alpha\, \delta_t\) // formula (65.6), update at once
- Monte Carlo: after the episode ends, for every visited \(s_t\): \(V(s_t) \leftarrow V(s_t) + \alpha\,(G_t - V(s_t))\) // formula (65.1)
Figure 1 runs both lines of Algorithm 1 side by side, one step at a time, on a corridor. A robot starts in cell 3 of seven cells and moves right with probability \(\pi(\text{right})\), otherwise left. Reaching the right end gives \(+1\), the left end \(-1\), and every other step gives 0. Both critics start at \(V = 0\) and see exactly the same episodes.
- \(V_{\text{TD}}(s) \leftarrow 0\), \(V_{\text{MC}}(s) \leftarrow 0\) for every state; \(V(\text{end}) = 0\)
- for each episode: \(s_0 \leftarrow 3\)
- take \(a_t \sim \pi\); observe \(r_t\) and \(s_{t+1}\)
- TD: \(\delta_t \leftarrow r_t + \gamma V_{\text{TD}}(s_{t+1}) - V_{\text{TD}}(s_t)\) // (65.6)
- TD: \(V_{\text{TD}}(s_t) \leftarrow V_{\text{TD}}(s_t) + \alpha\, \delta_t\) // at once
- repeat lines 3–5 until \(s_{t+1}\) is an end cell
- MC: for \(t = T-1\) down to 0: \(G_t \leftarrow r_t + \gamma\, G_{t+1}\) // (65.1)
- MC: \(V_{\text{MC}}(s_t) \leftarrow V_{\text{MC}}(s_t) + \alpha\,(G_t - V_{\text{MC}}(s_t))\) // only after the episode
How to read Figure 1
Step through the first episode at Slow. TD updates right away, but in the first episode its \(\delta_t\) is 0 at every step until the robot reaches an end: only the last cell learns anything. That information then spreads back one cell per visit. Monte Carlo waits, then pushes every visited cell toward the full return, which is \(+1\) or \(-1\). Now press Play at Fast. The orange MC bars jump up and down after each episode: every target is a whole random outcome. The blue TD bars move in small steps. With \(\alpha = 0.05\) and seeds 1–50, TD has the lower error after 100 episodes in all 50 runs (mean RMS error 0.082 against 0.263). Raise \(\alpha\) to 0.3: both get noisier, MC much more. Lower \(\gamma\): the true values shrink toward 0, because the end is several steps away.
Which one learns faster in general? Figure 2 averages 100 runs of the classic test from Sutton and Barto (Example 6.2): a random walk over five states A to E. Every episode starts in C and moves left or right with probability ½. Leaving on the right gives reward 1; leaving on the left gives 0. The true values are \(1/6, 2/6, \dots, 5/6\). [Certain] All guesses start at 0.5.
How to read Figure 2
Lower is better. With the default step sizes, TD reaches a lower error than Monte Carlo after 100 episodes. Raise the MC step size: it learns faster at first but then stays noisy, because each MC target is the full random outcome (0 or 1). Raise the TD step size to 0.3: it also gets noisy. A step size that is too small makes both slow. This is the bias–variance trade-off you will see again in GAE: TD targets are less noisy, MC targets are not biased by bad guesses.
On this task, TD is faster than MC for the step sizes Sutton and Barto test. [Certain] This is not a theorem for all problems, but it is common in practice. [Likely] Deep RL critics use TD-style targets for this reason.
Bias and variance: from one step to many
There is a whole range between TD and MC. We can use \(k\) real rewards and then bootstrap with the critic. Write it with TD errors, and the middle terms cancel (each \(+\gamma V(s_{t+1})\) cancels the next \(-V(s_{t+1})\)):
- \(k = 1\): \(\hat A^{(1)}_t = \delta_t\). One random reward, then trust the critic. Low variance, high bias if the critic is wrong.
- \(k\) up to the end of the episode: \(\hat A^{(\infty)}_t = G_t - V(s_t)\). The critic is only a baseline. No bias, high variance.
Here “bias” means: on average, the estimate is not equal to the true advantage (up to the baseline). “Variance” means: from one sample to the next, the estimate jumps around a lot. A policy gradient with a biased advantage pushes in a slightly wrong direction. A policy gradient with a high-variance advantage pushes in the right direction on average but needs many more samples. A3C (Mnih et al., 2016) used a fixed \(k\) for this reason. [Certain] GAE removes the need to pick one \(k\).
Generalized Advantage Estimation
Instead of choosing one \(k\), GAE takes an exponentially weighted average of all of them, with weight \((1 - \lambda)\lambda^{k-1}\) for \(\hat A^{(k)}\). After the algebra, this becomes a simple sum of TD errors (Schulman et al., 2015, equation 16): [Certain]
- Read the symbols
- \(\lambda \in [0, 1]\)
- how far into the future we look at real rewards before trusting the critic; it plays the same role as in TD(\(\lambda\))
- \((\gamma\lambda)^l\)
- the weight of the surprise \(l\) steps later; it shrinks geometrically
- \(T - t - 1\)
- the number of steps left in the episode (or in the collected segment)
You never compute the sum directly. Going from the last step backward, each advantage is the TD error plus a discounted copy of the next advantage:
This is one backward pass over the sequence, \(O(T)\) work. The two ends of the range are the estimators from the previous section:
The GAE paper says: \(\lambda = 0\) is biased unless \(V\) is exact but has much lower variance; \(\lambda = 1\) has no bias whatever \(V\) is, but high variance. [Certain] Values in between trade the two. A useful number is the effective horizon \(1/(1 - \gamma\lambda)\): with \(\gamma = 0.99\) and \(\lambda = 0.95\), \(\gamma\lambda = 0.9405\) and the horizon is about 16.8 steps.
The paper’s experiments found the best results at intermediate values: \(\lambda\) between 0.92 and 0.99 on cart-pole and between 0.96 and 0.99 on 3D bipedal walking. [Certain] PPO then used \(\gamma = 0.99\), \(\lambda = 0.95\) for MuJoCo, Roboschool and Atari. [Certain] Figure 3 lets you see what \(\lambda\) does to a sequence.
How to read Figure 3
At \(\lambda = 0\), each token’s advantage is just its own TD error, which here is mostly the critic’s own wiggles: the final reward reaches only the last token. At \(\lambda = 1\), every token gets “final reward minus my value”, so the reward reaches all tokens, but each token’s advantage no longer depends on what happened right after it. The readout shows how much of the final reward reaches the first token directly: \((\gamma\lambda)^{T-1}\). Set \(T = 256\) with \(\lambda = 0.95\): almost nothing reaches the start. Long chain-of-thought answers have thousands of tokens, which is why some reasoning recipes set \(\lambda = 1\) for the critic.
Watching bias and variance trade off
We can make the trade-off exact with a small model. Take an episode of \(T\) steps. Every reward has random noise with standard deviation \(\sigma\), independent across steps. The critic is off by the same amount \(b\) in every state after the first (\(V(s_l) = V^\pi(s_l) + b\) for \(l \ge 1\), and 0 at the end). With \(\gamma = 1\), the error of \(\hat A_0\) compared with the true “\(Q\) minus baseline” has
Formula (65.11) follows from (65.8): the critic errors telescope away except for the part that \(\lambda\) cuts off, and each reward’s noise enters with weight \(\lambda^l\). We checked both formulas with a 20,000-sample simulation. Figure 4 plots them.
How to read Figure 4
With the defaults (\(T = 20\), \(b = 0.5\), \(\sigma = 0.1\)), the best \(\lambda\) is 0.98: better than both TD (\(\lambda = 0\)) and MC (\(\lambda = 1\)). Set \(b = 0\) (a perfect critic): the bias vanishes and \(\lambda = 0\) wins. Set \(\sigma = 0\) (no noise): variance vanishes and \(\lambda = 1\) wins. Real problems are in between, which is why tuned values like 0.95 are common. This model is a simplification; in a real task the critic errors differ per state and can partly cancel.
Prompt: “What is the capital of France?” The policy writes five tokens: “The”, “ capital”, “ is”, “ Paris”, “.”. As in RLHF, each token gets a small KL penalty \(-\beta\,\text{kl}_t\) with \(\beta = 0.1\) and per-token KL estimates \((0.2, 0.1, 0.0, 0.5, 0.1)\), and the last token also gets the reward-model score 1.0. So the rewards are \(r = (-0.02, -0.01, 0, -0.05, 0.99)\).
The critic says \(V = (0.60, 0.70, 0.75, 0.85, 0.95)\), and \(V = 0\) after the last token. Use \(\gamma = 1\), \(\lambda = 0.95\).
TD errors (65.6): \(\delta_0 = -0.02 + 0.70 - 0.60 = 0.08\), \(\delta_1 = -0.01 + 0.75 - 0.70 = 0.04\), \(\delta_2 = 0 + 0.85 - 0.75 = 0.10\), \(\delta_3 = -0.05 + 0.95 - 0.85 = 0.05\), \(\delta_4 = 0.99 + 0 - 0.95 = 0.04\).
Backward recursion (65.9): \(\hat A_4 = 0.04\); \(\hat A_3 = 0.05 + 0.95 \times 0.04 = 0.088\); \(\hat A_2 = 0.10 + 0.95 \times 0.088 = 0.1836\); \(\hat A_1 = 0.04 + 0.95 \times 0.1836 = 0.2144\); \(\hat A_0 = 0.08 + 0.95 \times 0.2144 = 0.2837\).
Compare the ends: with \(\lambda = 0\) the advantages are the TD errors \((0.08, 0.04, 0.10, 0.05, 0.04)\); with \(\lambda = 1\) they are \(G_t - V(s_t) = (0.31, 0.23, 0.19, 0.09, 0.04)\). GAE with 0.95 lies between them. The critic targets are \(\hat A_t + V(s_t) = (0.8837, 0.9144, 0.9336, 0.938, 0.99)\). Figure 6 replays these steps.
Formula (65.11) is a model. Figure 5 measures the same trade-off on real episodes. It uses the corridor of Figure 1 with a policy that goes right 70% of the time. Each step’s reward also has a small random part with standard deviation \(\sigma\), the way the per-token KL term of the next section adds a small reward at every token. The critic is deliberately wrong: it knows only part of the true value, \(\hat V(s) = (1 - \kappa)\, V^\pi(s)\), like a critic early in training. For every episode the figure computes \(\hat A_0\) for 21 values of \(\lambda\) and compares it with the true “\(Q\) minus baseline” \(Q^\pi(s_0, a_0) - \hat V(s_0)\). Averaged over many episodes, the errors give the bias and the variance of each \(\lambda\).
- roll out \(\tau = (s_0, a_0, r_0, \dots, s_T)\) with \(\pi\); \(\hat V\) is fixed
- for \(t = 0..T-1\): \(\delta_t \leftarrow r_t + \gamma \hat V(s_{t+1}) - \hat V(s_t)\) // (65.6)
- \(\hat A_T \leftarrow 0\)
- for \(t = T-1\) down to 0: \(\hat A_t \leftarrow \delta_t + \gamma\lambda\, \hat A_{t+1}\) // (65.9)
- critic targets \(\hat R_t \leftarrow \hat A_t + \hat V(s_t)\) // (65.13)
- for all 21 values of \(\lambda\): error \(\leftarrow \hat A_0 - \big(Q^\pi(s_0, a_0) - \hat V(s_0)\big)\); update its mean and mean square
How to read Figure 5
Step through one episode. Line 2 fills the \(\delta_t\) column from left to right; line 4 fills the \(\hat A_t\) column from right to left, each value reusing the one below it. In the bar chart, the grey dots (\(\lambda = 0\)) are small because the critic expects little, while the blue ticks (\(\lambda = 1\)) carry the full final \(\pm 1\) back to every step. Then press Turbo. With seed 1, after 2,000 episodes \(\lambda = 0\) has MSE 0.274, almost all of it \(\text{bias}^2\) (0.250): it trusts a critic that knows a quarter of the truth. \(\lambda = 1\) has no bias (its measured \(\text{bias}^2\) is near 0) but variance 0.187. The lowest MSE, 0.134, is at \(\lambda = 0.90\), close to the values used in practice. Now set \(\kappa = 0\) (a perfect critic): bias disappears and \(\lambda = 0\) wins. Set \(\kappa = 1\) (the critic always says 0): the best \(\lambda\) moves up to about 0.95. The best \(\lambda\) depends on how good the critic is, which is why it is a tuned hyperparameter.
GAE for LLM responses
In RLHF with PPO, one response is one episode, and each generated token is one step. The reward is mostly sparse: the reward model scores the whole response, and that score is placed on the last token (the end-of-sequence token). Every token also gets a KL penalty that keeps the policy close to the reference model:
- Read the symbols
- \(\beta\)
- strength of the KL penalty
- \(\pi_{\text{ref}}\)
- the frozen starting model (usually the SFT model)
- \(\mathbb{1}[t = T-1]\)
- 1 for the last token of the response, 0 otherwise
- \(r_{\text{RM}}(x, y)\)
- the reward model’s score for prompt \(x\) and full response \(y\)
This is the setup of the open reproduction of OpenAI’s summarization RLHF by Huang et al. (2024): the score at the EOS token, per-token KL penalties elsewhere, \(\gamma = 1\), \(\lambda = 0.95\), and a value network initialized from the reward model. [Certain] InstructGPT also initialized its value function from the reward model. [Likely] The critic is usually an LLM of similar size with a scalar “value head” (a linear layer from the hidden state to one number) instead of the vocabulary head; it outputs one value per token position.
How to read Figure 6
The rows fill in from top to bottom. The advantage row fills in from right to left: the last token’s advantage is known first, and each earlier one reuses the next. Notice “ is” gets a larger advantage than “ capital”: after “ is” the critic’s value jumped from 0.75 to 0.85, so the critic thinks “ is” moved the answer toward a good ending.
The decay problem for long answers
With \(\gamma = 1\) and \(\lambda = 0.95\), the final reward reaches a token \(l\) steps before the end with direct weight \(0.95^{l}\): 0.20 after 31 tokens, 0.006 after 99, and about \(6 \times 10^{-23}\) after 999. Early tokens of a long answer then learn almost only from the critic’s own TD errors. If the critic is poor at the start of training, those advantages are mostly noise or bias. Yuan et al. (2025) call this “reward signal decay”. Their fix (VC-PPO) is to pre-train the value model before PPO and to decouple GAE: \(\lambda_{\text{critic}} = 1.0\) for the critic’s targets, while the policy keeps \(\lambda_{\text{actor}} = 0.95\). [Certain]
Actor–critic: a policy plus a learned critic
An actor–critic method trains two things at once: the actor (the policy \(\pi_\theta\)) and the critic (the value function \(V_\phi\)). The critic learns to predict returns; the actor uses the critic’s advantages to decide which actions to make more likely. The critic is trained by regression on the GAE-based targets (also called \(\lambda\)-returns):
The full loss is the policy term, the value term, and often an entropy bonus \(\mathcal{H}\) that keeps the policy from becoming too certain too early:
- Read the symbols
- \(\hat A_t\)
- the GAE advantage, treated as a constant (no gradient flows into it)
- \(\hat R_t\)
- the critic’s target, computed once with the critic from before the update
- \(c_v, c_e\)
- weights of the value loss and the entropy bonus; PPO’s Atari setting used \(c_v = 1\), \(c_e = 0.01\) [Certain]
PPO keeps this structure but replaces the policy term with a clipped ratio objective, so it can take several gradient steps on the same batch (session 66). Everything about \(\hat A_t\) in this session carries over unchanged.
- input: policy \(\pi_\theta\), critic \(V_\phi\), reference \(\pi_{\text{ref}}\), reward model, prompts, \(\gamma, \lambda, \beta\)
- sample responses \(y \sim \pi_\theta(\cdot \mid x)\) for a batch of prompts // the rollout
- score each response with the reward model; compute per-token KL against \(\pi_{\text{ref}}\)
- build per-token rewards \(r_t\) // formula (65.12)
- run the critic once: \(V_t = V_\phi(s_t)\) for every response token; set \(V = 0\) after the last token
- for \(t = T-1\) down to 0: \(\delta_t \leftarrow r_t + \gamma V_{t+1} - V_t\); \(\hat A_t \leftarrow \delta_t + \gamma\lambda \hat A_{t+1}\) // (65.6), (65.9)
- \(\hat R_t \leftarrow \hat A_t + V_t\) // critic targets, (65.13)
- normalize \(\hat A\) over the batch (subtract mean, divide by std), using only real tokens
- update \(\theta\) and \(\phi\) by minimizing (65.14) // PPO: clipped policy loss, several epochs
Figure 8 runs Algorithm 2 on the token toy of session 63 and session 64: prompt “2+3=”, vocabulary 5, 6, 7, “.”, ⏎, at most 4 tokens, reward 1 for an answer whose digits are exactly “5” and that ends with ⏎. The actor is a table of logits indexed by the previous token (a bigram “LLM”, 25 weights). The critic is a table with one value per previous token (5 weights), so, like a real value model, it cannot tell every state apart. The reference policy is the starting policy (uniform), and the per-token reward is (65.12). We use \(\gamma = 1\), plain gradient steps and no advantage normalization, so every number can be followed by hand.
- sample \(N\) answers \(y \sim \pi_\theta(\cdot \mid x)\), keeping \(\log \pi_\theta(y_t \mid s_t)\)
- score each answer with the checker: \(R \in \{0, 1\}\)
- per-token rewards \(r_t \leftarrow -\beta(\log \pi_\theta - \log \pi_{\text{ref}})\), plus \(R\) at the last token // (65.12)
- critic: \(V_t \leftarrow V_\phi(s_t)\); \(V = 0\) after the last token
- for \(t = T-1\) down to 0: \(\delta_t \leftarrow r_t + \gamma V_{t+1} - V_t\); \(\hat A_t \leftarrow \delta_t + \gamma\lambda \hat A_{t+1}\) // (65.6), (65.9)
- \(\hat R_t \leftarrow \hat A_t + V_t\) // (65.13)
- one gradient step on (65.14) for \(\theta\) (step \(\eta_\pi\)) and for \(\phi\) (step \(\eta_v\)), averaged over all tokens
How to read Figure 8
Step through the first batch slowly. Line 3 gives every token a reward of exactly 0 in the first batch, because the policy still equals the reference, so the log-ratio is 0; only the last token of a correct answer gets 1. The critic starts at 0, so in the first batch \(\delta_t\) is 0 everywhere except at the last token, and only \(\lambda\) carries the final reward back to the first token. Compare this with Figure 8 of session 64, where every token got the same number: here the tokens of one answer get different advantages once the critic has learned something. Later, a correct answer’s first token “5” gets a smaller advantage than at the start, because the critic already expects a good answer from the prompt. Press Play. The critic learns that “after 5” is a good state and “after 6” is worthless. Now set \(\lambda = 0\): the first token gets credit only through the critic, which knows nothing at first, so learning is slower (over seeds 1–6, \(J\) after 50 iterations averages 0.835 with \(\lambda = 0\) and 0.933 with \(\lambda = 0.95\)). Raise \(\beta\): the KL penalty pulls the policy back toward the uniform reference.
Shared or separate critic?
In small RL problems the actor and critic often share one network body with two heads. For LLMs, the critic is usually a separate model, often initialized from the reward model or the SFT model. [Likely] A separate critic of the same size doubles the trainable weights and the optimizer memory: with AdamW at 16 bytes per parameter, a 7B critic alone needs about 112 GB of training state (session 38). This cost is one of the main reasons GRPO removes the critic and uses the mean reward of several answers to the same prompt as the baseline (session 70). The price: GRPO gives every token of an answer the same advantage, so it has no per-token credit assignment from a critic. [Certain]
Code: GAE and a tiny actor–critic in PyTorch
The function below computes GAE for a batch of right-padded responses, with a mask so that padding never leaks into the advantages. It reproduces the worked example and checks the two special cases. Then a tiny actor–critic learns a 4-token “sentence” from a reward given only at the end. It runs on a CPU in a few seconds and prints the same numbers every time.
import torch torch.manual_seed(0) # 1) GAE for a batch of responses, right-padded. mask = 1 on response tokens. def gae(rewards, values, mask, gamma=1.0, lam=0.95): B, T = rewards.shape adv = torch.zeros_like(rewards) last = torch.zeros(B) for t in reversed(range(T)): # formula (65.9), right to left if t + 1 < T: next_v = values[:, t + 1] * mask[:, t + 1] # V = 0 after the last token else: next_v = torch.zeros(B) delta = rewards[:, t] + gamma * next_v - values[:, t] # formula (65.6) last = (delta + gamma * lam * last) * mask[:, t] adv[:, t] = last returns = (adv + values) * mask # targets for the critic, (65.13) return adv, returns # the worked example: 5 tokens, KL penalty on every token, reward-model score 1.0 at the end kl = torch.tensor([[0.2, 0.1, 0.0, 0.5, 0.1]]) r = -0.1 * kl; r[0, -1] += 1.0 V = torch.tensor([[0.6, 0.7, 0.75, 0.85, 0.95]]) m = torch.ones(1, 5) A, R = gae(r, V, m) print(A.round(decimals=4), R.round(decimals=4)) # [[0.2837, 0.2144, 0.1836, 0.0880, 0.0400]] [[0.8837, 0.9144, 0.9336, 0.9380, 0.9900]] # special cases: lambda = 0 gives the TD errors, lambda = 1 gives return minus V print(gae(r, V, m, lam=0.0)[0].round(decimals=4), gae(r, V, m, lam=1.0)[0].round(decimals=4)) # [[0.08, 0.04, 0.10, 0.05, 0.04]] [[0.31, 0.23, 0.19, 0.09, 0.04]] # padding check: the second, shorter response must not see its padding r2 = torch.zeros(2, 5); r2[0, 4] = 1.0; r2[1, 2] = 1.0 m2 = torch.tensor([[1., 1, 1, 1, 1], [1, 1, 1, 0, 0]]) A2, _ = gae(r2, torch.full((2, 5), 0.5), m2) print(A2.round(decimals=4)) # [[0.4073, 0.4287, 0.4512, 0.4750, 0.5000], [0.4512, 0.4750, 0.5000, 0, 0]] # 2) actor-critic on a toy "sentence" task: 4 steps, 3 tokens, the reward comes only at the end T, K, N = 4, 3, 64 target = torch.tensor([2, 0, 1, 2]) logits = torch.zeros(T, K, requires_grad=True) # the actor: one softmax per position values = torch.zeros(T, requires_grad=True) # the critic: one number per position opt = torch.optim.Adam([logits, values], lr=0.05) for it in range(300): dist = torch.distributions.Categorical(logits=logits.expand(N, T, K)) a = dist.sample() # [N, T]: N sampled "sentences" rew = torch.zeros(N, T) rew[:, -1] = (a == target).float().mean(1) # sparse reward at the last step v = values.expand(N, T) adv, ret = gae(rew, v.detach(), torch.ones(N, T), gamma=1.0, lam=0.95) adv = (adv - adv.mean()) / (adv.std() + 1e-8) # advantage normalization pg_loss = -(dist.log_prob(a) * adv).mean() # actor, formula (65.14) v_loss = 0.5 * ((v - ret) ** 2).mean() # critic, formula (65.13) opt.zero_grad(); (pg_loss + v_loss).backward(); opt.step() p = torch.softmax(logits, -1)[torch.arange(T), target] print(p.detach().round(decimals=3), values.detach().round(decimals=3)) # tensor([0.999, 0.999, 0.999, 0.999]) tensor([1., 1., 1., 1.]): the right token everywhere, # and the critic predicts the maximum reward 1.0 from every position
Two details matter in real code. The advantages are computed with values.detach() (or under torch.no_grad()), so the policy loss does not train the critic. And the padding mask is used both to stop the recursion at the end of each response and to exclude padding from the mean and standard deviation in normalization.
Where this is used
| Setting | What the critic and GAE do | Typical choices |
|---|---|---|
| RLHF with PPO (InstructGPT-style chat models) | Value model gives per-token \(V\); GAE turns the end-of-sequence score plus KL penalties into per-token advantages | \(\gamma = 1\), \(\lambda = 0.95\), critic initialized from the reward model [Certain for Huang et al. 2024] |
| Long chain-of-thought RL with PPO | Pre-trained value model; decoupled \(\lambda\) for actor and critic | \(\lambda_{\text{critic}} = 1.0\), \(\lambda_{\text{actor}} = 0.95\) (VC-PPO) [Certain] |
| GRPO, RLOO and similar critic-free methods | No critic: the baseline is the mean reward of a group of sampled answers | Saves a full model in memory; one advantage per answer [Certain] |
| Robot control, games (the original GAE and PPO setting) | Dense rewards, long episodes; separate or shared critic network | \(\gamma = 0.99\), \(\lambda = 0.95\) [Certain for PPO’s benchmarks] |
| VLM agents (web or GUI agents that click through screens) | Multi-turn episodes where each turn is a step; a critic helps credit which turn helped | Turn-level or token-level GAE; active research area [Speculation] |
| Video key-moment detection or video summarization trained with RL | The policy picks frames or clips one by one; the reward (for example, summary quality) comes at the end, so a critic gives each choice its own advantage | Sparse end reward, \(\gamma\) near 1; critic-free baselines are also common [Likely] |
| Video-RAG or video QA agents that decide which segment to retrieve next | Each retrieval is an action; the answer’s correctness is the reward; the value estimates how promising the current evidence is | Short episodes (a few retrievals), often \(\lambda\) near 1 [Speculation] |
Common mistakes
- Off-by-one in \(V(s_{t+1})\). The critic’s value at position \(t\) must be read from the hidden state before token \(a_t\) is chosen, and the value after the last token must be 0. Mixing these up shifts all advantages by one token.
- Letting padding into the recursion. Without the mask, the recursion runs into padding tokens and adds their fake values and rewards to the last real tokens. Always stop at the end of each response.
- Bootstrapping through a true end. When the episode really ended (EOS), \(V(s_T) = 0\). When it was only cut off by a length limit, a true end did not happen; some implementations bootstrap with \(V(s_T)\) there, others give a penalty. Know which one your code does.
- Training the critic through the advantage. If \(\hat A_t\) is not detached, the policy loss sends gradients into the critic and both learn the wrong thing.
- Forgetting to normalize advantages, or normalizing over padding. Raw advantages change scale as the reward model changes; most PPO code normalizes them per batch, over real tokens only.
- Using \(\lambda \lt 1\) with a cold critic on very long answers. The reward barely reaches early tokens (\(0.95^{999} \approx 6\times10^{-23}\)); pre-train the critic or use \(\lambda = 1\) for its targets.
- Thinking \(\gamma\) and \(\lambda\) are the same knob. Both shrink the weights by \(\gamma\lambda\) in (65.8), but \(\gamma\) also changes the objective itself (what return we want to maximize), while \(\lambda\) only changes how we estimate the advantage.
Interview questions
Define \(V^\pi(s)\), \(Q^\pi(s,a)\) and \(A^\pi(s,a)\). How are they related?basic
Why use the advantage instead of the raw return in the policy gradient?basic
What is the TD error, and why is it an advantage estimate?core
Explain the bias–variance trade-off between TD and Monte Carlo.core
Write the GAE formula and its recursive form. What does \(\lambda\) control?core
What is the difference between \(\gamma\) and \(\lambda\)?core
How is GAE computed for an LLM response in RLHF? Where does the reward go?core
Why might PPO with GAE fail on long chain-of-thought, and how can you fix it?advanced
What does GRPO give up by removing the critic?advanced
Why is the critic’s target \(\hat A_t + V_{\text{old}}(s_t)\), and what is a \(\lambda\)-return?advanced
Does the value head need to be as large as the policy?advanced
Exercises
- Three tokens, \(\gamma = 1\), rewards \((0, 0, 1)\), critic values \((0.5, 0.6, 0.9)\), \(V = 0\) after the end. Compute \(\delta_t\) and GAE with \(\lambda = 0.5\).
- For the same data, compute the advantages with \(\lambda = 0\) and \(\lambda = 1\). Which tokens change most?
- Show that if the critic is \(V(s) = c\) (the same constant for every state, but 0 at the end) and \(\gamma = \lambda = 1\), GAE gives \(G_t - c\).
- What is the effective horizon \(1/(1 - \gamma\lambda)\) for \(\gamma = 1\), \(\lambda = 0.9\)? How much direct weight does the final reward get at a token 50 steps before the end?
- In Figure 2, find a pair of step sizes for which Monte Carlo beats TD after 100 episodes, if any. What does this tell you about comparing algorithms?
- A 7B policy and a 7B critic are trained with AdamW in mixed precision. How much training-state memory (weights, gradients, optimizer) do the two need together, before activations?
- In Figure 5, set the critic error \(\kappa = 0\). Why does \(\lambda = 0\) now have no bias, and why is its variance still above 0 when \(\sigma \gt 0\)?
Answerscheck
Key takeaways
- \(V^\pi(s)\) is the expected return from a state, \(Q^\pi(s,a)\) from a state and an action (65.2); they are linked by the Bellman equations (65.3).
- The advantage \(A = Q - V\) (65.4) has zero mean under the policy; using it in the policy gradient (65.5) keeps the gradient unbiased and lowers its variance.
- The TD error \(\delta_t\) (65.6) is a low-variance, possibly biased, one-step advantage; the Monte Carlo return is unbiased but noisy (65.7).
- GAE (65.8) sums TD errors with weights \((\gamma\lambda)^l\), computed backward in one pass (65.9); \(\lambda = 0\) is TD and \(\lambda = 1\) is Monte Carlo (65.10).
- In RLHF, the reward model score sits on the last token and KL penalties on every token (65.12); typical \(\gamma = 1\), \(\lambda = 0.95\). For very long answers, the reward decays before reaching early tokens.
- Actor–critic trains a policy and a critic together (65.13, 65.14). The critic costs a full model of memory, which is why GRPO replaces it with group baselines.
Key formulas
References for this session
- Schulman et al. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., chapter 6: Temporal-Difference Learning
- OpenAI (Achiam) (2018). Spinning Up in Deep RL
- Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., chapter 12
- Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., chapter 13: Policy Gradient Methods
- Schulman et al. (2017). Proximal Policy Optimization Algorithms
- Mnih et al. (2016). Asynchronous Methods for Deep Reinforcement Learning
- Huang et al. (2024). The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
- Yuan et al. (2025). What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
- Ouyang et al. (2022). Training language models to follow instructions with human feedback
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Lambert (2025). Reinforcement Learning from Human Feedback (book)