LLM & VLM courseBlog
74Track 4: Post-training and RL · session 22 of 25

Safety alignment: RLAIF and Constitutional AI

Human preference labels are slow, expensive, and hard to collect for harmful content. RLAIF replaces the human labeler with an LLM that reads both answers and picks the better one; Constitutional AI writes the rules down as a short list of principles and uses them twice, first to critique and revise answers for supervised training, then to label preferences for RL. This session builds both methods with real numbers, then covers the helpful-versus-harmless trade-off, over-refusal, and the basics of red-teaming.

Time: 3 to 4 hours (reading 2 to 2.5 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 54, 66, 67 and 68

What you will learn

  • Why AI feedback can replace most human labels, and how an LLM labeler turns its next-token probabilities into a soft preference label
  • Position bias in AI labelers and how swapping the two answers and averaging removes it
  • Constitutional AI step by step: principles, critiques, revisions, the SL-CAI model, AI preference labels, and RL-CAI, with the real numbers from Bai et al. (2022)
  • The RLAIF results of Lee et al. (2023): win rates, harmless rates, labeler size, and direct RLAIF
  • Helpful versus harmless: how Llama 2 combined two reward models, why models over-refuse, and how to measure over-refusal (XSTest)
  • Red-teaming basics: manual and automated attacks, jailbreak families, attack success rate, and why 0 failures in 300 tries does not mean safe
  • A runnable PyTorch pipeline: debiased AI labels, a soft-label reward model, a combined safety reward and a Wilson interval

Why this matters

RLHF needs people to compare thousands of answer pairs. For safety this is the hardest part: someone must read harmful requests and harmful answers all day, and the labels they give are noisy and hard to explain. This session shows how to move most of that work to a model, and how to write the rules down so everyone can read them.

In session 67 people picked the better of two answers, a reward model learned from those picks, and PPO pushed the policy toward high reward. Two problems appear when the goal is a safe assistant:

  • Cost and speed. Every new safety rule (for example, “do not give medical dosages without a warning”) needs a new round of human labels.
  • Hidden rules. The real “specification” of the model is spread over tens of thousands of individual human choices. Nobody can read it, and nobody can change one rule without labeling again.

Two connected ideas solve this. RLAIF (reinforcement learning from AI feedback) uses an LLM to produce the preference labels. Constitutional AI (CAI) writes the safety rules as a short list of natural-language principles (a “constitution”) and uses an LLM to apply them: first to critique and rewrite bad answers, then to label preference pairs. In the original CAI work, humans gave labels only for helpfulness; all harmlessness labels came from the model. [Certain]

Then we look at the other side of safety. A model that refuses too much is also broken: it refuses “How do I kill a Python process?” because it sees the word “kill”. This is over-refusal, and good safety training keeps it low while also keeping harmful compliance low. Finally, red-teaming is how we find out whether all this worked: people and models attack the system on purpose and we count how often they succeed.

Three pipelines side by side

All three methods end the same way: a reward model (also called a preference model, PM) scores answers and an RL algorithm such as PPO trains the policy against it. They differ only in who writes the labels and what the labeler is told.

human workAI worksame in all three
Figure 1. RLHF, RLAIF and Constitutional AI. The reward model and RL steps are the same. RLAIF swaps the human labeler for an LLM. Constitutional AI adds a written constitution and a supervised stage where the model fixes its own answers before any RL.

RLAIF: an LLM as the labeler

The idea is simple. Give an LLM (the AI labeler) a prompt, two candidate answers and an instruction such as “Which summary is more accurate and more useful? Answer 1 or 2.” Then read the labeler’s probabilities for the next token instead of its text. If the logits for the answer tokens “1” and “2” are \(\ell_1\) and \(\ell_2\), the AI preference is a softmax over just these two tokens:

\[ p_{\text{AI}}(A \succ B) = \frac{e^{\ell_1}}{e^{\ell_1} + e^{\ell_2}} = \sigma(\ell_1 - \ell_2) \qquad (A \text{ shown as answer 1}) \tag{74.1} \]
Read the symbols
\(A \succ B\)
“answer A is better than answer B”
\(\ell_1, \ell_2\)
the labeler’s logits for the tokens “1” and “2” at the position where it must answer
\(\sigma\)
the sigmoid \(1/(1+e^{-x})\); a softmax over two items is a sigmoid of their difference

This gives a soft label: a number such as 0.73, not just “A wins”. Soft labels keep the labeler’s doubt. When both answers are about equally good, the label is near 0.5 and the reward model is not pushed to invent a big gap.

Position bias and how to remove it

LLM labelers often prefer whichever answer comes first (or second), regardless of content. This is position bias. Lee et al. (2023) measured how often the labeler picked the same position after the two answers were swapped: 18% for PaLM 2 L, 21% for PaLM 2 S and 56% for PaLM 2 XS. [Certain] Small labelers are much worse. The fix is to ask twice, once in each order, and average:

\[ y = \tfrac12 \Big[\, p_{\text{AI}}(A \succ B \mid A \text{ first}) + p_{\text{AI}}(A \succ B \mid B \text{ first}) \,\Big] \tag{74.2} \]

The reward model is then trained with the usual Bradley–Terry loss from session 67, but with the soft target \(y\) instead of a hard 1:

\[ \mathcal{L}_{\text{RM}} = -\Big[\, y \log \sigma(\Delta) + (1 - y) \log \sigma(-\Delta) \,\Big], \qquad \Delta = r_\phi(x, A) - r_\phi(x, B) \tag{74.3} \]
Read the symbols
\(y\)
the debiased AI label from (74.2), a number between 0 and 1
\(r_\phi(x, A)\)
the reward model’s scalar score for answer A to prompt \(x\); \(\phi\) are its weights
\(\Delta\)
the reward gap; only the difference matters, as in every Bradley–Terry model

This is a binary cross-entropy with a soft target. Setting its derivative to zero gives the best gap for one pair:

\[ \frac{\partial \mathcal{L}_{\text{RM}}}{\partial \Delta} = \sigma(\Delta) - y = 0 \quad\Longrightarrow\quad \Delta^\star = \log \frac{y}{1 - y} \tag{74.4} \]

So a label of 0.5 asks for a gap of 0, and a label of 0.9 asks for a gap of 2.2. A hard label \(y = 1\) asks for an infinite gap, which is one reason soft labels are calmer to train on.

Worked example: one debiased AI label

Order 1 (A shown as “1”): \(\ell_1 = 1.2\), \(\ell_2 = 0.2\). Then \(p = \sigma(1.0) = 0.731\).

Order 2 (B shown as “1”, A as “2”): \(\ell_1 = 0.9\), \(\ell_2 = 0.5\). Now A is token “2”, so \(p(A \succ B) = \sigma(0.5 - 0.9) = \sigma(-0.4) = 0.401\).

The two orders disagree on the winner: the labeler likes slot “1”. The average is \(y = (0.731 + 0.401)/2 = 0.566\): a weak preference for A. The best reward gap is \(\Delta^\star = \log(0.566/0.434) = 0.266\). At that gap the loss is 0.684 (its lowest possible value, the entropy of the label); at \(\Delta = 0\) it is \(\log 2 = 0.693\); at \(\Delta = 2\) it rises to 0.995. Using only order 1 would have told the reward model “A clearly wins”. Explore it in Figure 3.

Direct RLAIF: skip the reward model

Lee et al. also tried direct RLAIF (d-RLAIF): during RL, ask the labeler to rate each answer from 1 to 10, take the probability-weighted average score, and rescale it to \([-1, 1]\). [Certain] With a linear rescale this is

\[ \bar s = \sum_{k=1}^{10} k \; p_{\text{AI}}(\text{score} = k \mid x, a), \qquad r(x, a) = \frac{\bar s - 5.5}{4.5} \tag{74.5} \]

For example, probabilities 0.2, 0.5 and 0.3 on the scores 7, 8 and 9 give \(\bar s = 8.1\) and \(r = 0.578\). (The exact rescale in the paper may differ in detail; the idea is a linear map of the expected score. [Likely]) There is no reward model to go stale, but every RL step now calls a large labeler, which is expensive. This is the same idea as “LLM-as-a-judge” rewards from session 73.

Does it work? The numbers

Lee et al. (2023) compared RLAIF and RLHF on three tasks with a PaLM 2 L labeler. Human raters compared the outputs. [Certain]

Comparison (human judges)SummarizationHelpful dialogue
RLAIF wins over SFT71%63%
RLHF wins over SFT73%64%
RLAIF wins over RLHF (head to head)50%52%
Same-size RLAIF (labeler = policy size) over SFT68%–
d-RLAIF over SFT74%66%

On the harmless dialogue task, the share of answers rated harmless was 64% for SFT, 76% for RLHF and 88% for RLAIF. [Certain] The labeler’s agreement with human labels was around 70–78% depending on task and prompt (for example 78.0% on summarization with a detailed prompt and chain of thought), and it fell from 78.0% to 62.7% when the labeler shrank from PaLM 2 L to PaLM 2 XS. [Certain] Note what these numbers say and do not say: RLAIF matched RLHF on these tasks, with a strong labeler, on tasks where the labeler is already good at judging. They do not show that AI feedback works for skills the labeler itself lacks.

Algorithm 1: RLAIF preference labeling
  1. input: prompts \(\{x\}\), policy \(\pi_{\text{SFT}}\), labeler LLM \(J\), labeling instruction \(I\), small human-labeled check set
  2. for each prompt \(x\): sample two answers \(A, B \sim \pi_{\text{SFT}}(\cdot \mid x)\)
  3. \(\quad p_1 \leftarrow \sigma(\ell_1 - \ell_2)\) from \(J(I, x, A, B)\) // formula (74.1), A shown first
  4. \(\quad p_2 \leftarrow \sigma(\ell_2 - \ell_1)\) from \(J(I, x, B, A)\) // A shown second; read token “2”
  5. \(\quad y \leftarrow (p_1 + p_2)/2\) // formula (74.2)
  6. \(\quad\) store \((x, A, B, y)\)
  7. measure agreement of \(\mathbb{1}[y \gt 0.5]\) with the human check set; fix \(I\) if it is low
  8. train \(r_\phi\) with the soft loss (74.3), then run PPO against \(r_\phi\) with a KL penalty

Constitutional AI, step by step

Constitutional AI (Bai et al., 2022, Anthropic) keeps the same RL ending but adds two things: a written list of principles, and a supervised stage that comes first. The goal is an assistant that is harmless but not evasive: when it declines, it explains why and still helps where it can, instead of a canned “I can’t help with that”. The paper reports that earlier HH-RLHF models were often evasive and that the RL-CAI model was “virtually never evasive”. [Certain]

The constitution

The constitution is a list of short principles written in plain English. The paper used 16 principles; each critique and each label uses one principle chosen at random. [Certain] A principle comes as a pair of instructions. For example, a critique request asks the model to identify specific ways in which its last answer is harmful, unethical, racist, sexist, toxic, dangerous or illegal, and the matching revision request asks it to rewrite the answer to remove that content. [Certain] Because the rules are text, anyone can read them, argue about them, and change one without collecting new labels.

Stage 1: supervised learning from critiques and revisions (SL-CAI)

  1. Start from a helpful-only RLHF model. It is very helpful and therefore answers harmful requests too.
  2. Give it a red-team prompt (a prompt written to provoke a harmful answer). It answers, often badly.
  3. Append a critique request with a random principle. The model writes a critique of its own answer.
  4. Append the matching revision request. The model rewrites its answer.
  5. Repeat the critique–revision loop with new random principles. The paper sampled four critique–revision pairs per prompt. [Certain]
  6. Fine-tune a pretrained model on (prompt, final revision) pairs, mixed with helpful answers to ordinary prompts so it does not become less helpful.

The data for this stage was large: 42,496 human-written red-team prompts plus 140,335 model-generated ones (182,831 in total), and 135,296 human-written helpfulness prompts. [Certain] The result is the SL-CAI model. It is already less harmful, and it is a good starting point for RL because its answers are on-distribution for what RL will reward.

Figure 2. One prompt through Constitutional AI, in the style of the paper’s Wi-Fi example. Steps 1–5 are the supervised stage (critique and revise). Steps 6–8 are the RL stage (AI preference label, preference model, RL). The text is shortened and paraphrased.
How to read Figure 2

The top row shows the pipeline; the highlighted box is the current step. The panel shows the conversation the model sees at that step. Notice that the same model writes the first answer, the critique and the revision: no human writes any of this text. Notice also that the revision does not just refuse; it says why and offers a legitimate path (talk to the neighbor, fix your own Wi-Fi). In step 6 the label is a soft number, averaged over principles, exactly as formula (74.6) says.

Stage 2: RL from AI feedback (RL-CAI)

Now the constitution is used again, this time as instructions for a labeler:

  1. The SL-CAI model samples two answers to a red-team prompt.
  2. A feedback model (in the paper, typically a pretrained LM) sees the conversation, one random principle, and the two answers as a multiple-choice question: “Which of these responses is less harmful? (A) … (B) …”. Few-shot examples of the task are put in its context. [Certain]
  3. The normalized log-probabilities of “(A)” and “(B)” give a soft label, exactly as in (74.1). Averaging over several principles gives an ensemble label:
\[ y = \frac{1}{K} \sum_{k=1}^{K} p_{\text{AI}}(A \succ B \mid x, c_k) \tag{74.6} \]
Read the symbols
\(c_k\)
principle number \(k\) from the constitution
\(K\)
how many principles are averaged for one pair

Example: four principles give 0.82, 0.64, 0.55 and 0.71; the label is \(y = 0.68\). The paper also tried letting a helpful RLHF model think step by step (chain of thought) before choosing. Such reasoning makes the probabilities close to 0 or 1, which is overconfident, so the labels were clamped:

\[ y_{\text{CoT}} = \min\!\big(\max(y, 0.4),\ 0.6\big) \tag{74.7} \]

A chain-of-thought label of 0.97 becomes 0.6. The paper found the 40–60% range worked better than 20–80%, and that for labels without chain of thought, soft labels were much better than hard 0/1 labels. [Certain]

  1. Train one preference model on a mix of human helpfulness labels and AI harmlessness labels.
  2. Run RL (PPO) from the SL-CAI model against this preference model, with the usual KL penalty:
\[ \max_{\theta}\ \E_{x \sim \mathcal{D},\ a \sim \pi_\theta(\cdot \mid x)} \big[\, r_\phi(x, a) \,\big] \;-\; \beta\, \KL\!\big(\pi_\theta(\cdot \mid x)\ \|\ \pi_{\text{SL-CAI}}(\cdot \mid x)\big) \tag{74.8} \]

The RL stage used an extra 491,142 model-generated red-team prompts and 474,300 model-generated helpfulness prompts. [Certain] The main results used 52B models. [Certain]

Algorithm 2: Constitutional AI (both stages)
  1. input: helpful-only model \(M_H\), pretrained model \(M_0\), constitution \(C = \{c_1, \dots, c_{16}\}\), red-team prompts \(X_R\), helpful prompts \(X_H\)
  2. Stage 1 (SL-CAI): for each \(x \in X_R\):
  3. \(\quad a \leftarrow M_H(x)\)
  4. \(\quad\) repeat \(n\) times (paper: 4): pick \(c \sim C\); \(\ \text{crit} \leftarrow M_H(x, a, c.\text{critique})\); \(\ a \leftarrow M_H(x, a, \text{crit}, c.\text{revise})\)
  5. \(\quad\) add \((x, a)\) to \(D_{\text{SL}}\)
  6. add helpful samples \((x, M_H(x))\) for \(x \in X_H\) to \(D_{\text{SL}}\); fine-tune \(M_0\) on \(D_{\text{SL}}\) \(\Rightarrow\) \(M_{\text{SL}}\)
  7. Stage 2 (RL-CAI): for each \(x \in X_R\): sample \(A, B \sim M_{\text{SL}}(x)\); \(\ y \leftarrow\) formula (74.6) with the feedback model
  8. train preference model \(r_\phi\) on AI harmlessness pairs + human helpfulness pairs, loss (74.3)
  9. PPO from \(M_{\text{SL}}\) against \(r_\phi\) with KL penalty (74.8) \(\Rightarrow\) \(M_{\text{RL}}\)

Explore the AI label

Figure 3 builds one AI label from two parts: the labeler’s real preference \(q\) (a logit; positive means A is better) and its position bias \(b\) (extra logit for whichever answer sits in slot “1”). In order 1 the labeler’s logit gap is \(q + b\); in order 2 it is \(-q + b\) for slot “1”, so \(p(A \succ B) = \sigma(q - b)\). Then the reward-model loss (74.3) is computed for a gap \(\Delta\) you choose.

loss (74.3) for the label usedloss with a hard label \(y = 1\)your gap \(\Delta\)
Figure 3. From labeler logits to a soft label to the reward-model loss. The curve is the loss (74.3) over the reward gap \(\Delta\). Its lowest point is at \(\Delta^\star = \log(y/(1-y))\), formula (74.4).
How to read Figure 3

Start with the defaults: the worked example (\(q = 0.3\), \(b = 0.7\)). With swapping, \(y = 0.566\) and the curve’s minimum sits near \(\Delta = 0.27\). Uncheck the box: now only order 1 is used, \(y = 0.731\), and the reward model is told to make a gap of 1.0, almost four times too large. Set \(q = 0\) (answers equally good) and raise \(b\): without swapping the label drifts toward 1 even though there is no real preference; with swapping it stays at 0.5. The grey hard-label curve never has a minimum: it keeps asking for a bigger gap.

Helpful versus harmless

Helpfulness and harmlessness pull in opposite directions on some prompts. The most “harmless” model is one that refuses everything, and it is useless. The most “helpful” model does whatever it is asked, including harmful things. The HH-RLHF work (Bai et al., 2022) collected the two kinds of preference data separately and studied them as competing objectives; the conflict is strongest on borderline prompts, where the same answer is helpful to one reader and risky with another. [Likely]

Two reward models: the Llama 2 recipe

Llama 2-Chat trained two separate reward models, one for helpfulness (\(R_h\)) and one for safety (\(R_s\)), and chose between them per prompt during PPO: [Certain]

\[ R_c(g \mid p) = \begin{cases} R_s(g \mid p) & \text{if is\_safety}(p) \ \text{ or } \ R_s(g \mid p) \lt 0.15 \\ R_h(g \mid p) & \text{otherwise} \end{cases} \qquad \tilde R_c = \text{whiten}\big(\text{logit}(R_c)\big) \tag{74.9} \]
Read the symbols
\(p, g\)
the prompt and the generated answer
\(R_s, R_h\)
the safety and helpfulness reward models, each giving a score in \((0, 1)\)
\(\text{is\_safety}(p)\)
true if the prompt is tagged as possibly unsafe
logit, whiten
undo the sigmoid, then normalize scores to mean 0 and variance 1 so they balance with the KL penalty

The threshold 0.15 was chosen to filter unsafe answers; on Meta’s safety test set it gave precision 0.89 and recall 0.55. [Certain] So for an ordinary prompt, an answer with \(R_s = 0.10\) and \(R_h = 0.90\) gets the safety score: \(\text{logit}(0.10) = -2.197\), a strong penalty, even though it looked very helpful. An answer with \(R_s = 0.60\), \(R_h = 0.80\) gets the helpfulness score, \(\text{logit}(0.80) = 1.386\) (before whitening).

Over-refusal

Over-refusal (also called exaggerated safety or false refusal) is refusing a safe request because it looks like an unsafe one. Typical triggers are words like “kill”, “shoot” or “attack” in harmless contexts (“How do I kill a Python process?”, “How do I shoot a good photo?”), questions about history, or fiction. The XSTest suite measures it with 250 safe prompts across ten prompt types, plus 200 unsafe contrast prompts that a model should refuse. [Certain]

A clean way to think about the trade-off: imagine the model computes an internal “harm score” \(s(x)\) for each prompt and refuses when \(s(x) \gt \tau\). Two error rates follow:

\[ \text{ORR}(\tau) = P\big(s(x) \gt \tau \mid x \text{ safe}\big), \qquad \text{HCR}(\tau) = P\big(s(x) \le \tau \mid x \text{ harmful}\big) \tag{74.10} \]
Read the symbols
ORR
over-refusal rate: the share of safe prompts that get refused
HCR
harmful compliance rate: the share of harmful prompts that get answered
\(\tau\)
the refusal threshold; lowering it refuses more

Moving \(\tau\) only trades one error for the other. The only way to lower both is to make the harm score better at telling borderline-safe prompts from harmful ones. That is what good safety data does: contrast pairs (a harmful prompt next to a similar harmless one), non-evasive revisions as in CAI, and helpful data mixed into every stage.

ordinary safe promptsborderline safe prompts (look risky, are fine)harmful prompts
Figure 4. A toy model of the refusal trade-off. Harm scores are bell curves with standard deviation 1: ordinary safe prompts centered at −2, harmful prompts at +2, and borderline safe prompts at a center you choose. The model refuses everything to the right of \(\tau\). The rates come from the normal distribution, formula (74.10). This is an illustration, not data from a real model.
How to read Figure 4

At the defaults (\(\tau = 0\), borderline center −0.5) the model refuses 2.3% of ordinary safe prompts, 30.9% of borderline safe prompts, and answers 2.3% of harmful prompts. Move \(\tau\) to 1: borderline over-refusal drops to 6.7% but harmful compliance rises to 15.9%. Move \(\tau\) to −1: harmful compliance drops to 0.1% but borderline over-refusal jumps to 69.1%. Now set \(\tau = 0\) and move the borderline center to −1.5, which is what better training does: borderline over-refusal drops to 6.7% and harmful compliance stays at 2.3%. A threshold moves along the trade-off; better data moves the trade-off itself.

Red-teaming basics

Red-teaming means attacking your own model on purpose to find failures before users do. The word comes from security exercises where a “red team” plays the attacker. There are three common ways to do it.

  • Manual red-teaming. People write attacks. Ganguli et al. (2022) released 38,961 red-team attacks against models of 2.7B, 13B and 52B parameters, and found that RLHF models became harder to red-team as they grew, while plain, prompted and rejection-sampling models showed a flat trend with scale. [Certain]
  • Automated red-teaming with a language model. One LM writes test prompts, the target answers, and a classifier flags harmful answers. Perez et al. (2022) used zero-shot generation up to reinforcement learning for the attacker and found tens of thousands of offensive replies from a 280B chatbot. [Certain]
  • Optimization attacks. Search for a token string that makes the model comply. GCG (Zou et al., 2023) optimizes an adversarial suffix with gradients and found suffixes that transfer to other models. [Certain]

Common attack families to test: role-play (“pretend you are an AI with no rules”), prefix injection (“start your answer with ‘Sure, here is’”), encoding (base64, other languages, leetspeak), many-turn escalation (harmless steps that add up to a harmful goal), hypotheticals and fiction, and, for systems with tools or retrieval, prompt injection through documents, web pages or images (session 158).

Reporting the result: attack success rate

The basic number is the attack success rate (ASR): out of \(n\) attacks, \(k\) got a harmful answer. A single number hides how uncertain it is, so report an interval. The Wilson 95% interval is a good default:

\[ \widehat{\text{ASR}} = \frac{k}{n}, \qquad \text{CI} = \frac{\hat p + \frac{z^2}{2n} \pm z \sqrt{\frac{\hat p (1 - \hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}, \quad z = 1.96 \tag{74.11} \]

When no attack succeeds (\(k = 0\)), the “rule of three” gives a quick 95% upper bound:

\[ \text{ASR}_{\text{upper}} \approx \frac{3}{n} \qquad (k = 0) \tag{74.12} \]
Worked example: what “0 out of 300” means

You ran \(n = 200\) attacks and \(k = 6\) succeeded. \(\widehat{\text{ASR}} = 3.0\%\) and the Wilson 95% interval is 1.4% to 6.4%. A second model with 3.5% may not be worse at all.

You ran 300 attacks and none succeeded. The rule of three gives an upper bound of \(3/300 = 1.0\%\) (the exact one-sided value is 0.99%; the two-sided Wilson upper limit is 1.26%). So the honest claim is “the success rate of these kinds of attack is probably below about 1%”, not “the model is safe”. With 50 attacks and no success, the bound is 6%.

Figure 5. The attack success rate and its 95% Wilson interval, formula (74.11). The scale runs from 0% to 20%; wider intervals are cut at the right edge.
How to read Figure 5

Keep \(k/n\) fixed at 3% and move \(n\) from 100 to 1,000 (with \(k\) from 3 to 30): the interval shrinks about 3.5 times (from 1.0–8.5% to 2.1–4.3%), close to the \(\sqrt{10} \approx 3.2\) you expect from the \(1/\sqrt{n}\) rule. Set \(k = 0\) and watch the upper limit fall as \(3/n\). Small red-team runs can rank two models only when their rates are far apart.

Algorithm 3: a red-teaming loop
  1. input: target system \(T\) (model + system prompt + filters), harm taxonomy, attacker pool (people, attacker LM), harm judge \(H\)
  2. repeat each round:
  3. \(\quad\) generate \(n\) attacks per taxonomy category and attack family
  4. \(\quad\) run them on \(T\); judge each answer with \(H\); spot-check a sample of judgments by hand
  5. \(\quad\) report ASR with a Wilson interval (74.11) per category, and the over-refusal rate on a safe contrast set (74.10)
  6. \(\quad\) turn successful attacks into training data (prompts for CAI revisions or preference pairs) and into regression tests
  7. \(\quad\) keep a held-out attack set that is never trained on, to measure real progress

Beyond the training loop: layers of defense

Training is one layer. Real systems add others around the model. [Likely]

  • Input and output classifiers. Llama Guard is a Llama 2 7B model instruction-tuned to classify both user prompts and model answers against a safety risk taxonomy. [Certain] Anthropic’s Constitutional Classifiers are trained on synthetic data generated from a constitution; the paper reports over 3,000 estimated hours of red-teaming, a 0.38% absolute increase in refusals on production traffic, and 23.7% inference overhead. [Certain]
  • Reasoning over the policy. Deliberative alignment (OpenAI, 2024) teaches a reasoning model the text of its safety policies and trains it to recall them and reason about them before it answers, without human-written reasoning chains. [Certain] It is a close cousin of CAI: the rules are text, and the model applies them.
  • System prompts, rate limits, monitoring. Cheap, fast to change, and often the first fix after a new jailbreak is found.

A runnable pipeline in PyTorch

The code below simulates an AI labeler with a known position bias on pairs whose true quality we know, builds debiased and ensembled soft labels, trains a reward model on them with the soft loss (74.3), and checks the Llama 2 reward rule and the Wilson interval. It needs only PyTorch and is deterministic.

import math, torch
import torch.nn.functional as F
torch.manual_seed(0)

# A hidden "true" preference: answer quality = feature . w_true
N, d, K, bias = 2000, 16, 4, 0.7             # pairs, feature size, principles, position bias
w_true = torch.randn(d)
fa, fb = torch.randn(N, d), torch.randn(N, d)
gap = (fa - fb) @ w_true / math.sqrt(d)      # true quality gap A - B
better = gap > 0                             # the truth we want the labels to recover

# 1) the AI labeler: logit("1") - logit("2"), the first slot gets +bias, formulas (74.1)-(74.2)
noise = lambda: 0.8 * torch.randn(N, K)      # each principle reads the pair a bit differently
p_first  = torch.sigmoid( gap[:, None] + bias + noise())    # A shown in slot "1"
p_second = torch.sigmoid( gap[:, None] - bias + noise())    # A shown in slot "2"
y_one = p_first[:, 0]                        # one order, one principle
y = (0.5 * (p_first + p_second)).mean(1)     # swap-average (74.2), then principle ensemble (74.6)
acc = lambda p: ((p > 0.5) == better).float().mean().item()
print(f"label accuracy: one order {acc(y_one):.3f}, debiased + ensemble {acc(y):.3f}")
print(f"mean label: one order {y_one.mean():.3f}, debiased {y.mean():.3f}  (truth: 0.5 on average)")

# 2) reward model trained on soft labels with the soft Bradley-Terry loss (74.3)
tr, te = slice(0, 1500), slice(1500, N)
rm = torch.nn.Linear(d, 1)
opt = torch.optim.Adam(rm.parameters(), lr=3e-2)
for step in range(300):
    delta = (rm(fa[tr]) - rm(fb[tr])).squeeze(-1)           # r(A) - r(B)
    loss = F.binary_cross_entropy_with_logits(delta, y[tr])
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    d_te = (rm(fa[te]) - rm(fb[te])).squeeze(-1)
print(f"held-out RM accuracy vs truth: {((d_te > 0) == better[te]).float().mean():.3f}")

# 3) the worked example by hand
y1 = 0.5 * (torch.sigmoid(torch.tensor(1.0)) + torch.sigmoid(torch.tensor(-0.4)))
print(f"worked example: y = {y1:.3f}, best reward gap = {torch.logit(y1):.3f}")

# 4) Llama 2 style choice between safety and helpfulness rewards (74.9)
def combined_reward(r_safe, r_help, is_safety_prompt, thr=0.15):
    use_safe = is_safety_prompt | (r_safe < thr)
    r = torch.where(use_safe, r_safe, r_help)
    return torch.logit(r.clamp(1e-6, 1 - 1e-6))             # then whitened over the batch
print(combined_reward(torch.tensor([0.10, 0.60]), torch.tensor([0.90, 0.80]),
                      torch.tensor([False, False])))

# 5) attack success rate with a 95% Wilson interval (74.11)-(74.12)
def wilson(k, n, z=1.96):
    p = k / n; den = 1 + z * z / n
    c = (p + z * z / (2 * n)) / den
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / den
    return max(0.0, c - h), c + h
print("ASR 6/200:", [round(v, 4) for v in wilson(6, 200)],
      " 0/300 upper:", round(wilson(0, 300)[1], 4))

Output (CPU, PyTorch 2.x):

label accuracy: one order 0.775, debiased + ensemble 0.940
mean label: one order 0.622, debiased 0.497  (truth: 0.5 on average)
held-out RM accuracy vs truth: 0.992
worked example: y = 0.566, best reward gap = 0.266
tensor([-2.1972,  1.3863])
ASR 6/200: [0.0138, 0.0639]  0/300 upper: 0.0126

Read the first two lines carefully. One biased label per pair is right only 77.5% of the time and its average is 0.622 instead of 0.5: the reward model would learn “first answers are better”. Swapping and averaging over four principles lifts label accuracy to 94.0% and removes the bias. Because the label noise is unbiased after that, the reward model still recovers the true ranking on held-out pairs (99.2%). In real RLAIF the labeler also has systematic errors that no averaging removes; those go straight into the reward model.

Where this is used

System or settingWhat is usedNotes
Anthropic Claude modelsConstitutional AI; a published constitution; constitutional classifiersCAI paper and classifier paper [Certain]; details of current training are not public [Likely]
Llama 2-ChatSeparate safety and helpfulness reward models, rule (74.9), safety SFT and red-teaming[Certain]
Open models trained with AI preference dataAn LLM judge labels pairs, then DPO or PPO (session 68)Very common in open post-training recipes [Likely]
Guard models in productionInput/output classifiers such as Llama GuardRun beside the main model; cheap to update [Certain]
Reasoning modelsDeliberative alignment: reason over a written policy before answering[Certain] for the method; usage across products [Likely]
Video and image models (VLMs)AI feedback with a VLM judge to label safe vs unsafe captions and answers; red-teaming with harmful images and text hidden in images or video framesImages add new attack surfaces: typed text inside an image can carry a jailbreak [Likely]
Video platformsContent moderation and policy-violation detection on video clips; key-moment detection of unsafe segments; Video-RAG that filters retrieved clips before the model sees themSame trade-off as Figure 4: missed violations vs wrongly flagged clips [Likely]

Common mistakes

  • Using one order per pair. The AI labeler’s position bias goes straight into the reward model. Always swap and average (74.2), and report position consistency.
  • Hard-thresholding AI labels. Turning 0.55 into “A wins” throws away the labeler’s doubt and asks the reward model for an unbounded gap. Use soft labels; clamp chain-of-thought labels (74.7).
  • Never checking the labeler against humans. A few hundred human-labeled pairs are enough to estimate agreement. Without them you do not know what you are optimizing.
  • Measuring only harmful compliance. A model that refuses more will always look “safer”. Report over-refusal on a safe contrast set (such as XSTest) next to it.
  • Training only on refusals. Safety data made of “I can’t help with that” teaches evasiveness. CAI-style revisions that explain and redirect keep the model useful.
  • Training on the red-team test set. Once attacks are used for training, they no longer measure anything. Keep a held-out set, and refresh it with new attack families.
  • Reporting ASR without an interval. “0 of 50” only bounds the rate below about 6%. Small runs cannot separate similar models.
  • Over-optimizing the AI preference model. RL against a learned harmlessness reward also Goodharts: harsh tone and repeated boilerplate are the signs. Watch KL and sample outputs by hand.

Interview questions

What is RLAIF, and how does it differ from RLHF?basic
RLAIF uses an LLM, not a person, to compare two answers and produce the preference label. The rest is the same: train a reward model with a Bradley–Terry loss and optimize the policy with RL and a KL penalty. The label is usually soft, read from the labeler’s probabilities for the answer tokens (74.1). Lee et al. (2023) found RLAIF and RLHF were preferred about equally by human judges on summarization and helpful dialogue (50% and 52% head to head), with a strong PaLM 2 L labeler.
Walk me through Constitutional AI.core
Write a constitution: a list of principles in plain language (16 in the paper). Stage 1 (SL-CAI): a helpful-only model answers red-team prompts; it critiques its answer using a random principle and rewrites it, several times (4 in the paper); a pretrained model is fine-tuned on the final revisions plus helpful data. Stage 2 (RL-CAI): the SL-CAI model samples two answers; a feedback model chooses the less harmful one under a random principle, giving a soft label; a preference model is trained on these AI harmlessness labels plus human helpfulness labels; PPO optimizes against it from the SL-CAI model with a KL penalty.
Why does CAI have a supervised stage before RL?core
The helpful-only model’s answers to red-team prompts are often harmful, so RL starting there would have to explore far to find good answers, and the reward model would be judging answers far from what it should learn to prefer. The critique–revision stage moves the starting policy close to harmless, non-evasive behavior cheaply, using only the model and the constitution. RL then refines it, and its samples are closer to the distribution the preference model was trained on.
What is position bias in an AI labeler and how do you fix it?core
The labeler prefers a slot (often the first answer) regardless of content. Lee et al. found the labeler kept the same position after swapping in 18% of cases for PaLM 2 L and 56% for PaLM 2 XS. Fix: label each pair in both orders and average the probabilities for the same answer (74.2). Report position consistency as a quality metric for the labeler. Using larger labelers also helps.
Why use soft labels, and why clamp chain-of-thought labels to 0.4–0.6?advanced
With the soft Bradley–Terry loss, the best reward gap is \(\log(y/(1-y))\) (74.4): a soft label asks for a gap matching the labeler’s confidence, while a hard label asks for an infinite gap and over-trusts noisy judgments. Chain-of-thought reasoning makes the labeler commit, so its probabilities sit near 0 or 1 and are overconfident; clamping keeps the target gap within \(\pm\log(0.6/0.4) = \pm0.405\). The CAI paper found 40–60% better than 20–80%, and soft labels much better than hard ones without chain of thought.
How did Llama 2 balance helpfulness and safety in RLHF?core
It trained two reward models. During PPO it used the safety reward if the prompt was tagged as safety-related or if the safety score was below 0.15, and the helpfulness reward otherwise (74.9). Scores were passed through a logit and whitened to balance with the KL term. The 0.15 threshold gave precision 0.89 and recall 0.55 for unsafe answers on Meta’s safety test set. Two models avoid one reward model having to trade off conflicting goals inside itself.
What is over-refusal and how do you measure it?basic
Refusing safe requests that look like unsafe ones, such as “How do I kill a Python process?”. Measure it with a set of safe prompts that share surface features with harmful ones (XSTest has 250 safe prompts in ten types and 200 unsafe contrasts), and report the refusal rate on the safe part next to the compliance rate on the unsafe part. Reduce it with contrast pairs in training, non-evasive revisions, and helpful data in every stage.
A new checkpoint cuts harmful compliance from 5% to 2%. Is it better?core
Not yet known. Check (1) over-refusal on a safe contrast set: if it rose a lot, the model may just refuse more (moving the threshold in Figure 4, not improving the score). (2) Helpfulness on normal prompts. (3) Uncertainty: with 200 attacks, 2% and 5% have Wilson intervals of about 0.8–5.0% and 2.7–9.0%, which overlap. (4) Whether the attacks were held out from training. Only if both error rates improve on held-out sets is it clearly better.
Name a few jailbreak families and how you would defend against each.advanced
Role-play and persona attacks, prefix injection (“start with ‘Sure’”), encodings and low-resource languages, many-turn escalation, fiction and hypotheticals, optimized adversarial suffixes (GCG), and prompt injection through tools or retrieved documents. Defenses: include each family in red-team prompts used for CAI revisions or preference data; input/output classifiers (Llama Guard, constitutional classifiers) as a second layer; perplexity or pattern filters for optimized suffixes; treat retrieved content as data, not instructions; and multi-turn evaluation, not only single prompts.
What are the risks of replacing humans with AI feedback?advanced
The labeler’s biases (position, length, self-preference, style) and blind spots become the reward. RL then amplifies them, and the policy can learn to please the labeler instead of the user (reward hacking). If the labeler is the same model family, errors are correlated and hard to notice. Results so far show RLAIF works where the labeler is already a good judge; it does not show it works for skills beyond the labeler. Mitigations: keep a human-labeled check set, use diverse principles and labelers, debias, soft labels, KL control, and human spot checks.
How would you report a red-teaming result to a product team?basic
Per harm category and attack family: number of attempts, number of successes, ASR with a 95% interval, plus the over-refusal rate on a matched safe set and examples of the worst failures. State what was tested (model, system prompt, filters) and what was not. If there were no successes, give the upper bound (about \(3/n\)), not “safe”.

Exercises

  1. An AI labeler gives logits \(\ell_1 = 2.0\), \(\ell_2 = 0.5\) with A first, and \(\ell_1 = 1.0\), \(\ell_2 = 1.5\) with B first. Compute the debiased label \(y\) and the best reward gap \(\Delta^\star\).
  2. Show that the gradient of (74.3) with respect to \(\Delta\) is \(\sigma(\Delta) - y\).
  3. Chain-of-thought labels for three pairs are 0.98, 0.52 and 0.10. What are the clamped labels (74.7), and the largest reward gap they can ask for?
  4. Under rule (74.9), compute the reward (before whitening) for: (a) an ordinary prompt with \(R_s = 0.30\), \(R_h = 0.95\); (b) a safety-tagged prompt with \(R_s = 0.70\), \(R_h = 0.20\).
  5. In Figure 4’s toy model with \(\tau = 0.5\) and the borderline center at −0.5, compute the borderline over-refusal rate and the harmful compliance rate.
  6. You ran 150 attacks with no success. Give the rule-of-three bound. How many attacks would you need to claim the rate is below 0.5%?
Answerscheck
1. Order 1: \(\sigma(1.5) = 0.818\). Order 2: A is “2”, so \(\sigma(1.5 - 1.0) = \sigma(0.5) = 0.622\). \(y = 0.720\), \(\Delta^\star = \log(0.720/0.280) = 0.944\). 2. \(\frac{d}{d\Delta}\log\sigma(\Delta) = 1 - \sigma(\Delta)\) and \(\frac{d}{d\Delta}\log\sigma(-\Delta) = -\sigma(\Delta)\); so the gradient is \(-[y(1-\sigma) - (1-y)\sigma] = \sigma(\Delta) - y\). 3. 0.6, 0.52, 0.4; the largest gap is \(\log(0.6/0.4) = 0.405\). 4. (a) \(R_s = 0.30 \ge 0.15\) and not tagged, so \(R_h\): \(\text{logit}(0.95) = 2.944\). (b) Tagged, so \(R_s\): \(\text{logit}(0.70) = 0.847\). 5. Borderline: \(1 - \Phi(0.5 + 0.5) = 1 - \Phi(1) = 15.9\%\). Harmful compliance: \(\Phi(0.5 - 2) = \Phi(-1.5) = 6.7\%\). 6. \(3/150 = 2\%\). For 0.5%: \(n = 3/0.005 = 600\) attacks with no success.

Key takeaways

  • RLAIF reads an LLM labeler’s token probabilities as a soft preference (74.1); swap-and-average (74.2) removes position bias; the reward model trains on the soft label (74.3), whose best gap is \(\log(y/(1-y))\) (74.4).
  • With a strong labeler, RLAIF matched RLHF in human judgments (50% and 52% head to head) and gave a higher harmless rate (88% vs 76%) in Lee et al. (2023).
  • Constitutional AI: principles → critiques → revisions → SL-CAI model → AI preference labels averaged over principles (74.6) → preference model → RL with KL (74.8). Humans labeled only helpfulness.
  • Helpful and harmless conflict on borderline prompts. Llama 2 used two reward models and a 0.15 switch rule (74.9). A refusal threshold only trades over-refusal for harmful compliance (74.10); better data moves both.
  • Red-team with people, attacker models and optimization attacks; report ASR with an interval (74.11); zero successes in \(n\) tries bounds the rate at about \(3/n\) (74.12).

Key formulas

\[ y = \tfrac12\big[p(A \succ B \mid A\text{ first}) + p(A \succ B \mid B\text{ first})\big], \quad \mathcal{L} = -\big[y\log\sigma(\Delta) + (1-y)\log\sigma(-\Delta)\big], \quad \Delta^\star = \log\tfrac{y}{1-y} \]
\[ \text{ORR} = P(s \gt \tau \mid \text{safe}), \quad \text{HCR} = P(s \le \tau \mid \text{harmful}), \quad \text{ASR}_{\text{upper}}(k = 0) \approx \tfrac{3}{n} \]

References for this session

  1. Bai et al. (2022). Constitutional AI: Harmlessness from AI Feedback
  2. Lee et al. (2023). RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  3. Bai et al. (2022). Constitutional AI: Harmlessness from AI Feedback
  4. Lee et al. (2023). RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  5. Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  6. Ouyang et al. (2022). Training language models to follow instructions with human feedback
  7. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  8. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  9. Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  10. Zou et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models
  11. Perez et al. (2022). Red Teaming Language Models with Language Models
  12. Ganguli et al. (2022). Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
  13. Röttger et al. (2023). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
  14. Inan et al. (2023). Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
  15. Sharma et al. (2025). Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming
  16. Guan et al. (2024). Deliberative Alignment: Reasoning Enables Safer Language Models
  17. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena