Pretraining objectives
Raw text has no labels, so pretraining must invent a task from the text itself. This session compares the tasks that were used at scale: next-token prediction (GPT), masked tokens (BERT, ELECTRA), span corruption (T5), prefix LM, UL2 mixtures and fill-in-the-middle. It ends with the reason the simplest one won: predicting the next token is the same as compressing text.
What you will learn
- The general recipe of self-supervised pretraining: a hidden view, a set of targets and an attention mask
- The causal LM loss from the chain rule, the shift by one, and why every token is a target
- Masked LM in BERT: the 15% selection, the 80/10/10 rule and why it exists, and what MLM cannot do; ELECTRA's replaced token detection
- Span corruption, prefix LM, the UL2 mixture of denoisers and fill-in-the-middle for code
- Why decoder-only next-token prediction became the default, with the evidence from Wang et al. (2022)
- The compression view: loss = entropy + KL, arithmetic coding, and bits per byte as a fair metric
- Multi-token prediction, and which tokens get a loss in VLMs and video models
Why this matters
A language model is pretrained on trillions of tokens of raw text, and nobody labels that text. So the first design question is: what task do we make out of raw text? That task is the pretraining objective. It decides what the model learns, what it can do afterwards, and how much it learns from each token of compute.
Over the last eight years, four families of objectives were tried at scale:
- Causal language modeling (GPT, Llama, Qwen, DeepSeek): predict the next token from the tokens before it.
- Masked language modeling (BERT, RoBERTa): hide some tokens and predict them from both sides.
- Span corruption (T5) and denoising (BART): damage the input, and let a decoder rebuild the missing parts.
- Prefix language modeling (UniLM, PaliGemma, part of UL2): read a prefix with full attention, then predict the rest left to right.
Today almost every large LLM and VLM is pretrained with the first one, the simplest. [Certain] GPT-3 (175B parameters, 300B training tokens) showed that a model trained only to predict the next token can translate, answer questions and do new tasks from a few examples in the prompt. [Certain] The last part of this session asks the deep question behind that result: why does such a simple objective learn so much? The answer is the link between prediction and compression.
This session builds on session 18 (which tokens can see which), session 23 (the three model shapes) and session 24 (T5 span corruption in detail). Here we compare the objectives side by side, add the ones those sessions skipped (ELECTRA, UL2, fill-in-the-middle, multi-token prediction), and study the compression view. The next session, session 32, follows one batch of the causal objective through the forward pass, the backward pass and the update.
What a pretraining objective is
All the objectives in this session follow one recipe, called self-supervised learning: the labels come from the data itself. Take a text \(\mathbf{x} = (x_1, \dots, x_T)\). Make a view \(\tilde{\mathbf{x}}\) of it that hides some information: cut off the future, put [MASK] tokens in, or delete spans. Then train the model to predict the hidden tokens from the view. The loss is the usual cross-entropy (session 3):
- Read the symbols
- \(\mathbf{x}\)
- one training text, as token IDs
- \(\tilde{\mathbf{x}}\)
- what the model is allowed to see when it predicts (the “view”); it depends on the objective
- \(\mathcal{T}\)
- the set of target positions: the tokens that get a loss
- \(p_\theta(x_t \mid \tilde{\mathbf{x}})\)
- the probability the model gives to the true token at position \(t\), from a softmax over the vocabulary
- \(\E\)
- average over texts and, for random objectives, over the random choice of hidden tokens
So an objective is fixed by three choices: (1) what is hidden, (2) which positions get a loss, and (3) which attention mask the model uses so that it cannot see the answer. The mask is the part people forget. If a token can attend to its own target, the loss drops to almost zero and the model learns nothing.
Figure 1 shows the same 10-token sentence under five objectives. Look at three things for each: the model input, the targets, and the attention pattern.
How to read Figure 1
Causal LM has the most targets (9 of 10) but each prediction sees only the left side. Masked LM sees both sides, but only 2 of 10 tokens are targets. Span corruption shortens both the input and the target, because each hidden span becomes one sentinel token. In prefix LM, move the slider: the top-left block (the prefix) becomes fully visible, and the number of targets falls to \(10 - k\). Fill in the middle is plain causal LM on a reordered text: the model reads prefix and suffix first, so when it writes the middle it has seen both sides.
Causal language modeling: predict the next token
The chain rule of probability writes the probability of any text as a product of next-token probabilities. It is exact; no assumption is made:
Here \(x_{\lt t}\) means all tokens before position \(t\) (for \(t = 1\), just a begin-of-sequence token). Taking \(-\log\) turns the product into a sum, and averaging over positions gives the causal language modeling (CLM) loss, also called the next-token loss:
- Read the symbols
- \(T\)
- number of target tokens in the sequence
- \(x_{\lt t}\)
- the past: what the causal mask lets position \(t\) see
- \(\mathcal{L}_{\text{CLM}}\)
- average negative log-likelihood in nats per token; \(e^{\mathcal{L}}\) is the perplexity
Three properties make this objective special:
- Every token is a target. In formula (31.1), \(\mathcal{T}\) is all positions. With the causal mask (session 18), one forward pass computes all \(T\) predictions at once. In code, this is the famous “shift by one”: the logits at positions \(1..T-1\) are scored against the tokens at positions \(2..T\).
- It is a real probability model. Formula (31.2) gives the exact probability of any text. You can score texts, compare models by perplexity, and sample.
- Training and use are the same task. Generation is just predicting the next token again and again (session 78). Nothing has to be changed after pretraining, and the KV cache works because the past never changes.
A model gives the true next tokens the probabilities 0.5, 0.25, 0.8 and 0.1. The per-token losses are \(-\ln 0.5 = 0.693\), \(-\ln 0.25 = 1.386\), \(-\ln 0.8 = 0.223\) and \(-\ln 0.1 = 2.303\) nats. The mean is \(4.605 / 4 = 1.151\) nats per token. Perplexity is \(e^{1.151} = 3.16\): on average the model is as unsure as if it chose among about 3 equally likely tokens. In bits, \(1.151 / \ln 2 = 1.661\) bits per token. Keep that last number in mind; it comes back in the compression section.
The weakness: each prediction sees only one side. For a task like “is this word a person’s name?”, the words to the right help a lot, and a causal model does not see them at that position. This is the motivation for masked language modeling.
Masked language modeling: use both sides
BERT (Devlin et al., 2018) removed the causal mask: every token sees every other token. Now a token cannot be its own target (it would just copy itself), so BERT hides some tokens and predicts only those. The masked language modeling (MLM) loss is
- Read the symbols
- \(\mathcal{M}\)
- the randomly selected positions (15% of tokens in BERT)
- \(\tilde{\mathbf{x}}\)
- the whole sequence after corruption; position \(t\) sees all of it, left and right
The 15% and the 80/10/10 rule
BERT selects 15% of the tokens. Of the selected tokens, 80% are replaced by [MASK], 10% by a random token, and 10% are left unchanged; all selected tokens are predicted. [Certain] The paper’s example: if “hairy” in “my dog is hairy” is selected, the input becomes “my dog is [MASK]” 80% of the time, “my dog is apple” 10% of the time, and stays “my dog is hairy” 10% of the time.
Why not always use [MASK]? Because [MASK] never appears when the model is later fine-tuned or used. If the model only ever had to predict at [MASK] positions, it could put little effort into the vectors of normal tokens. With random and unchanged tokens mixed in, the model never knows which visible token might be a target or might be wrong, so it must build a good vector for every token. [Certain: this is the paper’s reason.]
Selected: \(\operatorname{round}(0.15 \times 512) = 77\) tokens. On average \(0.8 \times 77 = 61.6\) become [MASK], \(7.7\) become random tokens, and \(7.7\) stay the same. So only \(0.8 \times 0.15 = 12\%\) of the input tokens are [MASK]. A causal LM on the same 512 tokens gets 511 or 512 targets, about 6.6 times more training signal per forward pass. (Each MLM prediction uses both sides, so one MLM target may teach more than one CLM target; the ratio is not the whole story. [Likely])
- \(n \leftarrow \operatorname{round}(\text{rate} \times N)\) // number of targets
- \(\mathcal{M} \leftarrow\) first \(n\) positions of a random shuffle
- for \(t \in \mathcal{M}\), left to right:
- draw \(u \sim U(0, 1)\)
- if \(u \lt 0.8\): \(\tilde{x}_t \leftarrow\) [MASK]
- else if \(u \lt 0.9\): \(\tilde{x}_t \leftarrow\) a random word
- else: keep \(\tilde{x}_t = x_t\) // still a target
- labels: \(y_t = x_t\) for \(t \in \mathcal{M}\), \(-100\) elsewhere // loss (31.4)
How to read Figure 2
Press Play. The first step picks the targets (outlined in blue in the original text). Then each selected token gets its own random number \(u\): left of 0.8 it becomes [MASK], between 0.8 and 0.9 a random word, right of 0.9 it stays as it is. Press “New random draw” several times. The number of selected tokens is fixed by the rate, but the split between [MASK], random and unchanged changes from draw to draw; only on average is it 80/10/10. Notice the green “unchanged” tokens: in the input they look normal, yet they are targets. That is why the model must check every token. Raise the rate to 40%: more targets per sequence, but less context is left to predict from.
What MLM cannot do
- It cannot generate text well. It was never trained to write left to right, and the [MASK] tokens it predicts are predicted independently of each other. If “New [MASK] [MASK]” could be “New York City” or “New Delhi, India”, the model may output “New York India”.
- It is not a full probability model. There is no exact formula like (31.2) for \(p(\mathbf{x})\), so you cannot get a perplexity for a text in one pass.
- Few targets. Only 15% of positions learn per step, as computed above.
Fixes inside the encoder family
RoBERTa (2019) kept MLM but drew a new mask every time a sequence was seen (dynamic masking), dropped BERT’s second task (next-sentence prediction), and trained longer on more data. [Certain] XLNet (2019) tried to get both sides and a real probability model by training on random orders of the factorization (31.2) instead of left to right. [Certain] ELECTRA (2020) fixed the few-targets problem. A small MLM “generator” fills the masked positions with plausible tokens; the main model, the discriminator, then says for every token whether it is original or replaced. This is called replaced token detection:
- Read the symbols
- \(\theta_G, \theta_D\)
- weights of the small generator and of the main discriminator
- \(D_t\)
- the discriminator’s probability that token \(t\) is the original
- \(\mathbb{1}(\cdot)\)
- 1 if the condition is true, else 0
- \(\lambda\)
- weight of the discriminator loss; ELECTRA uses 50 [Certain]
The discriminator loss covers all \(T\) positions, not just the 15%. ELECTRA-Small, trained on one GPU for 4 days, beat GPT on the GLUE benchmark although GPT used about 30 times more compute, and ELECTRA-Large matched RoBERTa and XLNet with less than a quarter of their compute. [Certain: from the paper’s abstract.] More on encoder models in session 33.
Span corruption and prefix LM
Span corruption (T5)
T5 drops about 15% of the tokens in short runs (mean length 3), replaces each run by one sentinel token, and trains a decoder to output only the dropped runs, each introduced by its sentinel. [Certain] The loss is the causal loss of the decoder on that short target, given the corrupted input \(\tilde{\mathbf{x}}\) read by the encoder:
Unlike MLM, the hidden tokens of one span are predicted one after another, so they depend on each other, and the model learns to write. Unlike CLM, the encoder sees both sides of every hole. The cost: only the target tokens are predicted, and the model has two stacks. The full mechanics, with sentinel IDs and sequence lengths, are in session 24. BART is the same idea with other kinds of damage (deleting, shuffling, filling spans) and a decoder that writes the whole original text. [Certain]
Prefix LM
A prefix LM is a single decoder stack with a mixed mask: the first \(k\) tokens (the prefix) see each other fully, and the rest is causal. Only the tokens after the prefix get a loss:
It is a middle point: the input part (an instruction, a document to summarize, or the image tokens of a VLM) gets bidirectional attention, as in an encoder, and the output part is generated, as in a decoder. UniLM used it in 2019, and PaliGemma uses it today: the image and the text prompt form the prefix, and the answer is the causal part. [Certain] Move the slider in Figure 1 to see how \(k\) changes the mask and the number of targets.
Mixing objectives: UL2
Different objectives teach different skills, so why not mix them? UL2 (Tay et al., 2022) trains one model on a mixture of 7 “denoisers”, with roughly equal shares [Certain]:
- R-denoisers (regular): T5 span corruption, 15% of tokens, mean span 3 or 8.
- S-denoiser (sequential): prefix LM, where the target is a single span at the end, about 25% of the text.
- X-denoisers (extreme): long spans (mean 64) or a high rate (50%), so the model must write a lot from little context.
A special token at the start of the input tells the model which mode it is in, and the same token can be used later to switch modes. The paper writes them [R], [S], [X]; the released checkpoints use [NLU], [NLG] and [S2S]. [Likely]
Fill in the middle (FIM)
Code editors need infilling: write the missing body of a function, given the code before and after the cursor. A causal model sees only the left side. Bavarian et al. (2022) found a trick that keeps the causal objective unchanged: cut a document into prefix \(\mathbf{p}\), middle \(\mathbf{m}\) and suffix \(\mathbf{s}\), then move the middle to the end, with three marker tokens:
Then train with the normal loss (31.3). When the model writes \(\mathbf{m}\), it has already read both \(\mathbf{p}\) and \(\mathbf{s}\). At inference you give it \(\texttt{[PRE]}\,\mathbf{p}\,\texttt{[SUF]}\,\mathbf{s}\,\texttt{[MID]}\) and let it generate until \(\texttt{[EOT]}\). This order is called PSM (prefix, suffix, middle); SPM puts the suffix first. The paper transforms 50% of documents in its main models, says that 50% to 90% is a reasonable choice, cuts at the character level before tokenization, and finds that left-to-right quality does not drop, which it calls “FIM for free”. [Certain] Most code LLMs are trained this way; StarCoder, for example, writes the markers as <fim_prefix>, <fim_suffix> and <fim_middle>. [Likely]
Building the training examples
The objectives differ mostly in how a raw token sequence becomes (input, targets, mask). Algorithm 1 writes all of them in one place.
- input: tokens \(x_1, \dots, x_T\), objective, random generator
- if causal LM: input \(\leftarrow x_{1..T-1}\); targets \(\leftarrow x_{2..T}\); mask \(\leftarrow\) lower triangle // formula (31.3)
- if masked LM: choose \(\mathcal{M}\), 15% of positions; for each \(t \in \mathcal{M}\) draw \(u \sim U(0,1)\):
- \(u \lt 0.8\): \(\tilde{x}_t \leftarrow\) [MASK]; \(\;0.8 \le u \lt 0.9\): \(\tilde{x}_t \leftarrow\) random token; else keep \(x_t\)
- targets \(\leftarrow x_t\) for \(t \in \mathcal{M}\), ignore elsewhere; mask \(\leftarrow\) all ones // formula (31.4)
- if span corruption: pick spans (15%, mean length 3); encoder input \(\leftarrow\) kept tokens with one sentinel per span; decoder target \(\leftarrow\) sentinel, span, sentinel, span, …, end // formula (31.6)
- if prefix LM: choose \(k\); mask \(\leftarrow\) lower triangle with the top-left \(k \times k\) block set to ones; targets only for \(t \gt k\) // formula (31.7)
- if FIM (with probability 0.5): cut at two random character positions; reorder as in (31.8); then treat as causal LM
- return input, targets (with ignore index), mask
The loss side is the same for every objective: cross-entropy over the positions whose target is not “ignore”. This is why one training loop can serve all of them.
- input: model, input \(\tilde{\mathbf{x}}\), targets \(\mathbf{y}\) (value \(-100\) means “no loss here”), mask \(A\)
- \(Z \leftarrow \text{model}(\tilde{\mathbf{x}}, A)\) // logits, shape (B, T, V)
- \(\ell_t \leftarrow -\log \softmax(Z_t)[y_t]\) for every \(t\) with \(y_t \ne -100\)
- \(\mathcal{L} \leftarrow\) mean of \(\ell_t\) over those positions // formula (31.1)
- return \(\mathcal{L}\)
Figure 3 runs Algorithm 2 on the sentence from Figure 1, for four objectives. The “model” is a toy that you can check by hand: bigram counts from six short football sentences, with add-0.1 smoothing over a 16-word vocabulary, so \(P(w \mid a) = (\text{count}(a, w) + 0.1) / (\text{count}(a) + 1.6)\). Its logit for a candidate word \(w\) is \(z[w] = \ln P(w \mid \text{left word})\). When the mask also lets it see the right side (masked LM), it adds \(\ln P(\text{right word} \mid w)\). Then a softmax turns the logits into probabilities, as in a real model. The loop visits every position: it skips positions whose label is \(-100\) and adds \(-\ln p\) for the others.
- \(\tilde{\mathbf{x}}, \mathbf{y}, A \leftarrow\) make the example // Algorithm 1
- \(S \leftarrow 0,\ n \leftarrow 0\)
- for each position \(t\):
- if \(y_t = -100\): skip // no loss here
- \(p \leftarrow \softmax(z_t)\) // \(z_t\) = model(\(\tilde{\mathbf{x}}, A\)) at \(t\)
- \(\ell_t \leftarrow -\ln p[y_t]\)
- \(S \leftarrow S + \ell_t,\ n \leftarrow n + 1\)
- return \(\mathcal{L} = S / n\) // formula (31.1)
How to read Figure 3
Press Play with causal LM. The label row is the input row shifted one place to the left: position \(t\) is scored against \(x_{t+1}\), and the last position has no label. The orange box is the position being scored; the tinted tokens are the ones the mask lets it see. Now switch to masked LM. Only the two [MASK] positions have labels, and each one now sees its right neighbor too. Compare “saves”: causal LM gives it \(p = 0.457\) (loss 0.784, it sees only “keeper”), masked LM gives \(p = 0.710\) (loss 0.342, it also sees “the”). The causal loss is 0.832 nats over 9 targets; the masked loss is 0.390 over only 2. A lower number from far fewer targets is not a better model. In span corruption, the encoder row gets no loss at all; the decoder writes only the dropped words and the sentinels. In this toy, the sentinel that comes next always gets \(p = 0.9\); this is a fixed rule of the toy, not a measured number. In prefix LM, move \(k\): the positions inside the prefix turn into \(-100\), and the mean is taken over \(10 - k\) targets only.
The PyTorch version. One tiny transformer block, four objectives, each made by changing only the mask and the targets. It runs on a CPU in about a second and prints the same numbers every time.
import math, torch import torch.nn.functional as F torch.manual_seed(0) V, D, T = 1000, 64, 16 # vocab, width, sequence length MASK, PRE, SUF, MID, EOT = 996, 997, 998, 999, 0 # toy special-token ids class TinyLM(torch.nn.Module): """One transformer block; the mask decides causal, bidirectional or prefix.""" def __init__(self): super().__init__() self.emb = torch.nn.Embedding(V, D) self.pos = torch.nn.Embedding(64, D) self.block = torch.nn.TransformerEncoderLayer(D, 4, 4 * D, dropout=0.0, batch_first=True) self.head = torch.nn.Linear(D, V) def forward(self, x, allowed): # allowed[i, j] = True: i may look at j h = self.emb(x) + self.pos(torch.arange(x.shape[1])) h = self.block(h, src_mask=~allowed) # PyTorch: True in src_mask means blocked return self.head(h) # logits (B, T, V) def causal(T): return torch.tril(torch.ones(T, T, dtype=torch.bool)) def full(T): return torch.ones(T, T, dtype=torch.bool) def prefix(T, k): # bidirectional on the first k tokens a = causal(T).clone(); a[:k, :k] = True; return a model = TinyLM().eval() x = torch.randint(1, 996, (2, T)) # a batch of 2 "documents" # 1) causal LM, formula (31.3): every position predicts the next token logits = model(x, causal(T)) loss_clm = F.cross_entropy(logits[:, :-1].reshape(-1, V), x[:, 1:].reshape(-1)) print("causal LM loss", round(loss_clm.item(), 3), "targets", x[:, 1:].numel(), "log V", round(math.log(V), 3)) # 2) masked LM, formula (31.4), with BERT's 80/10/10 rule def bert_mask(x, rate=0.15, g=None): sel = torch.rand(x.shape, generator=g) < rate # positions to predict u = torch.rand(x.shape, generator=g) inp = x.clone() inp[sel & (u < 0.8)] = MASK # 80%: [MASK] rnd = sel & (u >= 0.8) & (u < 0.9) # 10%: random token inp[rnd] = torch.randint(1, 996, (int(rnd.sum()),), generator=g) labels = torch.where(sel, x, torch.full_like(x, -100)) # 10%: unchanged, still predicted return inp, labels g = torch.Generator().manual_seed(1) inp, labels = bert_mask(x, g=g) logits = model(inp, full(T)) # no mask: both sides visible loss_mlm = F.cross_entropy(logits.reshape(-1, V), labels.reshape(-1), ignore_index=-100) print("masked LM loss", round(loss_mlm.item(), 3), "targets", int((labels != -100).sum())) # 3) prefix LM, formula (31.7): bidirectional prefix of k tokens, loss only after it k = 6 logits = model(x, prefix(T, k)) loss_plm = F.cross_entropy(logits[:, k - 1:-1].reshape(-1, V), x[:, k:].reshape(-1)) print("prefix LM loss", round(loss_plm.item(), 3), "targets", x[:, k:].numel()) # 4) fill in the middle, formula (31.8), PSM order: reorder, then plain causal LM def fim_psm(doc, i, j): p, m, s = doc[:i], doc[i:j], doc[j:] return torch.cat([torch.tensor([PRE]), p, torch.tensor([SUF]), s, torch.tensor([MID]), m, torch.tensor([EOT])]) f = fim_psm(x[0], 5, 9) logits = model(f[None], causal(len(f))) loss_fim = F.cross_entropy(logits[0, :-1], f[1:]) print("FIM length", len(f), "loss", round(loss_fim.item(), 3))
Output (PyTorch 2.8, CPU): causal LM loss 7.142 targets 30 log V 6.908, masked LM loss 7.554 targets 2, prefix LM loss 7.031 targets 20, FIM length 20 loss 7.139. An untrained model is close to uniform, so every loss is near \(\ln 1000 = 6.908\). Look at the target counts: 30 for causal LM, but only 2 for masked LM on the same 32 tokens (15% is about 5 on average; this draw happened to pick 2). Note also logits[:, k-1:-1] in the prefix LM: position \(k-1\) (the last prefix token, counting from 0) predicts the first target. Getting this off by one is the most common bug.
Comparing the objectives, and why next-token won
| Objective | Sees both sides? | Targets per 512 tokens | Exact \(p(\mathbf{x})\)? | Generates? | Used by |
|---|---|---|---|---|---|
| Causal LM (31.3) | No | 511 to 512 | Yes | Yes | GPT, Llama, Qwen, DeepSeek, most VLMs |
| Masked LM (31.4) | Yes | about 77 | No | Poorly | BERT, RoBERTa, ModernBERT |
| Replaced token detection (31.5) | Yes | 512 (binary) | No | No | ELECTRA, DeBERTa-v3 [Likely] |
| Span corruption (31.6) | Yes (encoder) | about 104 (decoder) | No | Yes | T5, mT5, Flan-T5 |
| Prefix LM (31.7) | In the prefix | \(512 - k\) | Of the part after the prefix | Yes | UniLM, PaliGemma, UL2 (S mode) |
| FIM + causal (31.8) | For the middle, yes | all positions, as in causal LM (4 marker tokens added) | Yes | Yes, and infills | Code LLMs |
(The 104 for span corruption is from session 23: 77 noise tokens in 26 spans, plus one sentinel per span and an end token.)
A careful study by Wang et al. (2022) trained many combinations of architecture and objective at the same compute. Right after pretraining, causal decoder-only models with the plain next-token objective had the best zero-shot results. After an extra stage of multitask fine-tuning, models trained with masked or span objectives (non-causal decoders and encoder–decoders) did best, and a causal decoder could be cheaply adapted into a non-causal one. [Certain] So “which objective is best” depends on how the model will be used. In practice, the field chose the causal decoder for several reasons together [Likely]:
- Most signal per FLOP. Every token is a target (the 6.6× from the worked example).
- One model, one task. The pretraining task is generation, so the model is ready for prompting, chat, tools and reasoning without a new head or a new mask.
- Simple and fast inference. One stack, a causal mask and a KV cache.
- Few-shot and in-context learning appeared with scale in GPT-3 and fit the left-to-right format naturally.
- It is a full probability model, which brings perplexity, scaling laws (session 35), sampling and the compression view below.
Why next-token prediction learns so much: the compression view
The next-token objective looks too simple to produce a model that writes code and explains physics. Two ideas explain why it is not.
Idea 1: the loss can only reach its minimum by modeling everything
Let \(p^\ast\) be the true distribution of text and \(q = p_\theta\) the model. The expected loss splits into two parts (session 3):
- Read the symbols
- \(H(p^\ast)\)
- the entropy of real text: the part of the next token that nobody can predict (a random name, a coin flip in a story)
- \(\KL(p^\ast \Vert q)\)
- how far the model is from the truth; it is the only part training can reduce
So the loss keeps going down only as long as the model captures more of the real structure of text. And the structure of text is the structure of the world that wrote it. To predict the next token of “The capital of Australia is”, the model needs a fact. Of “17 × 3 =”, arithmetic. Of a Python function after its docstring, programming. Of a mystery novel’s last chapter, “the murderer is”, it needs to track the whole plot. Each of these is a small part of the total loss, but there are trillions of such tokens, and the loss rewards every one. [Speculation: that this explains the breadth of LLM abilities is a common view, not a theorem.]
Idea 2: a predictor is a compressor
Any probability model can be turned into a lossless compressor with arithmetic coding. The idea: start with the interval \([0, 1)\). For each token, split the current interval into pieces, one per vocabulary token, with sizes equal to the model’s probabilities, and keep the piece of the true token. After the whole text, the interval has width \(q(\mathbf{x}) = \prod_t q(x_t \mid x_{\lt t})\), exactly formula (31.2). Any number inside the interval identifies the text, and a binary number inside an interval of width \(w\) needs only about \(-\log_2 w\) bits. The precise bound is
- Read the symbols
- \(\ell(\mathbf{x})\)
- length of the compressed file in bits
- \(-\log_2 q(x_t \mid x_{\lt t})\)
- the CLM loss of token \(t\), measured in bits instead of nats
- \(\lceil \cdot \rceil\)
- round up to a whole number
So the training loss, summed over a text and converted to bits, is the size of that text compressed by the model, up to 2 bits. [Certain] Lowering the loss and compressing better are the same thing. The decoder needs the same model to undo it: it rebuilds the same intervals token by token. Figure 5 lets you watch the interval shrink.
How to read Figure 5
Step through with the bigram model. When the model is confident (“on” after “sat”, probability 0.8), the interval barely shrinks and the word costs only 0.32 bits. Switch to uniform: every word costs exactly 3 bits (\(\log_2 8\)), so the sentence costs 21 bits. The unigram model knows that “the” and “.” are common, but nothing about context, and saves almost nothing. The bigram model knows which word follows which, and stores the sentence in 7 bits. The last step shows an actual codeword: a binary number whose whole interval fits inside the final interval.
The bigram model gives the seven words the probabilities 0.7, 0.4, 0.5, 0.8, 0.85, 0.2 and 0.89. Their product is \(0.01695\), so \(-\log_2 q(\mathbf{x}) = 5.88\) bits, and the shortest codeword that fits in the final interval has 7 bits (bound 31.10: \(\lceil 5.88 \rceil + 1 = 7\)). The uniform model needs \(7 \times 3 = 21\) bits. The unigram model (probabilities 0.25, 0.1, 0.1, 0.1, 0.25, 0.1, 0.15) needs 20.03 bits, and its codeword has 21 bits. In nats per token, the CLM losses are 2.079 (uniform), 1.983 (unigram) and 0.583 (bigram). The model with the lowest loss is the best compressor, by exactly the same ratio.
Bits per byte: comparing models fairly
Two models with different tokenizers cannot be compared by loss per token: a tokenizer with longer tokens has fewer, harder tokens. The fix is to divide the total bits by the number of bytes of text, which does not depend on the tokenizer:
- Read the symbols
- \(\text{bpb}\)
- bits per byte: compressed size of the text in bits, divided by its raw size in bytes
- \(\mathcal{L}_{\text{CLM}}\)
- the loss in nats per token, formula (31.3)
- \(b\)
- average bytes per token for this tokenizer and this text
Raw text uses 8 bits per byte, so \(\text{bpb}/8\) is the compression ratio. Try it in Figure 6.
How to read Figure 6
Keep the loss at 2.0 and change the bytes per token. A tokenizer with longer tokens (larger \(b\)) gives a better bpb for the same per-token loss, because each token covers more text. That is why per-token losses of models with different tokenizers must never be compared directly. Raise the loss to 5 with \(b = 1\) (a byte-level model that has barely learned): the ratio is above 90%, almost no compression.
A model with loss 2.0 nats per token, on text with 4 bytes per token (a typical value for English with a modern tokenizer [Likely]): \(2.0 / \ln 2 = 2.885\) bits per token, and \(2.885 / 4 = 0.721\) bits per byte. The compression ratio is \(0.721 / 8 = 9.0\%\): one gigabyte of such text would be stored in about 90 MB.
The evidence: LLMs as general compressors
Delétang et al. (2023), “Language Modeling Is Compression”, used Chinchilla 70B, trained mainly on text, as an arithmetic coder. It compressed ImageNet image patches to 43.4% of their raw size (PNG: 58.5%) and LibriSpeech audio to 16.4% (FLAC: 30.3%). [Certain] A model trained to predict text had learned patterns general enough to compress images and sound better than tools built for them. Two cautions. First, these numbers do not count the size of the model itself; the decoder needs the 70B weights, so for a single file the model is far too big to be a practical compressor. [Certain: the paper discusses this.] Second, “compression is intelligence” is a slogan, not a theorem; it explains why the objective rewards general structure, not which abilities appear at which scale. [Speculation]
Beyond one token, and beyond text
Multi-token prediction
Next-token prediction gives a signal for one step ahead only. Multi-token prediction (MTP) adds extra output heads that predict the tokens 2, 3, … steps ahead from the same hidden state:
Here \(p^{(j)}\) is head \(j\) and \(n\) is the number of future tokens. Gloeckle et al. (2024) used \(n = 4\) independent heads on a shared trunk. Their 13B model solved 12% more HumanEval and 17% more MBPP problems than a comparable next-token model, and the extra heads can be used for self-speculative decoding, up to 3 times faster. [Certain] DeepSeek-V3 uses a sequential version with depth 1 (one extra token), with loss weight 0.3 for the first 10T training tokens and 0.1 for the last 4.8T. [Certain] After training, the extra heads can be dropped or used as a draft model (session 129).
Objectives in VLMs and video models
The same objectives appear when images and video enter the model:
- Captioning and VLM pretraining use the causal loss (31.3) on the text tokens only. The image tokens are part of the input (often a bidirectional prefix, as in PaliGemma) but get no loss (session 177).
- Masked image modeling is MLM for pixels: MAE hides 75% of image patches and rebuilds them; VideoMAE hides 90% to 95% of video tubes, because neighboring frames are so similar that a lower rate is too easy. [Certain] See session 165.
- Contrastive objectives (CLIP) are a different family: they do not predict tokens but match images to their captions (session 166).
Where this is used
| Application | Objective | Why this one |
|---|---|---|
| Chat LLMs and reasoning models (Llama, Qwen, DeepSeek) | Causal LM, sometimes plus MTP | Pretraining task = generation task; most signal per FLOP |
| Code completion in an editor | Causal LM + FIM (31.8) | The cursor is in the middle of a file; the model must use the code after it |
| Search embeddings, classification, named-entity tagging | Masked LM or replaced token detection, then fine-tuning | Both sides of every token help; encoders are small and fast |
| Translation, summarization, speech recognition | Span corruption or denoising (T5, BART); Whisper is trained directly on audio-to-text pairs | Clear input and output; the encoder reads the input once |
| VLM question answering and captioning (PaliGemma) | Prefix LM: image + prompt as prefix, loss on the answer | The image and question are read with full attention; the answer is generated |
| Video captioning and key-moment description | Causal LM on text, conditioned on frame tokens | Same loss as text LLMs; frames are input only |
| Video encoders for action recognition, retrieval and fingerprinting | Masked video modeling (VideoMAE) or contrastive video–text | No labels needed; high masking rate fits redundant frames |
| Comparing tokenizers or data mixtures | Bits per byte (31.11) on held-out text | Fair across tokenizers; equals compressed size |
Common mistakes
- Off-by-one in the shift. Logits at position \(t\) must be scored against the token at \(t+1\). Scoring against \(x_t\) itself gives a loss near zero and a useless model. In Hugging Face models, pass
labels = input_ids; the model shifts internally. Shifting again yourself predicts two tokens ahead. - Using a bidirectional mask with the causal loss. Each token then sees its own target; the loss collapses toward 0 in a few hundred steps. A suspiciously fast-falling loss is the symptom.
- Computing loss on tokens that should be ignored. Padding, image tokens in a VLM, the prefix in prefix LM, or the prompt in SFT must have label \(-100\) (the ignore index), or the model learns to predict padding.
- Comparing perplexities across tokenizers. Per-token loss depends on token length. Use bits per byte (31.11).
- Thinking BERT’s 15% are all [MASK]. Only 80% of them are; and the 10% unchanged tokens still get a loss.
- Static masking for many epochs. If the same positions are masked every epoch, the model sees the same test again and again. Draw masks on the fly (RoBERTa’s dynamic masking).
- FIM split after tokenization. Cutting between tokens means the model never sees a middle that starts inside a token, but a real cursor can be anywhere. The FIM paper cuts at the character level.
- Averaging the loss over the wrong count. Mean over target tokens, not over all positions, and be careful when batches have different numbers of targets (see gradient accumulation in session 32).
Interview questions
Write the causal language modeling loss and explain each part.basic
Why does BERT replace only 80% of the selected tokens with [MASK]?core
Why can’t BERT generate text well?basic
Compare MLM and CLM on training efficiency.core
What is a prefix LM, and where is it used?core
How does fill-in-the-middle work, and does it hurt normal generation?core
Explain “language modeling is compression”.core
Model A has perplexity 8 and model B has perplexity 10 on the same text. Is A better?advanced
Why did decoder-only causal LM become the default for LLMs?advanced
What is multi-token prediction and why would you use it?advanced
In a VLM, which tokens get the language-modeling loss?basic
Exercises
- A causal model gives the true tokens of a 5-token text the probabilities 0.9, 0.5, 0.5, 0.2 and 0.9. Compute the loss in nats per token, the perplexity, and the compressed size in bits.
- With BERT’s rule on a 128-token sequence, how many tokens are selected, and how many [MASK] tokens do you expect in the input?
- In a prefix LM with \(T = 512\) and a 200-token prefix, how many targets are there? How many query–key pairs are allowed by the mask? (Hint: the prefix block plus a triangle.)
- Write the PSM training string for the code
def add(a, b): return a + bwith middlereturn a + b. - A byte-level model (\(b = 1\)) has loss 0.8 nats per byte. A BPE model has loss 3.0 nats per token with \(b = 4.2\). Which compresses better?
- In Figure 5, which single word costs the bigram model the most bits, and why?
Answerscheck
[PRE]def add(a, b): [SUF][MID]return a + b[EOT] (the suffix is empty here). 5. Byte model: \(0.8/\ln 2 = 1.154\) bpb. BPE: \(3.0/(\ln 2 \times 4.2) = 1.031\) bpb. The BPE model compresses better, although its per-token loss is much higher. 6. “mat” after “the”: probability 0.2, so \(-\log_2 0.2 = 2.32\) bits; the model expected “cat” or “dog” more.Key takeaways
- A pretraining objective is a view that hides information, a set of targets, and a mask (31.1). Get the mask wrong and the model sees its answers.
- Causal LM (31.2, 31.3) uses every token as a target, is an exact probability model, and matches how the model is used. It is the default for LLMs and VLMs.
- Masked LM (31.4) sees both sides but predicts only 15% of tokens, independently; the 80/10/10 rule fights the [MASK] mismatch. ELECTRA (31.5) puts a loss on every token.
- Span corruption (31.6), prefix LM (31.7), UL2 mixtures and FIM (31.8) are ways to get bidirectional context while still generating.
- Expected loss = entropy + KL (31.9), and the summed loss in bits is the compressed size (31.10). Lower loss means better compression, which rewards modeling the real structure of text. Compare models with bits per byte (31.11).
- Multi-token prediction (31.12) adds future-token heads; VLMs use the causal loss on text tokens only.
Key formulas
References for this session
- Brown et al. (2020). Language Models are Few-Shot Learners
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 7: Pre-training
- Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Tay et al. (2022). UL2: Unifying Language Learning Paradigms
- Dong, Yang, Wang, Wei, Liu, Wang, Gao, Zhou, Hon (2019). Unified Language Model Pre-training for Natural Language Understanding and Generation
- Wang, Roberts, Hesslow, Le Scao, Chung, Beltagy, Launay, Raffel (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
- Clark, Luong, Le, Manning (2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
- Yang et al. (2019). XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Bavarian et al. (2022). Efficient Training of Language Models to Fill in the Middle
- Delétang et al. (2023). Language Modeling Is Compression
- Gloeckle et al. (2024). Better & Faster Large Language Models via Multi-token Prediction
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report
- Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
- He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
- Tong et al. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners