LLM & VLM courseBlog
31Track 3: Pretraining and scale · session 1 of 22

Pretraining objectives

Raw text has no labels, so pretraining must invent a task from the text itself. This session compares the tasks that were used at scale: next-token prediction (GPT), masked tokens (BERT, ELECTRA), span corruption (T5), prefix LM, UL2 mixtures and fill-in-the-middle. It ends with the reason the simplest one won: predicting the next token is the same as compressing text.

Time: 3 hours (reading 2 hours, exercises 1 hour)Level: BeginnerBefore this: Session 3 (cross-entropy and KL), session 18 (masks) and session 23 (the three transformer families)

What you will learn

  • The general recipe of self-supervised pretraining: a hidden view, a set of targets and an attention mask
  • The causal LM loss from the chain rule, the shift by one, and why every token is a target
  • Masked LM in BERT: the 15% selection, the 80/10/10 rule and why it exists, and what MLM cannot do; ELECTRA's replaced token detection
  • Span corruption, prefix LM, the UL2 mixture of denoisers and fill-in-the-middle for code
  • Why decoder-only next-token prediction became the default, with the evidence from Wang et al. (2022)
  • The compression view: loss = entropy + KL, arithmetic coding, and bits per byte as a fair metric
  • Multi-token prediction, and which tokens get a loss in VLMs and video models

Why this matters

A language model is pretrained on trillions of tokens of raw text, and nobody labels that text. So the first design question is: what task do we make out of raw text? That task is the pretraining objective. It decides what the model learns, what it can do afterwards, and how much it learns from each token of compute.

Over the last eight years, four families of objectives were tried at scale:

  • Causal language modeling (GPT, Llama, Qwen, DeepSeek): predict the next token from the tokens before it.
  • Masked language modeling (BERT, RoBERTa): hide some tokens and predict them from both sides.
  • Span corruption (T5) and denoising (BART): damage the input, and let a decoder rebuild the missing parts.
  • Prefix language modeling (UniLM, PaliGemma, part of UL2): read a prefix with full attention, then predict the rest left to right.

Today almost every large LLM and VLM is pretrained with the first one, the simplest. [Certain] GPT-3 (175B parameters, 300B training tokens) showed that a model trained only to predict the next token can translate, answer questions and do new tasks from a few examples in the prompt. [Certain] The last part of this session asks the deep question behind that result: why does such a simple objective learn so much? The answer is the link between prediction and compression.

This session builds on session 18 (which tokens can see which), session 23 (the three model shapes) and session 24 (T5 span corruption in detail). Here we compare the objectives side by side, add the ones those sessions skipped (ELECTRA, UL2, fill-in-the-middle, multi-token prediction), and study the compression view. The next session, session 32, follows one batch of the causal objective through the forward pass, the backward pass and the update.

What a pretraining objective is

All the objectives in this session follow one recipe, called self-supervised learning: the labels come from the data itself. Take a text \(\mathbf{x} = (x_1, \dots, x_T)\). Make a view \(\tilde{\mathbf{x}}\) of it that hides some information: cut off the future, put [MASK] tokens in, or delete spans. Then train the model to predict the hidden tokens from the view. The loss is the usual cross-entropy (session 3):

\[ \mathcal{L}(\theta) = \E_{\mathbf{x} \sim \text{data}}\ \E_{\tilde{\mathbf{x}},\, \mathcal{T}}\left[ -\frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \log p_\theta\big(x_t \mid \tilde{\mathbf{x}}\big) \right] \tag{31.1} \]
Read the symbols
\(\mathbf{x}\)
one training text, as token IDs
\(\tilde{\mathbf{x}}\)
what the model is allowed to see when it predicts (the “view”); it depends on the objective
\(\mathcal{T}\)
the set of target positions: the tokens that get a loss
\(p_\theta(x_t \mid \tilde{\mathbf{x}})\)
the probability the model gives to the true token at position \(t\), from a softmax over the vocabulary
\(\E\)
average over texts and, for random objectives, over the random choice of hidden tokens

So an objective is fixed by three choices: (1) what is hidden, (2) which positions get a loss, and (3) which attention mask the model uses so that it cannot see the answer. The mask is the part people forget. If a token can attend to its own target, the loss drops to almost zero and the model learns nothing.

Figure 1 shows the same 10-token sentence under five objectives. Look at three things for each: the model input, the targets, and the attention pattern.

Model input
Targets (tokens that get a loss)
may attendblocked[MASK]sentinel or FIM markertarget
Figure 1. One sentence, five objectives. The grid shows which tokens each position may look at (rows look at columns). For span corruption there are two grids: the encoder over the corrupted input and the decoder over the target.
How to read Figure 1

Causal LM has the most targets (9 of 10) but each prediction sees only the left side. Masked LM sees both sides, but only 2 of 10 tokens are targets. Span corruption shortens both the input and the target, because each hidden span becomes one sentinel token. In prefix LM, move the slider: the top-left block (the prefix) becomes fully visible, and the number of targets falls to \(10 - k\). Fill in the middle is plain causal LM on a reordered text: the model reads prefix and suffix first, so when it writes the middle it has seen both sides.

Causal language modeling: predict the next token

The chain rule of probability writes the probability of any text as a product of next-token probabilities. It is exact; no assumption is made:

\[ p_\theta(x_1, \dots, x_T) = \prod_{t=1}^{T} p_\theta\big(x_t \mid x_{\lt t}\big) \tag{31.2} \]

Here \(x_{\lt t}\) means all tokens before position \(t\) (for \(t = 1\), just a begin-of-sequence token). Taking \(-\log\) turns the product into a sum, and averaging over positions gives the causal language modeling (CLM) loss, also called the next-token loss:

\[ \mathcal{L}_{\text{CLM}} = -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta\big(x_t \mid x_{\lt t}\big) \tag{31.3} \]
Read the symbols
\(T\)
number of target tokens in the sequence
\(x_{\lt t}\)
the past: what the causal mask lets position \(t\) see
\(\mathcal{L}_{\text{CLM}}\)
average negative log-likelihood in nats per token; \(e^{\mathcal{L}}\) is the perplexity

Three properties make this objective special:

  1. Every token is a target. In formula (31.1), \(\mathcal{T}\) is all positions. With the causal mask (session 18), one forward pass computes all \(T\) predictions at once. In code, this is the famous “shift by one”: the logits at positions \(1..T-1\) are scored against the tokens at positions \(2..T\).
  2. It is a real probability model. Formula (31.2) gives the exact probability of any text. You can score texts, compare models by perplexity, and sample.
  3. Training and use are the same task. Generation is just predicting the next token again and again (session 78). Nothing has to be changed after pretraining, and the KV cache works because the past never changes.
Worked example: the CLM loss for four tokens

A model gives the true next tokens the probabilities 0.5, 0.25, 0.8 and 0.1. The per-token losses are \(-\ln 0.5 = 0.693\), \(-\ln 0.25 = 1.386\), \(-\ln 0.8 = 0.223\) and \(-\ln 0.1 = 2.303\) nats. The mean is \(4.605 / 4 = 1.151\) nats per token. Perplexity is \(e^{1.151} = 3.16\): on average the model is as unsure as if it chose among about 3 equally likely tokens. In bits, \(1.151 / \ln 2 = 1.661\) bits per token. Keep that last number in mind; it comes back in the compression section.

The weakness: each prediction sees only one side. For a task like “is this word a person’s name?”, the words to the right help a lot, and a causal model does not see them at that position. This is the motivation for masked language modeling.

Masked language modeling: use both sides

BERT (Devlin et al., 2018) removed the causal mask: every token sees every other token. Now a token cannot be its own target (it would just copy itself), so BERT hides some tokens and predicts only those. The masked language modeling (MLM) loss is

\[ \mathcal{L}_{\text{MLM}} = -\frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \log p_\theta\big(x_t \mid \tilde{\mathbf{x}}\big) \tag{31.4} \]
Read the symbols
\(\mathcal{M}\)
the randomly selected positions (15% of tokens in BERT)
\(\tilde{\mathbf{x}}\)
the whole sequence after corruption; position \(t\) sees all of it, left and right

The 15% and the 80/10/10 rule

BERT selects 15% of the tokens. Of the selected tokens, 80% are replaced by [MASK], 10% by a random token, and 10% are left unchanged; all selected tokens are predicted. [Certain] The paper’s example: if “hairy” in “my dog is hairy” is selected, the input becomes “my dog is [MASK]” 80% of the time, “my dog is apple” 10% of the time, and stays “my dog is hairy” 10% of the time.

Why not always use [MASK]? Because [MASK] never appears when the model is later fine-tuned or used. If the model only ever had to predict at [MASK] positions, it could put little effort into the vectors of normal tokens. With random and unchanged tokens mixed in, the model never knows which visible token might be a target or might be wrong, so it must build a good vector for every token. [Certain: this is the paper’s reason.]

Worked example: what BERT predicts in one 512-token sequence

Selected: \(\operatorname{round}(0.15 \times 512) = 77\) tokens. On average \(0.8 \times 77 = 61.6\) become [MASK], \(7.7\) become random tokens, and \(7.7\) stay the same. So only \(0.8 \times 0.15 = 12\%\) of the input tokens are [MASK]. A causal LM on the same 512 tokens gets 511 or 512 targets, about 6.6 times more training signal per forward pass. (Each MLM prediction uses both sides, so one MLM target may teach more than one CLM target; the ratio is not the whole story. [Likely])

Input the model sees
Original text (targets marked)
selected, replaced by [MASK] (80%)selected, replaced by a random token (10%)selected, left unchanged (10%)target in the original text
  1. \(n \leftarrow \operatorname{round}(\text{rate} \times N)\) // number of targets
  2. \(\mathcal{M} \leftarrow\) first \(n\) positions of a random shuffle
  3. for \(t \in \mathcal{M}\), left to right:
  4. draw \(u \sim U(0, 1)\)
  5. if \(u \lt 0.8\): \(\tilde{x}_t \leftarrow\) [MASK]
  6. else if \(u \lt 0.9\): \(\tilde{x}_t \leftarrow\) a random word
  7. else: keep \(\tilde{x}_t = x_t\) // still a target
  8. labels: \(y_t = x_t\) for \(t \in \mathcal{M}\), \(-100\) elsewhere // loss (31.4)
Figure 2. BERT’s corruption on a 26-token text, step by step. The algorithm box lights up the line being run, and the number line shows where the random number \(u\) falls for the current token. Positions and \(u\) values come from a seeded random generator, so the same draw number always gives the same result.
How to read Figure 2

Press Play. The first step picks the targets (outlined in blue in the original text). Then each selected token gets its own random number \(u\): left of 0.8 it becomes [MASK], between 0.8 and 0.9 a random word, right of 0.9 it stays as it is. Press “New random draw” several times. The number of selected tokens is fixed by the rate, but the split between [MASK], random and unchanged changes from draw to draw; only on average is it 80/10/10. Notice the green “unchanged” tokens: in the input they look normal, yet they are targets. That is why the model must check every token. Raise the rate to 40%: more targets per sequence, but less context is left to predict from.

What MLM cannot do

  • It cannot generate text well. It was never trained to write left to right, and the [MASK] tokens it predicts are predicted independently of each other. If “New [MASK] [MASK]” could be “New York City” or “New Delhi, India”, the model may output “New York India”.
  • It is not a full probability model. There is no exact formula like (31.2) for \(p(\mathbf{x})\), so you cannot get a perplexity for a text in one pass.
  • Few targets. Only 15% of positions learn per step, as computed above.

Fixes inside the encoder family

RoBERTa (2019) kept MLM but drew a new mask every time a sequence was seen (dynamic masking), dropped BERT’s second task (next-sentence prediction), and trained longer on more data. [Certain] XLNet (2019) tried to get both sides and a real probability model by training on random orders of the factorization (31.2) instead of left to right. [Certain] ELECTRA (2020) fixed the few-targets problem. A small MLM “generator” fills the masked positions with plausible tokens; the main model, the discriminator, then says for every token whether it is original or replaced. This is called replaced token detection:

\[ \mathcal{L}_{\text{ELECTRA}} = \mathcal{L}_{\text{MLM}}(\theta_G) + \lambda\, \mathcal{L}_{\text{disc}}(\theta_D), \qquad \mathcal{L}_{\text{disc}} = -\sum_{t=1}^{T} \Big[ \mathbb{1}(\tilde{x}_t = x_t) \log D_t + \mathbb{1}(\tilde{x}_t \ne x_t) \log (1 - D_t) \Big] \tag{31.5} \]
Read the symbols
\(\theta_G, \theta_D\)
weights of the small generator and of the main discriminator
\(D_t\)
the discriminator’s probability that token \(t\) is the original
\(\mathbb{1}(\cdot)\)
1 if the condition is true, else 0
\(\lambda\)
weight of the discriminator loss; ELECTRA uses 50 [Certain]

The discriminator loss covers all \(T\) positions, not just the 15%. ELECTRA-Small, trained on one GPU for 4 days, beat GPT on the GLUE benchmark although GPT used about 30 times more compute, and ELECTRA-Large matched RoBERTa and XLNet with less than a quarter of their compute. [Certain: from the paper’s abstract.] More on encoder models in session 33.

Span corruption and prefix LM

Span corruption (T5)

T5 drops about 15% of the tokens in short runs (mean length 3), replaces each run by one sentinel token, and trains a decoder to output only the dropped runs, each introduced by its sentinel. [Certain] The loss is the causal loss of the decoder on that short target, given the corrupted input \(\tilde{\mathbf{x}}\) read by the encoder:

\[ \mathcal{L}_{\text{span}} = -\frac{1}{|\mathbf{y}|} \sum_{t=1}^{|\mathbf{y}|} \log p_\theta\big(y_t \mid y_{\lt t},\ \tilde{\mathbf{x}}\big) \tag{31.6} \]

Unlike MLM, the hidden tokens of one span are predicted one after another, so they depend on each other, and the model learns to write. Unlike CLM, the encoder sees both sides of every hole. The cost: only the target tokens are predicted, and the model has two stacks. The full mechanics, with sentinel IDs and sequence lengths, are in session 24. BART is the same idea with other kinds of damage (deleting, shuffling, filling spans) and a decoder that writes the whole original text. [Certain]

Prefix LM

A prefix LM is a single decoder stack with a mixed mask: the first \(k\) tokens (the prefix) see each other fully, and the rest is causal. Only the tokens after the prefix get a loss:

\[ \mathcal{L}_{\text{PLM}} = -\frac{1}{T - k} \sum_{t=k+1}^{T} \log p_\theta\big(x_t \mid x_{1..k},\ x_{k+1..t-1}\big) \tag{31.7} \]

It is a middle point: the input part (an instruction, a document to summarize, or the image tokens of a VLM) gets bidirectional attention, as in an encoder, and the output part is generated, as in a decoder. UniLM used it in 2019, and PaliGemma uses it today: the image and the text prompt form the prefix, and the answer is the causal part. [Certain] Move the slider in Figure 1 to see how \(k\) changes the mask and the number of targets.

Mixing objectives: UL2

Different objectives teach different skills, so why not mix them? UL2 (Tay et al., 2022) trains one model on a mixture of 7 “denoisers”, with roughly equal shares [Certain]:

  • R-denoisers (regular): T5 span corruption, 15% of tokens, mean span 3 or 8.
  • S-denoiser (sequential): prefix LM, where the target is a single span at the end, about 25% of the text.
  • X-denoisers (extreme): long spans (mean 64) or a high rate (50%), so the model must write a lot from little context.

A special token at the start of the input tells the model which mode it is in, and the same token can be used later to switch modes. The paper writes them [R], [S], [X]; the released checkpoints use [NLU], [NLG] and [S2S]. [Likely]

Fill in the middle (FIM)

Code editors need infilling: write the missing body of a function, given the code before and after the cursor. A causal model sees only the left side. Bavarian et al. (2022) found a trick that keeps the causal objective unchanged: cut a document into prefix \(\mathbf{p}\), middle \(\mathbf{m}\) and suffix \(\mathbf{s}\), then move the middle to the end, with three marker tokens:

\[ (\mathbf{p}, \mathbf{m}, \mathbf{s}) \ \longrightarrow\ \texttt{[PRE]}\ \mathbf{p}\ \texttt{[SUF]}\ \mathbf{s}\ \texttt{[MID]}\ \mathbf{m}\ \texttt{[EOT]} \tag{31.8} \]

Then train with the normal loss (31.3). When the model writes \(\mathbf{m}\), it has already read both \(\mathbf{p}\) and \(\mathbf{s}\). At inference you give it \(\texttt{[PRE]}\,\mathbf{p}\,\texttt{[SUF]}\,\mathbf{s}\,\texttt{[MID]}\) and let it generate until \(\texttt{[EOT]}\). This order is called PSM (prefix, suffix, middle); SPM puts the suffix first. The paper transforms 50% of documents in its main models, says that 50% to 90% is a reasonable choice, cuts at the character level before tokenization, and finds that left-to-right quality does not drop, which it calls “FIM for free”. [Certain] Most code LLMs are trained this way; StarCoder, for example, writes the markers as <fim_prefix>, <fim_suffix> and <fim_middle>. [Likely]

Building the training examples

The objectives differ mostly in how a raw token sequence becomes (input, targets, mask). Algorithm 1 writes all of them in one place.

Algorithm 1: from a token sequence to a training example, for each objective
  1. input: tokens \(x_1, \dots, x_T\), objective, random generator
  2. if causal LM: input \(\leftarrow x_{1..T-1}\); targets \(\leftarrow x_{2..T}\); mask \(\leftarrow\) lower triangle // formula (31.3)
  3. if masked LM: choose \(\mathcal{M}\), 15% of positions; for each \(t \in \mathcal{M}\) draw \(u \sim U(0,1)\):
  4.    \(u \lt 0.8\): \(\tilde{x}_t \leftarrow\) [MASK]; \(\;0.8 \le u \lt 0.9\): \(\tilde{x}_t \leftarrow\) random token; else keep \(x_t\)
  5.    targets \(\leftarrow x_t\) for \(t \in \mathcal{M}\), ignore elsewhere; mask \(\leftarrow\) all ones // formula (31.4)
  6. if span corruption: pick spans (15%, mean length 3); encoder input \(\leftarrow\) kept tokens with one sentinel per span; decoder target \(\leftarrow\) sentinel, span, sentinel, span, …, end // formula (31.6)
  7. if prefix LM: choose \(k\); mask \(\leftarrow\) lower triangle with the top-left \(k \times k\) block set to ones; targets only for \(t \gt k\) // formula (31.7)
  8. if FIM (with probability 0.5): cut at two random character positions; reorder as in (31.8); then treat as causal LM
  9. return input, targets (with ignore index), mask

The loss side is the same for every objective: cross-entropy over the positions whose target is not “ignore”. This is why one training loop can serve all of them.

Algorithm 2: one loss computation, for any objective
  1. input: model, input \(\tilde{\mathbf{x}}\), targets \(\mathbf{y}\) (value \(-100\) means “no loss here”), mask \(A\)
  2. \(Z \leftarrow \text{model}(\tilde{\mathbf{x}}, A)\) // logits, shape (B, T, V)
  3. \(\ell_t \leftarrow -\log \softmax(Z_t)[y_t]\) for every \(t\) with \(y_t \ne -100\)
  4. \(\mathcal{L} \leftarrow\) mean of \(\ell_t\) over those positions // formula (31.1)
  5. return \(\mathcal{L}\)

Figure 3 runs Algorithm 2 on the sentence from Figure 1, for four objectives. The “model” is a toy that you can check by hand: bigram counts from six short football sentences, with add-0.1 smoothing over a 16-word vocabulary, so \(P(w \mid a) = (\text{count}(a, w) + 0.1) / (\text{count}(a) + 1.6)\). Its logit for a candidate word \(w\) is \(z[w] = \ln P(w \mid \text{left word})\). When the mask also lets it see the right side (masked LM), it adds \(\ln P(\text{right word} \mid w)\). Then a softmax turns the logits into probabilities, as in a real model. The loop visits every position: it skips positions whose label is \(-100\) and adds \(-\ln p\) for the others.

position being scoredtokens it may see[MASK]sentineltarget and its probability
  1. \(\tilde{\mathbf{x}}, \mathbf{y}, A \leftarrow\) make the example // Algorithm 1
  2. \(S \leftarrow 0,\ n \leftarrow 0\)
  3. for each position \(t\):
  4. if \(y_t = -100\): skip // no loss here
  5. \(p \leftarrow \softmax(z_t)\) // \(z_t\) = model(\(\tilde{\mathbf{x}}, A\)) at \(t\)
  6. \(\ell_t \leftarrow -\ln p[y_t]\)
  7. \(S \leftarrow S + \ell_t,\ n \leftarrow n + 1\)
  8. return \(\mathcal{L} = S / n\) // formula (31.1)
Figure 3. Algorithm 2 running on one sentence. Top row: the model input \(\tilde{\mathbf{x}}\). Middle row: the label \(y_t\) at each position (\(-100\) means no loss). Bottom row: the loss \(\ell_t = -\ln p[y_t]\), filled in as the loop reaches each position. The bars at the bottom are the model’s top four candidates at the current position. In masked LM mode, click a word in the top row to mask or unmask it.
How to read Figure 3

Press Play with causal LM. The label row is the input row shifted one place to the left: position \(t\) is scored against \(x_{t+1}\), and the last position has no label. The orange box is the position being scored; the tinted tokens are the ones the mask lets it see. Now switch to masked LM. Only the two [MASK] positions have labels, and each one now sees its right neighbor too. Compare “saves”: causal LM gives it \(p = 0.457\) (loss 0.784, it sees only “keeper”), masked LM gives \(p = 0.710\) (loss 0.342, it also sees “the”). The causal loss is 0.832 nats over 9 targets; the masked loss is 0.390 over only 2. A lower number from far fewer targets is not a better model. In span corruption, the encoder row gets no loss at all; the decoder writes only the dropped words and the sentinels. In this toy, the sentinel that comes next always gets \(p = 0.9\); this is a fixed rule of the toy, not a measured number. In prefix LM, move \(k\): the positions inside the prefix turn into \(-100\), and the mean is taken over \(10 - k\) targets only.

The PyTorch version. One tiny transformer block, four objectives, each made by changing only the mask and the targets. It runs on a CPU in about a second and prints the same numbers every time.

import math, torch
import torch.nn.functional as F
torch.manual_seed(0)

V, D, T = 1000, 64, 16                              # vocab, width, sequence length
MASK, PRE, SUF, MID, EOT = 996, 997, 998, 999, 0    # toy special-token ids

class TinyLM(torch.nn.Module):
    """One transformer block; the mask decides causal, bidirectional or prefix."""
    def __init__(self):
        super().__init__()
        self.emb = torch.nn.Embedding(V, D)
        self.pos = torch.nn.Embedding(64, D)
        self.block = torch.nn.TransformerEncoderLayer(D, 4, 4 * D, dropout=0.0, batch_first=True)
        self.head = torch.nn.Linear(D, V)
    def forward(self, x, allowed):                # allowed[i, j] = True: i may look at j
        h = self.emb(x) + self.pos(torch.arange(x.shape[1]))
        h = self.block(h, src_mask=~allowed)      # PyTorch: True in src_mask means blocked
        return self.head(h)                       # logits (B, T, V)

def causal(T):  return torch.tril(torch.ones(T, T, dtype=torch.bool))
def full(T):    return torch.ones(T, T, dtype=torch.bool)
def prefix(T, k):                                 # bidirectional on the first k tokens
    a = causal(T).clone(); a[:k, :k] = True; return a

model = TinyLM().eval()
x = torch.randint(1, 996, (2, T))                 # a batch of 2 "documents"

# 1) causal LM, formula (31.3): every position predicts the next token
logits = model(x, causal(T))
loss_clm = F.cross_entropy(logits[:, :-1].reshape(-1, V), x[:, 1:].reshape(-1))
print("causal LM loss", round(loss_clm.item(), 3), "targets", x[:, 1:].numel(), "log V", round(math.log(V), 3))

# 2) masked LM, formula (31.4), with BERT's 80/10/10 rule
def bert_mask(x, rate=0.15, g=None):
    sel = torch.rand(x.shape, generator=g) < rate            # positions to predict
    u = torch.rand(x.shape, generator=g)
    inp = x.clone()
    inp[sel & (u < 0.8)] = MASK                              # 80%: [MASK]
    rnd = sel & (u >= 0.8) & (u < 0.9)                       # 10%: random token
    inp[rnd] = torch.randint(1, 996, (int(rnd.sum()),), generator=g)
    labels = torch.where(sel, x, torch.full_like(x, -100))  # 10%: unchanged, still predicted
    return inp, labels
g = torch.Generator().manual_seed(1)
inp, labels = bert_mask(x, g=g)
logits = model(inp, full(T))                                 # no mask: both sides visible
loss_mlm = F.cross_entropy(logits.reshape(-1, V), labels.reshape(-1), ignore_index=-100)
print("masked LM loss", round(loss_mlm.item(), 3), "targets", int((labels != -100).sum()))

# 3) prefix LM, formula (31.7): bidirectional prefix of k tokens, loss only after it
k = 6
logits = model(x, prefix(T, k))
loss_plm = F.cross_entropy(logits[:, k - 1:-1].reshape(-1, V), x[:, k:].reshape(-1))
print("prefix LM loss", round(loss_plm.item(), 3), "targets", x[:, k:].numel())

# 4) fill in the middle, formula (31.8), PSM order: reorder, then plain causal LM
def fim_psm(doc, i, j):
    p, m, s = doc[:i], doc[i:j], doc[j:]
    return torch.cat([torch.tensor([PRE]), p, torch.tensor([SUF]), s, torch.tensor([MID]), m, torch.tensor([EOT])])
f = fim_psm(x[0], 5, 9)
logits = model(f[None], causal(len(f)))
loss_fim = F.cross_entropy(logits[0, :-1], f[1:])
print("FIM length", len(f), "loss", round(loss_fim.item(), 3))

Output (PyTorch 2.8, CPU): causal LM loss 7.142 targets 30 log V 6.908, masked LM loss 7.554 targets 2, prefix LM loss 7.031 targets 20, FIM length 20 loss 7.139. An untrained model is close to uniform, so every loss is near \(\ln 1000 = 6.908\). Look at the target counts: 30 for causal LM, but only 2 for masked LM on the same 32 tokens (15% is about 5 on average; this draw happened to pick 2). Note also logits[:, k-1:-1] in the prefix LM: position \(k-1\) (the last prefix token, counting from 0) predicts the first target. Getting this off by one is the most common bug.

Comparing the objectives, and why next-token won

ObjectiveSees both sides?Targets per 512 tokensExact \(p(\mathbf{x})\)?Generates?Used by
Causal LM (31.3)No511 to 512YesYesGPT, Llama, Qwen, DeepSeek, most VLMs
Masked LM (31.4)Yesabout 77NoPoorlyBERT, RoBERTa, ModernBERT
Replaced token detection (31.5)Yes512 (binary)NoNoELECTRA, DeBERTa-v3 [Likely]
Span corruption (31.6)Yes (encoder)about 104 (decoder)NoYesT5, mT5, Flan-T5
Prefix LM (31.7)In the prefix\(512 - k\)Of the part after the prefixYesUniLM, PaliGemma, UL2 (S mode)
FIM + causal (31.8)For the middle, yesall positions, as in causal LM (4 marker tokens added)YesYes, and infillsCode LLMs

(The 104 for span corruption is from session 23: 77 noise tokens in 26 spans, plus one sentinel per span and an end token.)

A careful study by Wang et al. (2022) trained many combinations of architecture and objective at the same compute. Right after pretraining, causal decoder-only models with the plain next-token objective had the best zero-shot results. After an extra stage of multitask fine-tuning, models trained with masked or span objectives (non-causal decoders and encoder–decoders) did best, and a causal decoder could be cheaply adapted into a non-causal one. [Certain] So “which objective is best” depends on how the model will be used. In practice, the field chose the causal decoder for several reasons together [Likely]:

  • Most signal per FLOP. Every token is a target (the 6.6× from the worked example).
  • One model, one task. The pretraining task is generation, so the model is ready for prompting, chat, tools and reasoning without a new head or a new mask.
  • Simple and fast inference. One stack, a causal mask and a KV cache.
  • Few-shot and in-context learning appeared with scale in GPT-3 and fit the left-to-right format naturally.
  • It is a full probability model, which brings perplexity, scaling laws (session 35), sampling and the compression view below.
Figure 4. A timeline of pretraining objectives used at scale. Top row: objectives for models that generate left to right. Bottom row: objectives for encoders and encoder–decoders. Years are the first arXiv or report version; the spacing between years is not to scale.

Why next-token prediction learns so much: the compression view

The next-token objective looks too simple to produce a model that writes code and explains physics. Two ideas explain why it is not.

Idea 1: the loss can only reach its minimum by modeling everything

Let \(p^\ast\) be the true distribution of text and \(q = p_\theta\) the model. The expected loss splits into two parts (session 3):

\[ \E_{\mathbf{x} \sim p^\ast}\big[ -\log q(\mathbf{x}) \big] = \underbrace{H(p^\ast)}_{\text{fixed by the data}} + \underbrace{\KL\big(p^\ast \,\Vert\, q\big)}_{\ge\, 0,\ \text{= 0 only if } q = p^\ast} \tag{31.9} \]
Read the symbols
\(H(p^\ast)\)
the entropy of real text: the part of the next token that nobody can predict (a random name, a coin flip in a story)
\(\KL(p^\ast \Vert q)\)
how far the model is from the truth; it is the only part training can reduce

So the loss keeps going down only as long as the model captures more of the real structure of text. And the structure of text is the structure of the world that wrote it. To predict the next token of “The capital of Australia is”, the model needs a fact. Of “17 × 3 =”, arithmetic. Of a Python function after its docstring, programming. Of a mystery novel’s last chapter, “the murderer is”, it needs to track the whole plot. Each of these is a small part of the total loss, but there are trillions of such tokens, and the loss rewards every one. [Speculation: that this explains the breadth of LLM abilities is a common view, not a theorem.]

Idea 2: a predictor is a compressor

Any probability model can be turned into a lossless compressor with arithmetic coding. The idea: start with the interval \([0, 1)\). For each token, split the current interval into pieces, one per vocabulary token, with sizes equal to the model’s probabilities, and keep the piece of the true token. After the whole text, the interval has width \(q(\mathbf{x}) = \prod_t q(x_t \mid x_{\lt t})\), exactly formula (31.2). Any number inside the interval identifies the text, and a binary number inside an interval of width \(w\) needs only about \(-\log_2 w\) bits. The precise bound is

\[ \ell(\mathbf{x}) \ \le\ \left\lceil -\log_2 q(\mathbf{x}) \right\rceil + 1 = \left\lceil \sum_{t=1}^{T} -\log_2 q\big(x_t \mid x_{\lt t}\big) \right\rceil + 1 \ \text{bits} \tag{31.10} \]
Read the symbols
\(\ell(\mathbf{x})\)
length of the compressed file in bits
\(-\log_2 q(x_t \mid x_{\lt t})\)
the CLM loss of token \(t\), measured in bits instead of nats
\(\lceil \cdot \rceil\)
round up to a whole number

So the training loss, summed over a text and converted to bits, is the size of that text compressed by the model, up to 2 bits. [Certain] Lowering the loss and compressing better are the same thing. The decoder needs the same model to undo it: it rebuilds the same intervals token by token. Figure 5 lets you watch the interval shrink.

Figure 5. Arithmetic coding of “the cat sat on the mat .” with an 8-word vocabulary and three toy models. Each step zooms into the current interval and splits it by the model’s probabilities for the next word; the true word’s piece (highlighted) becomes the new interval.
How to read Figure 5

Step through with the bigram model. When the model is confident (“on” after “sat”, probability 0.8), the interval barely shrinks and the word costs only 0.32 bits. Switch to uniform: every word costs exactly 3 bits (\(\log_2 8\)), so the sentence costs 21 bits. The unigram model knows that “the” and “.” are common, but nothing about context, and saves almost nothing. The bigram model knows which word follows which, and stores the sentence in 7 bits. The last step shows an actual codeword: a binary number whose whole interval fits inside the final interval.

Worked example: three models, one sentence

The bigram model gives the seven words the probabilities 0.7, 0.4, 0.5, 0.8, 0.85, 0.2 and 0.89. Their product is \(0.01695\), so \(-\log_2 q(\mathbf{x}) = 5.88\) bits, and the shortest codeword that fits in the final interval has 7 bits (bound 31.10: \(\lceil 5.88 \rceil + 1 = 7\)). The uniform model needs \(7 \times 3 = 21\) bits. The unigram model (probabilities 0.25, 0.1, 0.1, 0.1, 0.25, 0.1, 0.15) needs 20.03 bits, and its codeword has 21 bits. In nats per token, the CLM losses are 2.079 (uniform), 1.983 (unigram) and 0.583 (bigram). The model with the lowest loss is the best compressor, by exactly the same ratio.

Bits per byte: comparing models fairly

Two models with different tokenizers cannot be compared by loss per token: a tokenizer with longer tokens has fewer, harder tokens. The fix is to divide the total bits by the number of bytes of text, which does not depend on the tokenizer:

\[ \text{bpb} = \frac{\sum_t -\ln q(x_t \mid x_{\lt t})}{\ln 2 \cdot N_{\text{bytes}}} = \frac{\mathcal{L}_{\text{CLM}}}{\ln 2 \cdot b}, \qquad b = \frac{N_{\text{bytes}}}{N_{\text{tokens}}} \tag{31.11} \]
Read the symbols
\(\text{bpb}\)
bits per byte: compressed size of the text in bits, divided by its raw size in bytes
\(\mathcal{L}_{\text{CLM}}\)
the loss in nats per token, formula (31.3)
\(b\)
average bytes per token for this tokenizer and this text

Raw text uses 8 bits per byte, so \(\text{bpb}/8\) is the compression ratio. Try it in Figure 6.

Figure 6. From training loss to compressed size. The full bar is the raw text (8 bits per byte); the colored part is the size after arithmetic coding with the model.
How to read Figure 6

Keep the loss at 2.0 and change the bytes per token. A tokenizer with longer tokens (larger \(b\)) gives a better bpb for the same per-token loss, because each token covers more text. That is why per-token losses of models with different tokenizers must never be compared directly. Raise the loss to 5 with \(b = 1\) (a byte-level model that has barely learned): the ratio is above 90%, almost no compression.

Worked example: what a loss of 2.0 means

A model with loss 2.0 nats per token, on text with 4 bytes per token (a typical value for English with a modern tokenizer [Likely]): \(2.0 / \ln 2 = 2.885\) bits per token, and \(2.885 / 4 = 0.721\) bits per byte. The compression ratio is \(0.721 / 8 = 9.0\%\): one gigabyte of such text would be stored in about 90 MB.

The evidence: LLMs as general compressors

Delétang et al. (2023), “Language Modeling Is Compression”, used Chinchilla 70B, trained mainly on text, as an arithmetic coder. It compressed ImageNet image patches to 43.4% of their raw size (PNG: 58.5%) and LibriSpeech audio to 16.4% (FLAC: 30.3%). [Certain] A model trained to predict text had learned patterns general enough to compress images and sound better than tools built for them. Two cautions. First, these numbers do not count the size of the model itself; the decoder needs the 70B weights, so for a single file the model is far too big to be a practical compressor. [Certain: the paper discusses this.] Second, “compression is intelligence” is a slogan, not a theorem; it explains why the objective rewards general structure, not which abilities appear at which scale. [Speculation]

Beyond one token, and beyond text

Multi-token prediction

Next-token prediction gives a signal for one step ahead only. Multi-token prediction (MTP) adds extra output heads that predict the tokens 2, 3, … steps ahead from the same hidden state:

\[ \mathcal{L}_{\text{MTP}} = -\frac{1}{T} \sum_{t=1}^{T} \sum_{j=1}^{n} \log p_\theta^{(j)}\big(x_{t+j} \mid x_{\le t}\big) \tag{31.12} \]

Here \(p^{(j)}\) is head \(j\) and \(n\) is the number of future tokens. Gloeckle et al. (2024) used \(n = 4\) independent heads on a shared trunk. Their 13B model solved 12% more HumanEval and 17% more MBPP problems than a comparable next-token model, and the extra heads can be used for self-speculative decoding, up to 3 times faster. [Certain] DeepSeek-V3 uses a sequential version with depth 1 (one extra token), with loss weight 0.3 for the first 10T training tokens and 0.1 for the last 4.8T. [Certain] After training, the extra heads can be dropped or used as a draft model (session 129).

Objectives in VLMs and video models

The same objectives appear when images and video enter the model:

  • Captioning and VLM pretraining use the causal loss (31.3) on the text tokens only. The image tokens are part of the input (often a bidirectional prefix, as in PaliGemma) but get no loss (session 177).
  • Masked image modeling is MLM for pixels: MAE hides 75% of image patches and rebuilds them; VideoMAE hides 90% to 95% of video tubes, because neighboring frames are so similar that a lower rate is too easy. [Certain] See session 165.
  • Contrastive objectives (CLIP) are a different family: they do not predict tokens but match images to their captions (session 166).

Where this is used

ApplicationObjectiveWhy this one
Chat LLMs and reasoning models (Llama, Qwen, DeepSeek)Causal LM, sometimes plus MTPPretraining task = generation task; most signal per FLOP
Code completion in an editorCausal LM + FIM (31.8)The cursor is in the middle of a file; the model must use the code after it
Search embeddings, classification, named-entity taggingMasked LM or replaced token detection, then fine-tuningBoth sides of every token help; encoders are small and fast
Translation, summarization, speech recognitionSpan corruption or denoising (T5, BART); Whisper is trained directly on audio-to-text pairsClear input and output; the encoder reads the input once
VLM question answering and captioning (PaliGemma)Prefix LM: image + prompt as prefix, loss on the answerThe image and question are read with full attention; the answer is generated
Video captioning and key-moment descriptionCausal LM on text, conditioned on frame tokensSame loss as text LLMs; frames are input only
Video encoders for action recognition, retrieval and fingerprintingMasked video modeling (VideoMAE) or contrastive video–textNo labels needed; high masking rate fits redundant frames
Comparing tokenizers or data mixturesBits per byte (31.11) on held-out textFair across tokenizers; equals compressed size

Common mistakes

  • Off-by-one in the shift. Logits at position \(t\) must be scored against the token at \(t+1\). Scoring against \(x_t\) itself gives a loss near zero and a useless model. In Hugging Face models, pass labels = input_ids; the model shifts internally. Shifting again yourself predicts two tokens ahead.
  • Using a bidirectional mask with the causal loss. Each token then sees its own target; the loss collapses toward 0 in a few hundred steps. A suspiciously fast-falling loss is the symptom.
  • Computing loss on tokens that should be ignored. Padding, image tokens in a VLM, the prefix in prefix LM, or the prompt in SFT must have label \(-100\) (the ignore index), or the model learns to predict padding.
  • Comparing perplexities across tokenizers. Per-token loss depends on token length. Use bits per byte (31.11).
  • Thinking BERT’s 15% are all [MASK]. Only 80% of them are; and the 10% unchanged tokens still get a loss.
  • Static masking for many epochs. If the same positions are masked every epoch, the model sees the same test again and again. Draw masks on the fly (RoBERTa’s dynamic masking).
  • FIM split after tokenization. Cutting between tokens means the model never sees a middle that starts inside a token, but a real cursor can be anywhere. The FIM paper cuts at the character level.
  • Averaging the loss over the wrong count. Mean over target tokens, not over all positions, and be careful when batches have different numbers of targets (see gradient accumulation in session 32).

Interview questions

Write the causal language modeling loss and explain each part.basic
\(\mathcal{L} = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{\lt t})\). The chain rule writes \(p(\mathbf{x})\) as a product of next-token probabilities; the log turns it into a sum; the mean over positions gives nats per token. In practice it is cross-entropy between the logits at positions \(1..T-1\) and the tokens at \(2..T\), computed for all positions in one pass thanks to the causal mask. \(e^{\mathcal{L}}\) is the perplexity.
Why does BERT replace only 80% of the selected tokens with [MASK]?core
[MASK] never appears at fine-tuning time, so a model that only learned to predict at [MASK] positions would have a train–test mismatch and could neglect the vectors of ordinary tokens. With 10% random and 10% unchanged tokens, any visible token might be a target or might be wrong, so the model must build a good contextual vector for every token. The unchanged ones also bias the model toward the real observed word.
Why can’t BERT generate text well?basic
It was trained to fill a few holes given both sides, not to write left to right. Its predictions for several masked positions are independent given the input, so consecutive outputs need not agree (“New York India”). It also does not define an exact \(p(\mathbf{x})\), so there is no clean sampling procedure. You can sample iteratively (mask, fill, repeat), but quality is far below a causal model.
Compare MLM and CLM on training efficiency.core
Per 512-token sequence, MLM at 15% gives 77 targets; CLM gives 511–512, about 6.6× more. Each MLM target uses context from both sides, so it can be more informative, but in total CLM extracts more signal per FLOP. ELECTRA shows the gap is real: by giving a (binary) loss on every token, it matched RoBERTa with under a quarter of the compute.
What is a prefix LM, and where is it used?core
One decoder stack with a mask that is fully bidirectional on the first \(k\) tokens and causal after them; loss only on tokens after the prefix (formula 31.7). The input gets encoder-like attention, the output is generated. Used in UniLM, as the S-denoiser in UL2, and in VLMs such as PaliGemma, where image tokens and the prompt form the prefix. Cost: prefix tokens give no loss, and the KV cache of the prefix must be computed with the full mask before decoding.
How does fill-in-the-middle work, and does it hurt normal generation?core
Cut a document into prefix, middle and suffix (at character level), reorder to [PRE] prefix [SUF] suffix [MID] middle [EOT], and train with the ordinary causal loss. At inference, give prefix and suffix and generate after [MID] until [EOT]. Bavarian et al. transformed 50% of documents and found no loss in left-to-right ability (“FIM for free”), with 50–90% a reasonable range.
Explain “language modeling is compression”.core
With arithmetic coding, a model \(q\) compresses a text into at most \(\lceil -\log_2 q(\mathbf{x})\rceil + 1\) bits. \(-\log_2 q(\mathbf{x})\) is the summed CLM loss in bits. So minimizing the training loss is the same as minimizing the compressed size of the training data. Since expected loss = entropy + KL, the only way to keep lowering it is to model more real structure of the text. Delétang et al. showed Chinchilla 70B compresses ImageNet patches to 43.4% (PNG 58.5%) and LibriSpeech to 16.4% (FLAC 30.3%), not counting model size.
Model A has perplexity 8 and model B has perplexity 10 on the same text. Is A better?advanced
Only if they use the same tokenizer. Perplexity is per token; if B’s tokenizer has longer tokens (more bytes per token), B may compress the text better. Convert to bits per byte: \(\text{bpb} = \ln(\text{ppl}) / (\ln 2 \cdot b)\). For example A with \(b = 3\): \(\ln 8/(\ln 2 \cdot 3) = 1.0\); B with \(b = 4\): \(\ln 10/(\ln 2 \cdot 4) = 0.83\), so B is better. Also check for contamination.
Why did decoder-only causal LM become the default for LLMs?advanced
Every token is a target (most signal per FLOP); the training task equals the deployment task (generation), so prompting, chat and in-context learning need no new head; one stack with a causal mask is simple and has an efficient KV cache; it is an exact probability model, which gives perplexity and clean scaling laws. Wang et al. (2022) found causal decoders best for zero-shot after pure pretraining, while MLM-style models were better after multitask fine-tuning, so the choice also reflects how models are used: prompted directly, at very large scale.
What is multi-token prediction and why would you use it?advanced
Extra heads predict tokens \(t+2, t+3, \dots\) from the hidden state at \(t\) (formula 31.12). It gives a denser training signal that rewards planning ahead; Gloeckle et al. report gains on code benchmarks for large models (13B: +12% HumanEval, +17% MBPP) and up to 3× faster inference by using the heads as a draft for self-speculative decoding. DeepSeek-V3 uses one extra token with a small loss weight (0.3, then 0.1).
In a VLM, which tokens get the language-modeling loss?basic
Only text tokens that the model should produce: the caption or the answer. Image (or video frame) tokens are input only; their labels are set to the ignore index, because they are continuous features, not tokens from the vocabulary. The prompt is often masked out too during instruction tuning. The image and prompt may get bidirectional attention as a prefix (PaliGemma) or causal attention (most LLaVA-style models).

Exercises

  1. A causal model gives the true tokens of a 5-token text the probabilities 0.9, 0.5, 0.5, 0.2 and 0.9. Compute the loss in nats per token, the perplexity, and the compressed size in bits.
  2. With BERT’s rule on a 128-token sequence, how many tokens are selected, and how many [MASK] tokens do you expect in the input?
  3. In a prefix LM with \(T = 512\) and a 200-token prefix, how many targets are there? How many query–key pairs are allowed by the mask? (Hint: the prefix block plus a triangle.)
  4. Write the PSM training string for the code def add(a, b): return a + b with middle return a + b.
  5. A byte-level model (\(b = 1\)) has loss 0.8 nats per byte. A BPE model has loss 3.0 nats per token with \(b = 4.2\). Which compresses better?
  6. In Figure 5, which single word costs the bigram model the most bits, and why?
Answerscheck
1. Losses: 0.105, 0.693, 0.693, 1.609, 0.105; sum 3.206, mean 0.641 nats per token; perplexity \(e^{0.641} = 1.90\). Bits: \(3.206 / \ln 2 = 4.63\), so at most \(\lceil 4.63 \rceil + 1 = 6\) bits. 2. \(\operatorname{round}(0.15 \times 128) = 19\) selected; expected [MASK] \(0.8 \times 19 = 15.2\). 3. Targets \(512 - 200 = 312\). Allowed pairs: the prefix block \(200 \times 200 = 40{,}000\), plus each of the 312 later rows sees the 200 prefix tokens and the later tokens up to itself: \(312 \times 200 + 312 \times 313 / 2 = 62{,}400 + 48{,}828 = 111{,}228\); total 151,228 (of \(512^2 = 262{,}144\)). 4. [PRE]def add(a, b): [SUF][MID]return a + b[EOT] (the suffix is empty here). 5. Byte model: \(0.8/\ln 2 = 1.154\) bpb. BPE: \(3.0/(\ln 2 \times 4.2) = 1.031\) bpb. The BPE model compresses better, although its per-token loss is much higher. 6. “mat” after “the”: probability 0.2, so \(-\log_2 0.2 = 2.32\) bits; the model expected “cat” or “dog” more.

Key takeaways

  • A pretraining objective is a view that hides information, a set of targets, and a mask (31.1). Get the mask wrong and the model sees its answers.
  • Causal LM (31.2, 31.3) uses every token as a target, is an exact probability model, and matches how the model is used. It is the default for LLMs and VLMs.
  • Masked LM (31.4) sees both sides but predicts only 15% of tokens, independently; the 80/10/10 rule fights the [MASK] mismatch. ELECTRA (31.5) puts a loss on every token.
  • Span corruption (31.6), prefix LM (31.7), UL2 mixtures and FIM (31.8) are ways to get bidirectional context while still generating.
  • Expected loss = entropy + KL (31.9), and the summed loss in bits is the compressed size (31.10). Lower loss means better compression, which rewards modeling the real structure of text. Compare models with bits per byte (31.11).
  • Multi-token prediction (31.12) adds future-token heads; VLMs use the causal loss on text tokens only.

Key formulas

\[ \mathcal{L}_{\text{CLM}} = -\frac{1}{T}\sum_{t} \log p_\theta(x_t \mid x_{\lt t}), \qquad \mathcal{L}_{\text{MLM}} = -\frac{1}{|\mathcal{M}|}\sum_{t \in \mathcal{M}} \log p_\theta(x_t \mid \tilde{\mathbf{x}}) \]
\[ \E[-\log q] = H(p^\ast) + \KL(p^\ast \Vert q), \qquad \ell(\mathbf{x}) \le \lceil -\log_2 q(\mathbf{x}) \rceil + 1, \qquad \text{bpb} = \frac{\mathcal{L}_{\text{CLM}}}{\ln 2 \cdot b} \]

References for this session

  1. Brown et al. (2020). Language Models are Few-Shot Learners
  2. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  3. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  4. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 7: Pre-training
  5. Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach
  6. Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  7. Tay et al. (2022). UL2: Unifying Language Learning Paradigms
  8. Dong, Yang, Wang, Wei, Liu, Wang, Gao, Zhou, Hon (2019). Unified Language Model Pre-training for Natural Language Understanding and Generation
  9. Wang, Roberts, Hesslow, Le Scao, Chung, Beltagy, Launay, Raffel (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
  10. Clark, Luong, Le, Manning (2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
  11. Yang et al. (2019). XLNet: Generalized Autoregressive Pretraining for Language Understanding
  12. Bavarian et al. (2022). Efficient Training of Language Models to Fill in the Middle
  13. Delétang et al. (2023). Language Modeling Is Compression
  14. Gloeckle et al. (2024). Better & Faster Large Language Models via Multi-token Prediction
  15. DeepSeek-AI (2024). DeepSeek-V3 Technical Report
  16. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
  17. He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
  18. Tong et al. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
  19. Radford et al. (2019). Language Models are Unsupervised Multitask Learners