LLM & VLM courseBlog
24Track 2: Attention and transformers · session 9 of 15

Encoder–decoder in depth: T5 span corruption, sentinel tokens, BART denoising

An encoder–decoder reads the whole input with one stack of layers and writes the output with another, and the writer looks back at the reader at every step through cross-attention. This session opens up the two most important encoder–decoders, T5 and BART: how T5 turns every task into text in and text out, how span corruption with sentinel tokens gives short and cheap training targets, how BART learns by rebuilding damaged text, and where this design still beats decoder-only models today.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 7, 20 and 23 (seq2seq and attention, self- vs cross-attention, the three transformer families)

What you will learn

  • The text-to-text idea of T5: one model, one loss, every task written as an input string and an output string
  • The three sublayers of a decoder block, and exactly what cross-attention reads from the encoder
  • Span corruption with sentinel tokens: how inputs and targets are built, how long they are, and why short targets make pretraining cheaper
  • How the decoder uses the encoder at every step, in training (teacher forcing) and at inference (encode once, cache K and V)
  • BART’s noising functions (masking, deletion, infilling, sentence permutation, rotation) and why text infilling won
  • The cost of an L+L encoder–decoder compared with decoder-only models: parameters, FLOPs and attention masks
  • When encoder–decoders still win: translation, summarization, speech (Whisper), OCR and text encoders for image and video generation

Why this matters

Many useful tasks have the shape “here is an input, write an output”: translate this sentence, summarize this article, transcribe this audio, read the text in this image. The encoder–decoder transformer is built for exactly this shape. One stack of layers, the encoder, reads the whole input at once. A second stack, the decoder, writes the output one token at a time, and at every step it looks back at the encoder’s result through cross-attention.

In session 23 we met this design as one of three families, next to encoder-only (BERT) and decoder-only (GPT, Llama). This session opens it up, using its two most famous members:

  • T5 (Raffel et al., 2019) treats every task as text in, text out, and pretrains with span corruption: it cuts pieces out of the input, puts a placeholder called a sentinel token in each hole, and asks the decoder to write only the missing pieces.
  • BART (Lewis et al., 2019) damages text in several ways (masking, deleting, shuffling) and trains the decoder to rebuild the full original.

Why should you care when almost all new chat LLMs are decoder-only? Three reasons. First, encoder–decoders are still the main design for speech recognition (Whisper), much of machine translation (NLLB), OCR and document reading (Donut, TrOCR), and they are the text encoders inside many image and video generators (T5-XXL in Imagen and Stable Diffusion 3). Second, the ideas travel: sentinel tokens, denoising objectives and cross-attention appear again in VLMs, where a language model attends to image features, and in session 31. Third, interviewers love the comparison: “What does cross-attention read from the encoder?”, “Why are T5 targets so short?”, “Is an encoder–decoder with 12+12 layers more expensive than a 12-layer GPT?”. By the end you can answer each one with a formula and a number.

T5: every task is text-to-text

Before T5, each task had its own output layer: a classifier head for sentiment, a span-pointer head for question answering, a decoder for translation. T5 removed all of them. Every task is written as an input string and an output string, and one model with one loss learns them all. A short task prefix at the start of the input tells the model which task it is doing. These examples come from the paper [Certain]:

TaskInput stringTarget string
Translationtranslate English to German: That is good.Das ist gut.
Grammar check (CoLA)cola sentence: The course is jumping well.not acceptable
Similarity score (STS-B)stsb sentence1: The rhino grazed on the grass. sentence2: A rhino is grazing in a field.3.8
Summarizationsummarize: state authorities dispatched emergency crews tuesday to …six people hospitalized after a storm in attala county.

Even a number becomes text. STS-B scores go from 1 to 5, so T5 rounds them to the nearest multiple of 0.2 and writes them as strings like 3.8; this turns a regression task into a choice among 21 strings (1.0, 1.2, …, 5.0). [Certain] Classification labels become words such as acceptable or entailment. If the model writes a word that is not a label, the answer simply counts as wrong.

The model is trained with the same objective for every task: maximum likelihood of the target string, given the input string. Write the input tokens as \(\mathbf{x} = (x_1, \dots, x_{n})\) and the target tokens as \(\mathbf{y} = (y_1, \dots, y_{m})\):

\[ p_\theta(\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{m} p_\theta\big(y_t \mid y_{\lt t},\ \mathbf{x}\big), \qquad \mathcal{L} = -\sum_{t=1}^{m} \log p_\theta\big(y_t \mid y_{\lt t},\ \mathbf{x}\big) \tag{24.1} \]
Read the symbols
\(\mathbf{x}\)
the input tokens, including the task prefix; the encoder reads all of them at once
\(\mathbf{y}\)
the target tokens; the decoder writes them one by one, ending with an end-of-sequence token </s>
\(y_{\lt t}\)
the target tokens before position \(t\). The decoder may look at these, but not at \(y_t\) or later.
\(\theta\)
all the model’s weights: encoder, decoder and the shared embedding table

This is the same formula as the seq2seq models of session 7, formula (7.5), with one change that matters a lot: the condition is the full sequence of encoder outputs, not one squeezed vector. It is also the same next-token loss as a decoder-only LLM (session 6), except the loss is computed only on the target tokens, and the input is read by separate weights.

The architecture: two stacks and a bridge

Figure 1 shows the whole model. Read it from the bottom left to the top right.

  1. The input tokens are looked up in the embedding table (session 11) and pass through the encoder: \(N\) identical blocks, each with bidirectional self-attention (every input token can see every other input token) and a feed-forward network (FFN). The result is one vector per input token, \(H_{\text{enc}} \in \R^{n \times d}\). This is the encoder’s full output; nothing is squeezed into one vector.
  2. The decoder also has \(N\) blocks. Each block has three sublayers instead of two: causal self-attention over the target tokens written so far, then cross-attention to \(H_{\text{enc}}\), then an FFN.
  3. The last decoder vector at each position goes through the output layer (in T5 the same matrix as the input embedding table, see weight tying in session 11) and a softmax gives the next-token probabilities.
Figure 1. The encoder–decoder transformer, as used by T5 and BART. Left: the encoder reads the corrupted input once. Right: the decoder writes the target. The orange and green arrows carry the encoder output \(H_{\text{enc}}\); inside every decoder block it becomes the keys (orange) and values (green) of cross-attention, while the queries (blue) come from the decoder itself. The same \(H_{\text{enc}}\) goes to all \(N\) decoder blocks.

Cross-attention, precisely

Cross-attention is ordinary scaled dot-product attention (session 17) with one twist: the queries and the keys/values come from different sequences. Let \(S \in \R^{m \times d}\) be the decoder’s hidden states after its self-attention sublayer (one row per target position). Then:

\[ Q = S\,W_Q, \quad K = H_{\text{enc}}\,W_K, \quad V = H_{\text{enc}}\,W_V, \qquad \operatorname{CrossAttn}(S, H_{\text{enc}}) = \softmax\!\Big(\frac{Q K^\top}{\sqrt{d_k}}\Big)\, V \tag{24.2} \]
Read the symbols
\(S\)
decoder states, shape \(m \times d\): “what the writer needs now”. They give the queries.
\(H_{\text{enc}}\)
the final encoder output, shape \(n \times d\): “what the reader understood”. It gives the keys and values.
\(W_Q, W_K, W_V\)
learned projection matrices of this decoder block, each \(d \times d\) when all heads are put side by side. Each decoder block has its own.
\(QK^\top\)
an \(m \times n\) score matrix: target positions by source positions. It is not square, and it has no causal mask, because the whole input is known from the start.
\(d_k\)
the size of one head’s query and key vectors (64 in T5-Base)

Two facts follow from the shapes. Every row of the softmax is a probability distribution over the \(n\) input tokens: for each target position, the decoder decides where in the input to read. And \(K\) and \(V\) depend only on the encoder output, so during generation they are computed once per input and reused at every step. We will use this in the section on inference.

A full decoder block, in the pre-norm form that T5 uses (session 22), is three residual updates:

\[ \begin{aligned} S' &= S + \operatorname{SelfAttn}_{\text{causal}}\big(\operatorname{Norm}(S)\big) \\ S'' &= S' + \operatorname{CrossAttn}\big(\operatorname{Norm}(S'),\ H_{\text{enc}}\big) \\ S''' &= S'' + \operatorname{FFN}\big(\operatorname{Norm}(S'')\big) \end{aligned} \tag{24.3} \]

The encoder block is the same without the middle line, and with a self-attention that has no causal mask. Note that \(H_{\text{enc}}\) is the output of the last encoder block, already normalized; every decoder block reads the same \(H_{\text{enc}}\), not the encoder layer at the same depth. [Certain]

T5’s small changes to the 2017 transformer

T5 follows the original transformer (session 23) closely, with a few changes the paper lists [Certain]:

  • Pre-norm without bias. Layer normalization is applied to the input of each sublayer. It only rescales the activations and has no additive bias; the code does not subtract the mean either, so it is what is now called RMSNorm.
  • Relative position bias. There are no position vectors added to the embeddings. Instead, each attention head adds a learned scalar to each attention logit, chosen by the distance between query and key (Shaw et al., 2018 introduced relative positions; T5 simplified them to one number):
\[ a_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j + b_{h,\ \operatorname{bucket}(j - i)} \tag{24.4} \]
Read the symbols
\(a_{ij}\)
the attention logit of query \(i\) for key \(j\), before the softmax
\(b_{h,\cdot}\)
a learned table of 32 numbers for head \(h\). The table is shared by all layers of the stack; each head has its own. [Certain]
\(\operatorname{bucket}(j - i)\)
maps the offset to one of 32 buckets: small offsets get their own bucket, larger ones share buckets whose ranges grow logarithmically up to 128; every offset beyond 128 falls in the last bucket. [Certain]

Two details surprise people. First, T5 does not divide by \(\sqrt{d_k}\): the scale is folded into the initialization of the weights, and the Hugging Face code sets the scaling factor to 1.0. [Certain] Second, cross-attention in T5 has no position bias at all: the decoder finds positions in the input only through the content of \(H_{\text{enc}}\), which already carries the encoder’s position information. [Likely]

Worked example: counting T5-Base

T5-Base has \(d = 768\), \(d_{ff} = 3072\), 12 heads of size 64, 12 encoder blocks and 12 decoder blocks, and a vocabulary of 32,128 rows (32,000 SentencePiece pieces, 100 sentinels, padded to a multiple of 128). [Certain, from the paper and the public config.]

Encoder block: attention \(4d^2 = 2{,}359{,}296\) plus FFN \(2 d\, d_{ff} = 4{,}718{,}592\), so \(7{,}077{,}888\). Twelve blocks: 84.9 M.
Decoder block: two attentions \(8d^2 = 4{,}718{,}592\) plus FFN, so \(9{,}437{,}184\). Twelve blocks: 113.2 M.
Shared embedding: \(32{,}128 \times 768 = 24{,}674{,}304\).

Total: \(222{,}855{,}168\), plus 48,384 norm scales and position biases \(= 222{,}903{,}552\), the “about 220 million” of the paper. Cross-attention alone is \(12 \times 4d^2 = 28.3\) M, about 12.7% of the model; the paper rounds this to “about 10%”.

Span corruption and sentinel tokens

To pretrain without labels, we need a task that can be made from raw text. T5 uses a denoising objective: damage the input, and train the model to undo the damage. The specific damage is span corruption:

  1. Choose about 15% of the tokens to drop, grouped into short runs called spans (on average 3 tokens long).
  2. In the input, replace each span by one sentinel token. Sentinels are special tokens added to the vocabulary that mean “something is missing here”; each span in one example gets a different sentinel. T5 writes them as <X>, <Y>, <Z> in the paper; in the released vocabulary they are <extra_id_0> to <extra_id_99>. [Certain]
  3. The target is only the dropped spans, each one introduced by the same sentinel that replaced it in the input, plus one final sentinel to mark the end.

The paper’s example [Certain]: from “Thank you for inviting me to your party last week.” it drops “for inviting” and “last”:

Encoder inputThank you <X> me to your party <Y> week .
Decoder target<X> for inviting <Y> last <Z>

The sentinel does two jobs. In the input it marks the place of the hole, so the encoder knows where text is missing. In the target it says which hole the next tokens belong to, so the decoder never has to repeat the parts of the sentence it already sees. Why not just mark every dropped token with the same [MASK] and predict the full sentence, as in BERT? Because then the decoder must copy the 85% of tokens that were never damaged. That costs compute and teaches little.

How long are the input and the target?

Let the original text have \(n\) tokens, a corruption rate \(r\) and a mean span length \(\mu\). T5’s code first fixes the number of noise tokens and spans, then splits them at random:

\[ N = \operatorname{round}(r\,n), \qquad S = \operatorname{round}(N / \mu) \tag{24.5} \]
\[ |\tilde{\mathbf{x}}| = n - N + S, \qquad |\mathbf{y}| = N + S + 1 \]
Read the symbols
\(N\)
the number of dropped (noise) tokens
\(S\)
the number of spans, so also the number of sentinels in the input
\(\tilde{\mathbf{x}}\)
the corrupted input: the \(n - N\) kept tokens plus one sentinel per span
\(\mathbf{y}\)
the target: the \(N\) dropped tokens, one sentinel before each span, and one final token. (The paper ends with a last sentinel; the released code ends with </s> and also adds </s> to the input. [Likely]) Add 1 to each length if you count the end token.

The loss is formula (24.1) applied to this pair: the decoder predicts the target tokens, sentinels included, given the corrupted input:

\[ \mathcal{L}_{\text{span}} = -\sum_{t=1}^{|\mathbf{y}|} \log p_\theta\big(y_t \mid y_{\lt t},\ \tilde{\mathbf{x}}\big) \tag{24.6} \]
Worked example: why T5 targets are 114 tokens long

T5 trains with encoder inputs of 512 tokens. How much raw text does that need? Try \(n = 568\), \(r = 0.15\), \(\mu = 3\): \(N = \operatorname{round}(85.2) = 85\) noise tokens, \(S = \operatorname{round}(28.3) = 28\) spans.

Input: \(568 - 85 + 28 = 511\), plus the end token: 512. Target: \(85 + 28 + 1 = 114\) (here the last token is the end token). So 568 raw tokens give a full 512-token input and a 114-token target. These are exactly the lengths the Hugging Face T5 pretraining script computes. [Certain]

Compare a BERT-style objective that reconstructs the full text: the decoder would process all 568 tokens. Span corruption cuts the decoder’s work to \(114/568 \approx 20\%\), and the decoder self-attention, which is quadratic in target length, shrinks even more.

Original text (click tokens to drop or keep them)
Encoder input \(\tilde{\mathbf{x}}\)
Decoder target \(\mathbf{y}\)
Figure 2. A span corruption builder. A two-sentence text of 21 tokens (one word = one token here). “Paper example” drops the same words as Figure 2 of the T5 paper. “Random spans” uses T5’s procedure: fix \(N\) and \(S\) by formula (24.5), then split the noise tokens and the kept tokens into \(S\) random pieces each and interleave them (seeded, so “New sample” is repeatable). Sentinels are drawn filled.
How to read Figure 2

Crossed-out tokens are dropped. Each run of dropped tokens becomes one sentinel in the encoder input and reappears, after the same sentinel, in the target. The readout counts the lengths and compares the target with a BERT-style target that rebuilds all 21 tokens. Click “for” and “inviting”: two neighbors share one sentinel. Click a lonely word: it costs a whole sentinel for one token, so many short spans make both sequences longer. Raise \(\mu\) at fixed \(r\): fewer spans, so fewer sentinels and shorter sequences. Raise \(r\) to 0.5: the target grows toward the size of the input, and the paper found such high rates hurt GLUE and SQuAD.

What the T5 paper found about objectives

T5 compared many objectives with everything else fixed (a T5-Base-sized model, 34 B pretraining tokens). Some of the numbers, all from the paper’s validation sets [Certain]:

ObjectiveTargetGLUECNN/DM (ROUGE-2)SuperGLUE
Prefix language modelingsecond half of the text80.6918.9465.27
BERT-style (15%, with random tokens)full original text82.9619.1769.85
Deshufflingfull original text73.1718.5958.47
Replace corrupted spans (i.i.d. 15%)dropped tokens + sentinels83.2819.2471.36
Span corruption, mean span 3dropped spans + sentinels83.4919.6272.53

The lesson of the paper: denoising beats plain language modeling and deshuffling for this setup, and the many denoising variants are close to each other. Their advice was to choose among them mostly by cost, and span corruption is the cheapest because it gives the shortest sequences. [Certain] The corruption rate mattered little between 10% and 25%; 50% hurt GLUE and SQuAD. [Certain]

How the decoder uses cross-attention at every step

The decoder works differently in training and at inference, and many bugs live in the gap between the two.

Training: teacher forcing, all steps at once

In training we know the whole target. We feed the decoder the target shifted right by one, with a start token in front, and ask it to predict the unshifted target at every position in parallel. This is called teacher forcing:

\[ \mathbf{y}^{\text{in}} = (\,s,\ y_1,\ y_2,\ \dots,\ y_{m-1}\,), \qquad \text{position } t \text{ predicts } y_t \tag{24.7} \]
Read the symbols
\(s\)
the decoder start token. In T5 it is the padding token (ID 0); in BART it is </s> (ID 2). [Certain, from the public configs]
\(\mathbf{y}^{\text{in}}\)
what the decoder reads. The causal mask makes position \(t\) see only \(s, y_1, \dots, y_{t-1}\), so it cannot cheat by looking at \(y_t\).

All \(m\) positions run in one forward pass: the encoder runs once on \(\tilde{\mathbf{x}}\), the decoder runs once on \(\mathbf{y}^{\text{in}}\), and cross-attention (24.2) is one \(m \times n\) matrix product per head.

Inference: encode once, then decode step by step

At inference we do not know the target. We run the encoder once. Then we generate: start with \(s\), predict \(y_1\), append it, predict \(y_2\), and so on until </s>. At step \(t\) only one new query row is computed in each decoder block. Two caches make this fast (session 80):

  • The self-attention cache stores the keys and values of the target tokens so far. It grows by one row per step, exactly like a decoder-only model’s KV cache.
  • The cross-attention cache stores \(K = H_{\text{enc}} W_K\) and \(V = H_{\text{enc}} W_V\) for every decoder block. It is computed at the first step and never changes, because the input never changes.
\[ M_{\text{cross}} = 2 \cdot L_{\text{dec}} \cdot n \cdot d \cdot b \ \text{ bytes}, \qquad M_{\text{self}}(t) = 2 \cdot L_{\text{dec}} \cdot t \cdot d \cdot b \ \text{ bytes} \tag{24.8} \]
Read the symbols
\(2\)
one tensor for keys and one for values
\(L_{\text{dec}}\)
the number of decoder blocks: each has its own \(W_K, W_V\), so its own cache
\(n\), \(t\)
the input length, and the number of target tokens generated so far
\(d\), \(b\)
the width (all heads together) and the bytes per number (2 for bfloat16)
Worked example: one attention step, and Whisper’s cross-attention cache

One step by hand. Take one head with \(d_k = 2\). The decoder query is \(\mathbf{q} = (1, 2)\). Three encoder positions have keys \((1, 0), (0, 1), (1, 1)\) and values \((1, 0), (0, 1), (2, 2)\). Scores \(\mathbf{q}\cdot\mathbf{k}_j/\sqrt{2}\): \(0.707,\ 1.414,\ 2.121\). Softmax: \(0.140,\ 0.284,\ 0.576\) (they sum to 1). Output: \(0.140\,(1,0) + 0.284\,(0,1) + 0.576\,(2,2) = (1.292,\ 1.436)\). The decoder read mostly from the third input position.

A real cache. Whisper large-v3 has 32 decoder blocks of width 1,280, and its encoder turns 30 seconds of audio into 1,500 positions. [Certain, from the config] By formula (24.8), the cross-attention cache is \(2 \times 32 \times 1{,}500 \times 1{,}280 = 122{,}880{,}000\) numbers, which is 245.8 MB in 16-bit, per audio clip, before a single word is written. Its decoder writes at most 448 tokens, so the self-attention cache can never be larger than \(448/1500 \approx 30\%\) of the cross cache. For Whisper small (12 blocks, width 768) it is 55.3 MB.

Figure 3. Decoding “translate English to German: That is good .” → “Das ist gut . </s>” one step at a time. Top: the 9 encoder outputs (encoded once). Bottom: the decoder tokens; the blue box is the current query. The orange lines and bars show one head’s cross-attention weights over the input. The scores are set by hand to look like a trained model; the softmax and all counts are computed live.
How to read Figure 3

Press Next. At each step the newest decoder token sends one query to all 9 encoder positions; the thicker the line, the larger the weight. The weights in every step sum to 1. Writing “Das” reads mostly from “That”, writing “gut” from “good”: this is the alignment that Bahdanau attention discovered (session 7). At the last step the weights spread out, because ending the sentence does not depend on one word. Watch the cache counts in the readout: the cross-attention K/V rows stay at 9 per block from the first step to the last, while the self-attention cache grows by one per step.

BART: corrupt the text, then rebuild it

BART (Lewis et al., 2019) uses the same encoder–decoder shape, with a different philosophy. It calls itself a denoising autoencoder: corrupt a document with some noising function, feed it to a bidirectional encoder, and train a left-to-right decoder to write the full original document. [Certain]

\[ \mathcal{L}_{\text{BART}} = -\sum_{t=1}^{n} \log p_\theta\big(x_t \mid x_{\lt t},\ g(\mathbf{x})\big) \tag{24.9} \]
Read the symbols
\(g(\mathbf{x})\)
the corrupted document: the encoder input. \(g\) is a random noising function.
\(x_t\)
token \(t\) of the original document. The target is the clean text, all \(n\) tokens, not only the damaged parts.

Because the target is always the full clean text, BART can use any damage, even damage that changes the length or the order. The paper tries five noising functions [Certain]:

  • Token masking: replace random tokens with <mask>, as in BERT.
  • Token deletion: delete random tokens. Now the model must also find where something is missing.
  • Text infilling: choose spans whose lengths come from a Poisson distribution with \(\lambda = 3\) and replace each span with a single <mask>. A span of length 0 inserts a <mask> where nothing is missing. The model must learn how many tokens each mask hides.
  • Sentence permutation: split the document at full stops and shuffle the sentences.
  • Document rotation: pick a random token and rotate the document so it starts there. The model must find the true start.
\[ P(\ell = k) = \frac{\lambda^k e^{-\lambda}}{k!}, \qquad \lambda = 3:\quad P(0) = 0.050,\ P(1) = 0.149,\ P(2) = P(3) = 0.224 \tag{24.10} \]

So about 5% of the sampled “spans” are empty insertions, and spans of length 2 and 3 are the most common. Text infilling is close to T5’s span corruption; the difference is the target. T5 writes only the missing pieces after numbered sentinels. BART writes the whole document after one kind of <mask>, so its decoder also practices fluent copying, which helps tasks like summarization where the output reuses much of the input.

Encoder input \(g(\mathbf{x})\)
Decoder target: the original document \(\mathbf{x}\)
Figure 4. BART’s noising functions on an 18-token document of three sentences. Masking, deletion and infilling damage 30% of the tokens (5 of 18), the rate BART used for its final model. Infilling draws span lengths from \(\operatorname{Poisson}(3)\), formula (24.10). All randomness is seeded.
How to read Figure 4

The target never changes: it is always the clean document. Only the input changes. Compare the readout lengths: masking keeps the length, deletion shortens it and hides the positions, infilling shortens it and hides how many tokens each <mask> covers. Press “New sample” on infilling a few times: sometimes one <mask> hides all 5 tokens, sometimes three masks hide 1, 2 and 2. Spans are placed so they do not touch, so the hidden counts always add up to 5. A mask that hides 0 tokens (an inserted mask) comes from the 5% of empty spans, so it is rare in a text this short. Permutation and rotation keep every token but break the order; BART found these weak alone. “Infilling + permutation” is BART’s final recipe.

Which noise works best? BART’s ablation, with base-sized models trained for 1 M steps on books and Wikipedia [Certain]:

BART-Base noiseSQuAD 1.1 F1MNLI acc.XSum perplexityCNN/DM perplexity
Token masking90.484.17.086.10
Token deletion90.484.16.905.87
Text infilling90.884.06.615.83
Document rotation77.275.317.1410.59
Sentence shuffling85.481.510.937.89
Infilling + shuffling90.883.86.625.41

Lower perplexity is better. Infilling is the most consistent; rotation and shuffling alone are poor; deletion beats masking on the generation tasks. The final BART-Large (12 + 12 layers, width 1,024, GPT-2’s byte-level BPE, 160 GB of text) used infilling with 30% of tokens masked plus permutation of all sentences. [Certain] It matched RoBERTa on GLUE and SQuAD and set a new best on summarization: ROUGE-1/2/L of 44.16 / 21.28 / 40.90 on CNN/DailyMail and 45.14 / 22.27 / 37.25 on XSum. [Certain]

Using BART and T5 after pretraining

  • Generation tasks (summarization, translation, question answering with free answers): fine-tune with formula (24.1) and decode with beam search or sampling (session 79).
  • Classification with BART: feed the same text to the encoder and the decoder, and put a classifier on the decoder’s final state at the last token, which has seen the whole input. [Certain]
  • Classification with T5: just generate the label word. No new layers.

Beyond T5 and BART

A few descendants are worth knowing by name, because you will meet their checkpoints:

  • T5 v1.1 and Flan-T5. T5 v1.1 changed the FFN to a gated GELU (\(d_{ff} = 2048\) at Base) and stopped tying the input and output embeddings. [Certain, from the configs] Flan-T5 (Chung et al., 2022) is T5 instruction-tuned on about 1.8 K tasks; it is a strong, small model for following instructions on one input. [Certain]
  • mT5 (Xue et al., 2020): T5 pretrained on Common Crawl text in 101 languages, with a vocabulary of 250,112 rows. [Certain] A large vocabulary is the price of many scripts (session 8).
  • mBART: BART pretrained on many languages, a common start for translation models. [Certain]
  • UL2 (Tay et al., 2022): a mixture of denoisers. It mixes T5-style span corruption with short spans, prefix language modeling, and “extreme” corruption with long spans or high rates, and adds a mode token so the model knows which kind of task it is doing. [Certain] It showed that the objective and the architecture are separate choices.

Encoder–decoder or decoder-only: cost and masks

T5 made a careful comparison that is still the best way to think about the trade-off. Let one stack of \(L\) layers have \(P\) parameters. An encoder–decoder with \(L\) encoder layers and \(L\) decoder layers then has about \(2P\) parameters, but it costs about the same compute as a decoder-only model with only \(L\) layers. The reason: the decoder-only model must push both the input and the output through all its layers, while the encoder–decoder pushes the input only through the encoder and the output only through the decoder. [Certain]

Let us check this with the counting rules of session 25 (a matrix–vector product with a \(d \times d\) matrix costs \(2d^2\) FLOPs; attention-score FLOPs are ignored; \(d_{ff} = 4d\)). Per token, a decoder-only layer uses \(12d^2\) weights. An encoder layer also uses \(12d^2\), plus the cross-attention \(K, V\) projections in the decoder are applied to the input tokens (\(2d^2\)). A decoder layer applies \(4d^2\) self-attention, \(2d^2\) for the cross-attention \(Q\) and output projections, and \(8d^2\) FFN to each target token:

\[ \begin{aligned} N_{\text{enc-dec}} &\approx 12Ld^2 + 16Ld^2 = 28Ld^2, & C_{\text{enc-dec}} &\approx 2Ld^2\,\big(14\,n + 14\,m\big) \\ N_{\text{dec-only}} &\approx 12Ld^2, & C_{\text{dec-only}} &\approx 2Ld^2\,\big(12\,n + 12\,m\big) \end{aligned} \tag{24.11} \]
Read the symbols
\(N\)
parameters without embeddings
\(C\)
forward FLOPs of the weight matrices for one (input, target) pair with \(n\) input and \(m\) target tokens
\(L\)
layers per stack: \(L + L\) for the encoder–decoder, \(L\) for the decoder-only model

So the encoder–decoder has \(28/12 = 2.33\times\) the parameters (the extra cross-attention makes it more than \(2\times\)) for only \(14/12 = 1.17\times\) the compute. With T5-Base sizes (\(d = 768\), \(L = 12\)) and T5’s lengths (\(n = 512\), \(m = 114\)): 198 M versus 85 M parameters, and 124 versus 106 GFLOPs per example. T5 measured nearly identical step times for the two. [Certain]

The attention masks show the other half of the story. Write the input and the target as one long sequence. A decoder-only causal LM lets each token see only earlier tokens, even inside the input. An encoder–decoder lets input tokens see each other in both directions, and target tokens see all input tokens plus earlier target tokens. A prefix LM is a single stack with exactly that encoder–decoder visibility pattern (session 18). Figure 5 draws all three.

Figure 5. Who can see whom. Rows are queries, columns are keys, over the input tokens \(x\) followed by the target tokens \(y\). A filled cell means “this row may attend to this column”. The readout counts the visible pairs for the lengths you chose, and gives the parameters and forward GFLOPs of formula (24.11) at T5-Base size (\(d = 768\), \(L = 12\), \(n = 512\), \(m = 114\)), with the scores T5 measured for each architecture trained with its denoising objective (T5 Table 2).
How to read Figure 5

Start with the encoder–decoder. The top-left block (input sees input) is full: bidirectional encoder self-attention. The bottom-left block is cross-attention: every target row sees every input column. The bottom-right block is a triangle: causal decoder self-attention. The top-right block is empty: the input never sees the target. Now switch to prefix LM: the same cells are filled, but all by one stack of weights. Switch to causal LM: the top-left block becomes a triangle, so early input tokens can no longer see later ones. That one change cost T5 more than 7 GLUE points in its comparison. With \(n = 4\), \(m = 3\), the encoder–decoder has 34 visible pairs and the causal LM 28.

T5’s results for these three, all trained with the same denoising objective [Certain]: encoder–decoder 83.28 GLUE and 19.24 ROUGE-2 on CNN/DM; prefix LM 81.82 and 18.61; causal LM 74.70 and 17.93. Sharing the weights between encoder and decoder (\(P\) parameters) did nearly as well as the full encoder–decoder (82.81 GLUE), while halving the depth to 6+6 layers hurt more (80.88). [Certain]

So why did decoder-only win at scale?

If encoder–decoders did better at equal compute, why are GPT, Llama and Qwen all decoder-only? A few reasons, the first one measured, the others widely believed [Likely]:

  • Zero-shot behavior. Wang et al. (2022) trained models of more than 5 B parameters on more than 170 B tokens and found that causal decoder-only models with plain next-token training gave the best zero-shot results right after pretraining, while non-causal models with masked objectives were best only after multitask fine-tuning. [Certain] Chat models are used zero-shot and few-shot, on any task, so this matters.
  • Every token is a training signal. A causal LM learns from every token of the text; span corruption learns from only about 15% of positions (the target), although it reads all of them.
  • Simplicity of one stack. One stack, one cache, no split of the context into “input” and “output”. Multi-turn chat has no natural split: yesterday’s output is today’s input. Serving systems, kernels and parallelism tools were built for this shape.
  • Parameter efficiency at inference. For generation-heavy chat, the decoder does most of the work, and in an encoder–decoder only half of the parameters sit in the decoder.

When encoder–decoders still win

The design is strongest when the input is long, fully known in advance, and the output is shorter and tightly tied to the input:

  • Speech recognition. Whisper (Radford et al., 2022) is an encoder–decoder trained on 680,000 hours of audio. [Certain] The encoder reads a 30-second log-Mel spectrogram (1,500 positions), and the decoder writes the transcript with cross-attention. The input is not text, so a separate encoder for it is natural; the encoder runs once per 30 seconds while the decoder writes many tokens.
  • Translation. NLLB-200, Meta’s model for 200 languages, is an encoder–decoder (12 + 12 layers in the 600 M distilled version). [Certain] A bidirectional encoder over the source sentence is a good fit for translation, where word order changes between languages.
  • Summarization. BART and its descendants were for years the standard fine-tuned summarizers. [Likely] For a fixed, high-volume summarization job, a fine-tuned encoder–decoder of a few hundred million parameters is cheap to serve. [Likely]
  • OCR and documents. Donut uses an image encoder and a BART-style text decoder to read documents without a separate OCR step; TrOCR uses an image transformer encoder and a text decoder. [Certain] See session 218.
  • Text encoders for generation. Imagen uses a frozen T5-XXL encoder to turn the prompt into vectors for its image model, and Stable Diffusion 3 uses T5-XXL as one of its text encoders. [Certain] Many text-to-video models do the same. [Likely] Only the encoder half is used; the image or video model cross-attends to it.
  • VLMs with cross-attention. Some VLMs keep the language model frozen and add cross-attention layers that read image features, exactly like formula (24.2) with \(H_{\text{enc}}\) replaced by vision features (session 179).

Where this is used

ApplicationWhat the encoder–decoder doesLink
Speech-to-text, subtitlesAudio encoder (Whisper) once per 30 s window; text decoder cross-attends to the 1,500 audio positionssession 184
Machine translationBidirectional encoder over the source sentence; decoder with cross-attention writes the target languagesession 148
SummarizationBART / T5 fine-tuned with formula (24.1); the decoder copies and compresses from the encoded documentsession 147
Video captioning and dense video captioningA video encoder turns frames into features; a text decoder writes captions (and time stamps for key moments) with cross-attention to themsession 205
Video-RAG and long video question answeringA small encoder–decoder (for example Flan-T5) summarizes or rewrites transcripts and frame captions into short notes that a retriever indexessession 210
Text-to-image and text-to-video generationThe T5 encoder turns the prompt into a sequence of vectors; the generator cross-attends to themsession 208
Document AI and OCRImage encoder over the page; text decoder writes the text or a structured output (JSON fields)session 218
Query rewriting, data-to-text, grammar correctionShort, constrained outputs from a known input: the classic text-to-text shape, served cheaply by a small T5session 54

Pseudo-code

Algorithm 1: T5 span corruption for one example
  1. input: tokens \(x_1, \dots, x_n\), rate \(r\), mean span \(\mu\), sentinels \(s_0, s_1, \dots\)
  2. \(N \leftarrow \operatorname{round}(r n)\), \(\ S \leftarrow \max(1, \operatorname{round}(N/\mu))\) // formula (24.5)
  3. split \(N\) into \(S\) random positive parts \(z_1..z_S\); split \(n - N\) into \(S\) parts \(k_1..k_S\)
  4. mask \(\leftarrow\) \(k_1\) keeps, \(z_1\) drops, \(k_2\) keeps, \(z_2\) drops, … // interleave
  5. \(\tilde{\mathbf{x}} \leftarrow [\,]\), \(\mathbf{y} \leftarrow [\,]\), \(j \leftarrow 0\)
  6. for \(i = 1\) to \(n\) do
  7. if mask\(_i\) and not mask\(_{i-1}\) then append \(s_j\) to \(\tilde{\mathbf{x}}\) and to \(\mathbf{y}\); \(j \leftarrow j + 1\) // a new span
  8. if mask\(_i\) then append \(x_i\) to \(\mathbf{y}\) else append \(x_i\) to \(\tilde{\mathbf{x}}\)
  9. append \(s_j\) (or </s>) to \(\mathbf{y}\) // end of the target
  10. return \((\tilde{\mathbf{x}}, \mathbf{y})\) // train with formula (24.6)
Algorithm 2: greedy generation with an encoder–decoder
  1. input: input tokens \(\mathbf{x}\), start token \(s\), max length \(T\)
  2. \(H_{\text{enc}} \leftarrow \operatorname{Encoder}(\mathbf{x})\) // once
  3. for each decoder block \(l\) do \(K^{\times}_l \leftarrow H_{\text{enc}} W_K^{(l)}\), \(V^{\times}_l \leftarrow H_{\text{enc}} W_V^{(l)}\) // cross cache, formula (24.8)
  4. \(y_0 \leftarrow s\); self-attention caches \(\leftarrow\) empty
  5. for \(t = 1\) to \(T\) do
  6. \(\mathbf{h} \leftarrow \operatorname{Embed}(y_{t-1})\)
  7. for \(l = 1\) to \(L_{\text{dec}}\) do
  8. \(\mathbf{h} \leftarrow \mathbf{h} + \operatorname{SelfAttn}_l(\mathbf{h};\ \text{cache}_l)\) // append this step’s k, v to cache\(_l\)
  9. \(\mathbf{h} \leftarrow \mathbf{h} + \operatorname{CrossAttn}_l(\mathbf{h};\ K^{\times}_l, V^{\times}_l)\) // formula (24.2), no causal mask
  10. \(\mathbf{h} \leftarrow \mathbf{h} + \operatorname{FFN}_l(\mathbf{h})\) // formula (24.3), norms omitted
  11. \(y_t \leftarrow \argmax \softmax(E\,\mathbf{h})\)
  12. if \(y_t =\) </s> then break
  13. return \(y_1, \dots, y_t\)
Algorithm 3: BART text infilling
  1. input: tokens \(x_1..x_n\), mask ratio \(\rho = 0.3\), \(\lambda = 3\)
  2. budget \(\leftarrow \operatorname{round}(\rho n)\); lengths \(\leftarrow [\,]\)
  3. while \(\operatorname{sum}(\text{lengths}) \lt \text{budget}\) do append \(\ell \sim \operatorname{Poisson}(\lambda)\) // formula (24.10)
  4. shorten the last length so that \(\operatorname{sum}(\text{lengths}) = \text{budget}\)
  5. for each \(\ell\) in lengths do pick a random start \(p\); mark \(x_p..x_{p+\ell-1}\) as span \(\ell\) (\(\ell = 0\): insert a mask before \(x_p\))
  6. \(g(\mathbf{x}) \leftarrow\) the text with every marked run replaced by one <mask>
  7. return \((g(\mathbf{x}),\ \mathbf{x})\) // target = full original, formula (24.9)

The same ideas in PyTorch. The script builds T5 inputs and targets, checks that cross-attention reuses the same keys and values at every step, and trains a tiny encoder–decoder with span corruption on toy sequences. It runs on a CPU in a few seconds:

import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)

# 1) T5 span corruption, formula (24.5)
def span_mask(n, rate=0.15, mean_span=3.0, g=None):
    n_noise = max(1, round(n * rate))
    n_spans = max(1, round(n_noise / mean_span))
    def split(total, parts):                       # random split of `total` into `parts` pieces >= 1
        cuts = (torch.randperm(total - 1, generator=g)[:parts - 1] + 1).sort().values
        b = torch.cat([torch.tensor([0]), cuts, torch.tensor([total])])
        return (b[1:] - b[:-1]).tolist()
    mask = []
    for keep, noise in zip(split(n - n_noise, n_spans), split(n_noise, n_spans)):
        mask += [False] * keep + [True] * noise    # keep-run, then noise-run, like T5
    return mask

def span_corrupt(toks, mask, sentinel):            # sentinel(k) gives the k-th sentinel
    inp, tgt, k = [], [], 0
    for i, (t, m) in enumerate(zip(toks, mask)):
        if m and (i == 0 or not mask[i - 1]):      # a new span starts
            inp.append(sentinel(k)); tgt.append(sentinel(k)); k += 1
        (tgt if m else inp).append(t)
    tgt.append(sentinel(k))                        # final sentinel, as in the paper
    return inp, tgt

words = "Thank you for inviting me to your party last week .".split()
mask = [w in ("for", "inviting", "last") for w in words]
inp, tgt = span_corrupt(words, mask, lambda k: "<extra_id_%d>" % k)
print(" ".join(inp))   # Thank you <extra_id_0> me to your party <extra_id_1> week .
print(" ".join(tgt))   # <extra_id_0> for inviting <extra_id_1> last <extra_id_2>
m = span_mask(568, g=torch.Generator().manual_seed(0))
print(sum(m), len(span_corrupt(list(range(568)), m, lambda k: -1 - k)[0]) + 1)  # 85 512

# 2) cross-attention, formula (24.2): queries from the decoder, keys and values from the encoder
class CrossAttention(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h, self.dk = h, d // h
        self.wq, self.wk, self.wv, self.wo = (nn.Linear(d, d, bias=False) for _ in range(4))
    def split(self, x):                            # (B, T, d) -> (B, h, T, dk)
        return x.view(x.shape[0], x.shape[1], self.h, self.dk).transpose(1, 2)
    def forward(self, y, kv):                      # y: (B, Ty, d); kv = (K, V), computed once
        q = self.split(self.wq(y))
        a = torch.softmax(q @ kv[0].transpose(-1, -2) / self.dk ** 0.5, dim=-1)  # (B, h, Ty, Tx)
        out = (a @ kv[1]).transpose(1, 2).reshape(y.shape)
        return self.wo(out), a
    def encode_kv(self, h_enc):                    # the cross-attention "KV cache"
        return self.split(self.wk(h_enc)), self.split(self.wv(h_enc))

ca = CrossAttention(d=32, h=4)
h_enc, y = torch.randn(2, 7, 32), torch.randn(2, 5, 32)   # Tx = 7 source, Ty = 5 target positions
kv = ca.encode_kv(h_enc)
out, a = ca(y, kv)
print(out.shape, a.shape, torch.allclose(a.sum(-1), torch.ones(2, 4, 5)))  # (2,5,32) (2,4,5,7) True
step3, _ = ca(y[:, 3:4], kv)                       # one decoding step reuses the same K, V
print(torch.allclose(step3, out[:, 3:4], atol=1e-6))  # True: no causal mask in cross-attention

# 3) a tiny encoder-decoder trained with span corruption on toy sequences
V, n, S0 = 16, 24, 16                             # 16 ordinary tokens; sentinels are ids 16, 17, ...
BOS, vocab = 20, 21
def batch(B, g):
    a = torch.randint(0, V, (B, 1), generator=g); b = torch.randint(1, V, (B, 1), generator=g)
    x = (a + b * torch.arange(n)) % V             # x_i = a + i*b (mod 16): spans are predictable
    X, Y = [], []
    for row in x.tolist():
        mk = span_mask(n, rate=0.25, mean_span=3.0, g=g)       # 6 noise tokens in 2 spans
        i, t = span_corrupt(row, mk, lambda k: S0 + k)
        X.append(i); Y.append(t)
    return torch.tensor(X), torch.tensor(Y)        # (B, 20) and (B, 9)

class TinyT5(nn.Module):
    def __init__(self, d=64):
        super().__init__()
        self.emb, self.pos = nn.Embedding(vocab, d), nn.Embedding(32, d)
        self.tf = nn.Transformer(d, 4, 2, 2, 128, dropout=0.0, batch_first=True)
        self.head = nn.Linear(d, vocab)
    def forward(self, src, tgt_in):
        e = lambda z: self.emb(z) + self.pos(torch.arange(z.shape[1]))
        causal = nn.Transformer.generate_square_subsequent_mask(tgt_in.shape[1])
        return self.head(self.tf(e(src), e(tgt_in), tgt_mask=causal))

model, g = TinyT5(), torch.Generator().manual_seed(1)
opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
for step in range(301):
    src, tgt = batch(64, g)
    tgt_in = torch.cat([torch.full((64, 1), BOS), tgt[:, :-1]], 1)  # teacher forcing: shift right (24.7)
    loss = F.cross_entropy(model(src, tgt_in).reshape(-1, vocab), tgt.reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 100 == 0: print(step, round(loss.item(), 3))
# 0 3.161 / 100 1.076 / 200 0.548 / 300 0.386 on our machine (PyTorch 2.8, CPU);
# the start is near ln(21) = 3.04, the guess of a model that knows nothing

Common mistakes

  • Forgetting the shift. Feeding the target itself (not shifted right) as the decoder input lets position \(t\) see \(y_t\): the training loss drops to almost zero and generation is garbage. Use the model’s own start token (T5: pad ID 0; BART: </s> ID 2), formula (24.7).
  • Putting a causal mask on cross-attention, or forgetting the padding mask over the encoder input. The first hides real input from early target positions; the second lets the decoder read padding vectors.
  • Recomputing the cross-attention K and V at every step. They depend only on the input. Compute them once per decoder block (Algorithm 2); otherwise each step costs \(O(n d^2)\) extra work per block.
  • Reusing sentinels wrongly. Each span in an example needs its own sentinel, in the same order in the input and the target. Using one shared mask token with a short target makes the target ambiguous: the model cannot know which hole a piece belongs to.
  • Fine-tuning T5 without its task prefix, or with a different one. The original T5 checkpoints were trained with a mix of tasks and prefixes like summarize:. Small changes to the prefix can change the output a lot. [Likely]
  • Counting an L+L encoder–decoder like a 2L-layer GPT for compute. In parameters it is close to 2L layers (a bit more, because of cross-attention); in FLOPs per example it is close to an L-layer model, formula (24.11).
  • Expecting T5 sentinels in a generation to be text. A pretrained (not fine-tuned) T5 answers in the span-corruption format, <extra_id_0> … <extra_id_1>. Strip or use them; do not show them to users.

Interview questions

What does cross-attention take from the encoder and what from the decoder? What are the shapes?basic
Queries come from the decoder states \(S \in \R^{m \times d}\) (after the decoder’s self-attention sublayer); keys and values come from the final encoder output \(H_{\text{enc}} \in \R^{n \times d}\), formula (24.2). The score matrix per head is \(m \times n\), not square, with no causal mask (only a padding mask on the input). Each target position gets a softmax distribution over the input positions and returns a weighted sum of input values. Every decoder block has its own \(W_Q, W_K, W_V, W_O\), but all blocks read the same \(H_{\text{enc}}\).
Explain T5’s span corruption. Why use sentinel tokens instead of BERT’s [MASK]?core
Drop about 15% of tokens in spans of mean length 3; replace each span in the input with a unique sentinel (<extra_id_0>, …); the target is only the dropped spans, each after its sentinel, plus an end marker. The sentinels tell the encoder where text is missing and tell the decoder which hole each piece fills, so the target can skip all the unchanged text. With one shared [MASK], a short target would be ambiguous, so you would have to rebuild the full text, which makes the decoder sequence about 5× longer (568 vs 114 tokens in T5’s setting). T5 found the variants perform about the same, so the cheaper one wins.
T5 uses 512-token inputs. How long is the target, and how much raw text is used per example?core
With \(r = 0.15\), \(\mu = 3\), 568 raw tokens give \(N = 85\) noise tokens and \(S = 28\) spans. Input \(= 568 - 85 + 28 + 1 = 512\) (with the end token); target \(= 85 + 28 + 1 = 114\). So each example reads 568 tokens of text, the encoder processes 512 and the decoder only 114 (formula 24.5).
How do training and inference differ in an encoder–decoder?basic
Training uses teacher forcing: the decoder reads the true target shifted right with a start token (24.7), and all positions are predicted in one parallel pass under a causal mask. Inference is autoregressive: run the encoder once, compute the cross-attention K and V once per decoder block, then generate one token per step, feeding back the model’s own previous token and growing a self-attention KV cache. The mismatch (training on true prefixes, testing on own predictions) is called exposure bias.
How big is the cross-attention cache of Whisper large-v3 for one 30 s clip?advanced
Formula (24.8): \(2 \cdot L_{\text{dec}} \cdot n \cdot d \cdot b = 2 \times 32 \times 1500 \times 1280 \times 2\) bytes \(= 245.8\) MB in 16-bit. It is fixed for the whole transcription of that window, and is larger than the self-attention cache, which is at most 448 positions. In batched serving it can be the main memory cost. It is identical for all beams of one input, so an efficient server keeps one copy per input instead of one per beam, and frees it as soon as the window is done. [Likely]
What noising functions does BART use, and which worked best?core
Token masking, token deletion, text infilling (\(\operatorname{Poisson}(3)\) span lengths, each span replaced by one <mask>, zero-length spans insert a mask), sentence permutation and document rotation. The target is always the full original document (24.9). Text infilling was the most consistent; rotation and permutation alone were poor; deletion beat masking on generation. The final model used infilling with 30% of tokens masked plus permutation of all sentences.
Compare T5’s and BART’s pretraining targets. Which is cheaper, and what does each teach the decoder?core
T5 predicts only the dropped spans with sentinels: short targets (about 20% of the text), cheap, and the decoder learns to fill gaps. BART predicts the full document: targets as long as the text, more expensive, but the decoder also practices writing long fluent text and copying from the input, which suits summarization and other generation tasks. BART’s infilling also hides span length (one mask per span), so the model must decide how many tokens to write; T5’s sentinels hide it too, but the target is not a full sentence.
An encoder–decoder with 12+12 layers vs a 12-layer decoder-only model of the same width: parameters and compute?advanced
Parameters: about \(28Ld^2\) vs \(12Ld^2\), so 2.33× (2× from two stacks plus cross-attention). Compute per (input, target) pair: the decoder-only model runs all \(n + m\) tokens through 12 layers (\(24Ld^2(n + m)\) FLOPs); the encoder–decoder runs the input through 12 encoder layers and the target through 12 decoder layers, \(28Ld^2(n + m)\) FLOPs, only 1.17× (formula 24.11). T5 measured nearly identical step times. So at equal compute, an encoder–decoder has about twice the parameters, and in T5’s tests it scored best.
What is a prefix LM, and how does it relate to an encoder–decoder?core
A single decoder stack whose mask is bidirectional over the input prefix and causal over the target. Its visibility pattern is the same as an encoder–decoder’s (input sees input; target sees all input and earlier target), but one set of weights does both jobs, so it has half the parameters at the same compute. In T5’s comparison it was between the encoder–decoder and the causal LM (81.82 vs 83.28 vs 74.70 GLUE with the denoising objective).
If encoder–decoders did better in T5’s study, why are most modern LLMs decoder-only?advanced
T5’s comparison was about fine-tuning on fixed tasks. For general zero-shot use, Wang et al. (2022) found causal decoder-only models with next-token training best right after pretraining. A causal LM also gets a learning signal from every token, has one stack and one KV cache, handles multi-turn chat without an artificial input/output split, and the tooling grew up around it. Encoder–decoders still win where the input is fixed and long and the output is short and tied to it: speech, translation, summarization, OCR, and as text encoders for generators.
Why does T5 not need position embeddings, and how does its relative bias work?advanced
Each head adds a learned scalar \(b_{h,\operatorname{bucket}(j-i)}\) to its attention logit (24.4). 32 buckets cover offsets: exact buckets for small offsets, log-spaced buckets up to 128, one bucket beyond. The table is shared across layers (one per head), so it adds almost no parameters, and it works for any length because it depends on offsets, not absolute positions. One layer cannot tell offsets beyond 128 apart, but stacked layers can combine local information. Note also that T5 does not scale logits by \(1/\sqrt{d_k}\); this is folded into the weight initialization.

Exercises

  1. Apply span corruption by hand to “the cat sat on the warm mat today” with dropped tokens “sat on” and “today”. Write the input and the target with <X>, <Y>, <Z>, and give both lengths.
  2. With \(n = 1000\), \(r = 0.15\) and \(\mu = 3\), compute \(N\), \(S\), the input length and the target length by formula (24.5). How long would the target be with \(\mu = 1\)?
  3. A decoder query is \(\mathbf{q} = (2, 0)\) and the keys are \((1, 0), (0, 1), (1, 1)\) with \(d_k = 2\) (no other scaling). Compute the cross-attention weights.
  4. Whisper small has 12 decoder blocks of width 768. Compute its cross-attention cache for one 30-second window in bfloat16.
  5. Using formula (24.11), how many times more parameters and compute does a 24+24-layer encoder–decoder have than a 24-layer decoder-only model of the same width?
  6. In BART text infilling with \(\lambda = 3\), what is the chance that a sampled span is empty, and what is the chance that it is longer than 4 tokens?
  7. In Figure 5, count the visible pairs for \(n = 5\), \(m = 2\) for the encoder–decoder and for the causal LM. Give a general formula for both.
Answerscheck
1. Input: the cat <X> the warm mat <Y> (7 tokens); target: <X> sat on <Y> today <Z> (6 tokens). Check: \(n = 8\), \(N = 3\), \(S = 2\): \(8 - 3 + 2 = 7\) and \(3 + 2 + 1 = 6\). 2. \(N = 150\), \(S = 50\), input \(1000 - 150 + 50 = 900\), target \(150 + 50 + 1 = 201\). With \(\mu = 1\): \(S = 150\), target \(= 301\) and input \(= 1000\): every dropped token costs a sentinel too. 3. Scores \(2/\sqrt2 = 1.414\), \(0\), \(1.414\); softmax: \(0.446, 0.108, 0.446\). 4. \(2 \times 12 \times 1500 \times 768 \times 2\) bytes \(= 55{,}296{,}000\) bytes \(\approx 55.3\) MB. 5. Parameters \(28/12 = 2.33\times\); compute \(14/12 = 1.17\times\), for any \(L\). 6. \(P(0) = e^{-3} = 0.050\); \(P(\ell \gt 4) = 1 - (0.050 + 0.149 + 0.224 + 0.224 + 0.168) = 0.185\). 7. Encoder–decoder: \(n^2 + mn + m(m+1)/2 = 25 + 10 + 3 = 38\). Causal LM: \((n+m)(n+m+1)/2 = 7 \cdot 8/2 = 28\).

Key takeaways

  • T5 writes every task as text in, text out, with a task prefix, and trains all of them with one loss, formula (24.1).
  • A decoder block has three sublayers: causal self-attention, cross-attention, FFN (24.3). Cross-attention takes queries from the decoder and keys and values from the final encoder output (24.2), with no causal mask.
  • Span corruption drops about 15% of tokens in spans of mean length 3, puts one unique sentinel per span, and predicts only the dropped spans. Targets are short: 114 tokens for a 512-token input (24.5).
  • At inference the encoder runs once and its K and V are cached per decoder block (24.8); for Whisper large-v3 that is about 246 MB per 30-second window.
  • BART corrupts text (masking, deletion, infilling with \(\operatorname{Poisson}(3)\) spans, permutation, rotation) and rebuilds the whole document (24.9); infilling plus sentence permutation worked best.
  • An L+L encoder–decoder has about 2.3× the parameters of an L-layer decoder-only model for about 1.17× the compute (24.11). Decoder-only won for general chat; encoder–decoders still win for speech, translation, summarization, OCR and as text encoders for generators.

Key formulas

\[ p_\theta(\mathbf{y} \mid \mathbf{x}) = \prod_{t} p_\theta(y_t \mid y_{\lt t}, \mathbf{x}), \qquad \operatorname{CrossAttn} = \softmax\!\Big(\frac{(S W_Q)(H_{\text{enc}} W_K)^\top}{\sqrt{d_k}}\Big) H_{\text{enc}} W_V \]
\[ N = \operatorname{round}(rn),\ \ S = \operatorname{round}(N/\mu),\ \ |\tilde{\mathbf{x}}| = n - N + S,\ \ |\mathbf{y}| = N + S + 1 \]
\[ M_{\text{cross}} = 2 L_{\text{dec}}\, n\, d\, b, \qquad N_{\text{enc-dec}} \approx 28Ld^2, \quad C_{\text{enc-dec}} \approx 28Ld^2(n+m) \]

References for this session

  1. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  2. Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  3. Vaswani et al. (2017). Attention Is All You Need
  4. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  5. Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  6. Vaswani et al. (2017). Attention Is All You Need
  7. Sutskever et al. (2014). Sequence to Sequence Learning with Neural Networks
  8. Bahdanau et al. (2014). Neural Machine Translation by Jointly Learning to Align and Translate
  9. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  10. Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
  11. Shaw, Uszkoreit, Vaswani (2018). Self-Attention with Relative Position Representations
  12. Xue et al. (2020). mT5: A massively multilingual pre-trained text-to-text transformer
  13. Chung et al. (2022). Scaling Instruction-Finetuned Language Models
  14. Tay et al. (2022). UL2: Unifying Language Learning Paradigms
  15. Wang et al. (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
  16. NLLB Team (2022). No Language Left Behind: Scaling Human-Centered Machine Translation
  17. Kim et al. (2021). OCR-free Document Understanding Transformer
  18. Li et al. (2021). TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
  19. Liu et al. (2020). Multilingual Denoising Pre-training for Neural Machine Translation