The full transformer and its three families
The last sessions built every part of a transformer block. Now we put the blocks together and see that one design gives three families: encoder-only (BERT), decoder-only (GPT, Llama, Qwen) and encoder–decoder (T5, Whisper). They differ in who may look at whom, in what they are trained to predict, and in what jobs they do well. We end with how to choose one for a task, and why decoder-only won at large scale.
What you will learn
- How a full transformer is built from one repeated block, and how the original 2017 model joined an encoder stack and a decoder stack with cross-attention
- Why the attention mask, not the block, decides the family: bidirectional, causal, prefix and encoder–decoder visibility, with exact pair counts
- Encoder-only models (BERT): masked language modeling, the 15% rule, the [CLS] vector, and why they are still used for classification and embeddings
- Decoder-only models (GPT, Llama, Qwen): the causal LM loss, why every token is a training signal, and how one stack both reads and writes
- Encoder–decoder models (T5, Whisper): a reader and a writer joined by cross-attention, for input-to-output tasks such as translation and speech
- Parameter counts per family (12d² and 16d² per block), checked against BERT-base, T5-base, GPT-3 and Llama 3 8B
- How to pick a family for a task, and the main reasons decoder-only won at scale
Why this matters
In the last seven sessions we built every part of a transformer block: attention, masks, multiple heads, cross-attention, the feed-forward network, the residual stream and normalization. This session puts the blocks together into a full model. The surprise is that one block design gives three different families of models, and the main thing that separates them is a small matrix of zeros and minus-infinities: the attention mask.
The three families are:
- Encoder-only (BERT, RoBERTa, ModernBERT). Every token sees every other token. The model is trained to fill in hidden words. It is good at understanding: classification, tagging, and embedding vectors for search.
- Decoder-only (GPT, Llama, Qwen, Gemma, DeepSeek). Each token sees only the tokens before it. The model is trained to predict the next token. It is good at generation, and at large scale it does almost everything else too. Nearly all modern LLMs and VLMs are decoder-only. [Certain]
- Encoder–decoder (the original 2017 Transformer, T5, BART, Whisper). One stack reads the input with full attention; a second stack writes the output one token at a time and reads the first stack through cross-attention. It fits input-to-output tasks: translation, summarization, speech recognition.
This choice matters in real systems. A search team that serves 10,000 queries per second needs a small, fast encoder that turns text into vectors. A chat product needs a decoder. A speech-to-text service on a phone often runs Whisper, an encoder–decoder. A video platform may use all three: a vision encoder to index frames, a text encoder for search queries, and a decoder-only VLM to write captions. If you understand the three families, you can read any new model card and place it in a few seconds.
Plan: first the full original transformer, then the mask that defines each family, then each family with its loss and real sizes, then parameter counts, and last the question of how to choose, and why decoder-only won at scale.
From one block to a full model
Recall the block from session 22. A modern block has two sublayers, each wrapped in a residual connection with a normalization in front (the Pre-LN layout). Let \(H\) be the matrix of token vectors entering the block, one row per token:
Attention lets tokens exchange information; the FFN then processes each token on its own (session 21). A full model is just this block repeated \(L\) times, with an embedding layer in front and a task-specific head (a final linear layer) at the end. Input and output of every block have the same shape \(n \times d\) (\(n\) tokens, model width \(d\)), so blocks stack like bricks.
The original Transformer (Vaswani et al., 2017) was built for translation, so it had two stacks. The encoder stack read the English sentence. The decoder stack wrote the German sentence. Each decoder block had a third sublayer between the two above: cross-attention, where the queries come from the decoder and the keys and values come from the final encoder output \(Z\) (session 20):
- Read the symbols
- \(H, H', H''\)
- the token vectors before the block, after attention, and after the FFN; each is \(n \times d\)
- \(\operatorname{Norm}\)
- LayerNorm or RMSNorm, applied to each token vector
- \(Z\)
- the output of the last encoder block, one vector per input token (\(m \times d\) for an input of \(m\) tokens)
- \(W_Q, W_K, W_V\)
- the cross-attention projection matrices; they are separate from the self-attention ones
The base model of 2017 had 6 encoder blocks and 6 decoder blocks, \(d = 512\), 8 heads, an FFN of width 2,048, and about 65 million parameters. [Certain] (The original paper put the norm after each residual add, the Post-LN layout; modern models put it before.)
Figure 1 shows the three ways to wire the block. Change the family, the width \(d\) and the depth \(L\) and watch the parameter count.
How to read Figure 1
Each tall box is a stack of \(L\) identical blocks; only one block is drawn inside. In encoder-only mode the head sits on top of every token (or on the [CLS] token for classification). In decoder-only mode the head is the LM head that scores the next token. In encoder–decoder mode the orange arrow carries the encoder output \(Z\) into the cross-attention sublayer of every decoder block. Try \(d = 768, L = 12\): about 85 million block parameters for BERT-base or GPT-2 small, and about 198 million for a T5-base-like encoder–decoder with 12 + 12 blocks. Try \(d = 12288, L = 96\) in decoder-only mode: about 174 billion, close to GPT-3’s 175B.
Figure 1 shows the wiring. Figure 2 runs it. It is a real tiny model in your browser: three tokens “the cat sat”, width \(d = 4\), one attention head, an FFN of width 8 with ReLU, RMSNorm without a gain, and a vocabulary of six words. The weights are fixed pseudo-random numbers (the model is not trained, so its guess for the next word means nothing). Follow one token and watch its vector move through formulas (23.1) and (23.2), block after block. Its attention uses the keys and values of the other tokens, which travel through the same blocks in parallel.
- \(h \leftarrow E[x_i] + P_i\) // token embedding + position
- for \(\ell = 1, \dots, L\):
- \(u \leftarrow \operatorname{Norm}(h) = h / \operatorname{rms}(h)\)
- \(a_j \leftarrow \softmax_j\big(q_i \cdot k_j / \sqrt{d} + M_{ij}\big)\); \(o \leftarrow \big(\sum_j a_j v_j\big)\, W_O\) // (23.4)
- \(h \leftarrow h + o\) // (23.1) residual add
- \(u \leftarrow \operatorname{Norm}(h)\); \(f \leftarrow \operatorname{ReLU}(u W_1)\, W_2\) // FFN, this token only
- \(h \leftarrow h + f\) // (23.2) residual add
- \(p \leftarrow \softmax\big(\operatorname{Norm}(h)\, E^\top\big)\) // LM head, tied to \(E\)
How to read Figure 2
Press Play, or use Step and Back. The orange frame marks the stage being computed; the box on the right of the code shows the formula with the live numbers. Follow “sat” with the causal mask: in block 1 its weights are 0.278, 0.167, 0.554 on “the”, “cat”, “sat”. Switch to the bidirectional mask: block 1 does not change at all, because the last token sees every token under both masks. Block 2 does change (weights 0.313, 0.108, 0.579 instead of 0.283, 0.114, 0.603), because “the” and “cat” now saw the future in block 1, so their keys and values are different. Follow “the” with the causal mask: it can only read itself, so its weight is exactly 1. Notice that the residual adds change \(h\) only a little: each sublayer writes a small update on top of the stream.
The mask decides the family
All three families use the same attention formula from session 17, with a mask matrix \(M\) added to the scores before the softmax (session 18):
Since \(e^{-\infty} = 0\), a blocked pair gets attention weight exactly 0. The families differ only in which entries are blocked:
A prefix LM is a decoder-only model where the first \(p\) tokens (the prompt) see each other in both directions, and only the rest is causal. The T5 paper and UL2 studied it. [Certain] It is a useful bridge: its visibility pattern is exactly the pattern of an encoder–decoder, as Figure 3 shows.
How many (query, key) pairs are allowed? For \(n\) tokens:
For \(n = 8\): 64, 36, and (with \(p = 4\)) 42. The causal mask throws away almost half of the pairs. That is the price of being able to generate: a token cannot use information from tokens that do not exist yet.
How to read Figure 3
Encoder-only fills all 64 cells. Decoder-only fills the lower triangle: 36 cells. Prefix LM fills a full \(4 \times 4\) square for the prompt and a triangle for the rest: 42 cells. Now press Encoder–decoder: the filled cells are the same 42 as the prefix LM, but they are colored in three parts that live in three different attention layers with different weights: encoder self-attention (16 pairs), cross-attention (16 pairs) and causal decoder self-attention (10 pairs). Note that in an encoder–decoder the source tokens never see the target tokens.
Encoder-only: BERT
BERT (Devlin et al., 2018) is a stack of encoder blocks with a bidirectional mask. Every output vector depends on the whole input, left and right. This is great for understanding (“bank” in “river bank” gets a different vector from “bank” in “bank account”), but it creates a training problem. If you train it to predict the next token, the answer is visible: the model can just look at it. So BERT uses a different task.
Masked language modeling
Masked language modeling (MLM) hides some tokens and asks the model to recover them. BERT picks 15% of the token positions. Of those, 80% are replaced by a special [MASK] token, 10% by a random token, and 10% are left unchanged. [Certain] The last two cases exist because [MASK] never appears at fine-tuning time, so the model must also learn to give good vectors for normal tokens. Let \(\mathcal{M}\) be the set of chosen positions and \(\tilde{x}\) the corrupted input:
- Read the symbols
- \(\mathcal{M}\)
- the positions picked for prediction (about 15% of all positions)
- \(\tilde{x}\)
- the whole input after the 80/10/10 corruption; the model sees all of it, both sides
- \(p_\theta(x_i \mid \tilde{x})\)
- the probability the model gives to the true token at position \(i\), from a softmax over the vocabulary
Only the chosen positions produce a loss. With 512 tokens, that is 77 predictions per sequence, not 512. Figure 5 compares this with the other families.
BERT also had a second task, next sentence prediction (NSP): is sentence B the real next sentence after A? RoBERTa (2019) showed it was not needed and dropped it, training longer on more data instead. [Certain]
Using an encoder
BERT adds a special [CLS] token at the start and [SEP] between and after sentences. After fine-tuning, the final vector of [CLS] (or the mean of all token vectors) is used for whole-text tasks such as sentiment. Each token’s own vector is used for token tasks such as named-entity tagging. For search, two texts are turned into vectors and compared with cosine similarity (Sentence-BERT, E5; see session 60). An encoder does all of this in one forward pass, with no generation loop. Deeper details are in session 33.
| Model | Blocks | \(d\) | Heads | FFN width | Parameters | Context |
|---|---|---|---|---|---|---|
| BERT-base | 12 | 768 | 12 | 3,072 | 110M | 512 |
| BERT-large | 24 | 1,024 | 16 | 4,096 | 340M | 512 |
| ModernBERT-base (2024) | 22 | 768 | 12 | 1,152 (gated) | about 150M [Likely] | 8,192 |
Sizes come from the papers and the Hugging Face config.json files. [Certain] BERT uses a WordPiece vocabulary of 30,522 tokens and was pretrained on BooksCorpus (800M words) and English Wikipedia (2,500M words). [Certain]
Embeddings: \(30{,}522 \times 768 = 23{,}440{,}896\) for tokens, plus \(512 \times 768\) for positions and \(2 \times 768\) for the two segment types.
One block, exactly: four attention matrices \(4 \times (768^2 + 768)\), the FFN \(768 \times 3{,}072 + 3{,}072 + 3{,}072 \times 768 + 768\), and two LayerNorms \(4 \times 768\). Total \(7{,}087{,}872\). The shortcut \(12d^2 = 7{,}077{,}888\) is within 0.15%.
Twelve blocks plus embeddings plus the small pooler (\(768^2 + 768\)) give \(109{,}482{,}240\): the “110M” of the paper. The token table alone is 21% of the model, a large share for a small model.
Decoder-only: GPT, Llama, Qwen
A decoder-only model is a stack of blocks with a causal mask and no cross-attention. GPT-1 (2018) was the first well-known one, a few months before BERT. [Certain] It is trained with the causal language modeling loss from session 6: predict each token from the ones before it.
Two properties make this loss powerful.
- Every position is a training example. Thanks to the causal mask, the output at position \(t - 1\) cannot see \(x_t\), so it can be trained to predict it. One forward pass over \(n\) tokens gives \(n - 1\) predictions, all in parallel. With 512 tokens: 511 predictions, against 77 for BERT.
- Training and use are the same task. At inference, the model generates by predicting a token, appending it and repeating. Because past tokens never look at future ones, their keys and values never change, so they can be stored in a KV cache and reused; each new token costs one block pass, not a re-read of everything (session 80).
The price is that each token’s vector is built only from its left context. In a decoder, the vector for “bank” in “the bank of the river” cannot use “river”. For understanding tasks, this is a real weakness at small scale. At large scale, the model works around it: it reads the whole prompt first, and the last token’s vector has seen everything.
| Model | Blocks | \(d\) | Query heads / KV heads | FFN width | Vocabulary | Parameters |
|---|---|---|---|---|---|---|
| GPT-2 small (2019) | 12 | 768 | 12 / 12 | 3,072 | 50,257 | 124M |
| GPT-3 (2020) | 96 | 12,288 | 96 / 96 | 49,152 | 50,257 [Likely] | 175B |
| Llama 3.1 8B (2024) | 32 | 4,096 | 32 / 8 | 14,336 (gated) | 128,256 | 8.03B |
| Qwen2.5-7B (2024) | 28 | 3,584 | 28 / 4 | 18,944 (gated) | 152,064 | 7.62B |
Numbers from the GPT-3 paper and the config.json files; the Llama and Qwen totals are computed from their configs. [Certain] Modern decoders change the block in small ways (RMSNorm, RoPE, SwiGLU, grouped-query attention with fewer KV heads; see session 26), but the family is the same as GPT-1.
With \(12d^2\) per block (formula 23.10 below): \(12 \times 96 \times 12{,}288^2 = 173{,}946{,}175{,}488\), about 174B. The token embedding adds \(50{,}257 \times 12{,}288 \approx 0.62\)B. Together about 174.6B, the “175B” of the paper. At this size, the blocks are 99.6% of the weights.
Encoder–decoder: T5 and Whisper
An encoder–decoder keeps both stacks of the 2017 design. The encoder reads the input \(x\) (\(m\) tokens) once, with a bidirectional mask, and produces \(Z\). The decoder generates the output \(y\) with a causal mask, and each of its blocks reads \(Z\) by cross-attention (23.3). The loss is the causal loss on the output only, given the whole input:
During training the decoder gets the target shifted right by one (a start token in front) and predicts all \(T\) target tokens in parallel; this is teacher forcing, the same trick as in session 7.
The generation loop, with cross-attention
At inference there is no target to shift, so the decoder writes one token per step. The key point of the data flow: the encoder runs once, and the cross-attention keys and values \(K = ZW_K\), \(V = ZW_V\) are computed once and reused by every decoding step. Only the query changes from step to step. Figure 4 runs this loop on a toy translator with hand-set weights, so that you can see each number. Each source vector \(z_j\) holds the word (a one-hot over 7 source words) and its position \(j\) as an angle, \((\cos j\theta, \sin j\theta)\) with \(\theta = 36°\). \(W_K\) keeps only the position, so \(k_j = (\cos j\theta, \sin j\theta)\); \(W_V\) keeps only the word. At step \(t\) the query is \(q_t = \kappa(\cos t\theta, \sin t\theta)\), so the score is
which is largest when \(j = t\). The context \(c_t = \sum_j a_{tj} v_j\) is then mostly the word at position \(t\), and the LM head (logits \(= 6\, c_t\), a word-to-word dictionary) turns it into the French word. A real model learns all of this, including word order and agreement; the toy only translates word by word, and its decoder skips the causal self-attention sublayer because it does not need it.
- \(Z \leftarrow \operatorname{Encoder}(x)\) // bidirectional, runs once
- \(K \leftarrow Z W_K\); \(V \leftarrow Z W_V\) // once, cached for all steps
- \(y_0 \leftarrow \langle s \rangle\); for \(t = 1, 2, \dots\):
- \(q_t \leftarrow\) query of the decoder at step \(t\)
- \(s_{tj} \leftarrow q_t \cdot k_j / \sqrt{d_k}\) for every \(j\) // no mask on the source
- \(a_t \leftarrow \softmax(s_t)\); \(c_t \leftarrow \sum_j a_{tj} v_j\) // (23.3)
- \(p \leftarrow \softmax(6\, c_t)\); \(y_t \leftarrow \argmax p\)
- if \(y_t = \langle /s \rangle\): stop
How to read Figure 4
Press Play. Steps 1 and 2 run once: the encoder and the K, V cache. Then each decoding step has four frames: the query, the scores, the weights and context, and the pick. With \(\kappa = 20\) at step 2, the scores are 11.441, 14.142, 11.441, 4.370, −4.370, the weights are 0.059, 0.882, 0.059, 0.000, 0.000, and “chat” gets probability 0.967. Lower \(\kappa\) to 4: the attention spreads over neighbors (0.250, 0.429, 0.250, …) and the probability of “chat” falls to 0.493. The counter at the bottom shows the point of the cache: 5 encoder token passes in total, while re-running the encoder at every step would cost 5 per step.
T5: every task is text-to-text
T5 (Raffel et al., 2019) casts every task as “text in, text out”: the input translate English to German: That is good. gives the output Das ist gut.; sst2 sentence: a lovely film gives positive. Pretraining uses span corruption: about 15% of tokens are removed in short spans (mean length 3), each span is replaced by one sentinel token, and the decoder writes the missing spans, each after its sentinel. [Certain] Sizes go from 60M to 11B. [Certain] T5-base has 12 + 12 blocks with \(d = 768\), about 220M parameters. Session 24 covers span corruption and BART’s denoising in detail.
Whisper: audio in, text out
Whisper (Radford et al., 2022) is an encoder–decoder for speech, trained on 680,000 hours of audio with transcripts. [Certain] The audio is cut into 30-second chunks and turned into a log-Mel spectrogram (a picture of sound energy over time and frequency). Two small convolution layers turn it into 1,500 frame vectors, which the encoder reads with full attention. The decoder then writes the text (up to 448 tokens), using cross-attention to look at the right part of the audio for each word. [Certain] Whisper large-v3 has 32 encoder and 32 decoder blocks with \(d = 1{,}280\), about 1.55B parameters. [Certain]
Whisper shows the real strength of this family: the input and the output can be different kinds of things (sound and text, or two languages) with very different lengths. The encoder can be shaped for the input (here, convolutions over audio), and the decoder for the output.
How the training signal differs
The three losses (23.7), (23.8) and (23.9) use the data very differently. Figure 5 shows a 24-token sentence and which positions produce a prediction under each objective.
How to read Figure 5
In masked LM mode, only the orange tokens are predicted: 4 of 24 at 15%. In causal mode, every token except the first is a target: 23 of 24, and the corruption slider has no effect. In span-corruption mode, the input shrinks (each span becomes one sentinel) and the decoder target is short: the hidden tokens, one sentinel per span, and an end token. Raise the rate and watch the masked-LM count grow; but the more you hide, the less context is left to predict from.
Masked LM at 15%: \(\operatorname{round}(0.15 \times 512) = 77\) predicted tokens.
Causal LM: \(512 - 1 = 511\) predicted tokens, 6.6 times more per forward pass.
T5 span corruption at 15%, mean span 3: 77 noise tokens in \(\operatorname{round}(77/3) = 26\) spans. The encoder input becomes \(512 - 77 + 26 + 1 = 462\) tokens (each span replaced by a sentinel, plus an end token). The decoder target is \(77 + 26 + 1 = 104\) tokens (the noise tokens, a sentinel before each span, and an end token).
So per token of compute, causal LM gets the most direct supervision. This is not the whole story (an MLM prediction uses context from both sides, so each one can be more informative), but it is one reason decoders learn efficiently from huge text collections. [Likely]
Counting parameters per family
With FFN width \(4d\) and ignoring biases and norms, self-attention has four \(d \times d\) matrices (\(4d^2\)) and the FFN has two \(d \times 4d\) matrices (\(8d^2\)). Cross-attention adds four more \(d \times d\) matrices:
So for the same \(d\) and \(L\) blocks per stack, an encoder–decoder has \(L(12 + 16)d^2 = 28 L d^2\) block parameters, \(2.33\) times a decoder-only model. Comparisons between families must say what is held fixed: total parameters, or compute per token. An encoder–decoder with \(L\) + \(L\) blocks has about twice the parameters of an \(L\)-block decoder, but each token passes through only one of the two stacks, so the compute per token is similar. [Certain] The exact counting, including embeddings, GQA and SwiGLU, is the topic of session 25.
| Model | Formula | Block parameters | Reported total |
|---|---|---|---|
| BERT-base (enc, \(d = 768\), \(L = 12\)) | \(12 L d^2\) | 84.9M | 110M (with 23.8M of embeddings) |
| T5-base (enc–dec, 12 + 12) | \(28 L d^2\) | 198.2M | about 220M (computed 222.9M with the 32,128-token table) |
| Whisper large-v3 (32 + 32, \(d = 1{,}280\)) | \(28 L d^2\) | 1.47B | 1.55B |
| GPT-3 (dec, \(d = 12{,}288\), \(L = 96\)) | \(12 L d^2\) | 173.9B | 175B |
The three families as algorithms
- input: token IDs \(x_1..x_n\) (and target IDs \(y_1..y_T\) for encoder–decoder)
- \(H \leftarrow \operatorname{Embed}(x) + \text{positions}\)
- if encoder-only: for \(\ell = 1..L\): \(H \leftarrow \operatorname{Block}(H, \text{mask} = \text{none})\); return \(\operatorname{Head}(H)\) // one vector per token, or [CLS]
- if decoder-only: for \(\ell = 1..L\): \(H \leftarrow \operatorname{Block}(H, \text{mask} = \text{causal})\); return \(\operatorname{LMHead}(H)\) // row t scores token t+1
- if encoder–decoder:
- \(Z \leftarrow H\); for \(\ell = 1..L_{\text{enc}}\): \(Z \leftarrow \operatorname{Block}(Z, \text{mask} = \text{none})\) // run once
- \(G \leftarrow \operatorname{Embed}(\langle s \rangle, y_1..y_{T-1})\) // target shifted right
- for \(\ell = 1..L_{\text{dec}}\): \(G \leftarrow \operatorname{Block}(G, \text{mask} = \text{causal}, \text{cross} = Z)\) // formula (23.3) in every block
- return \(\operatorname{LMHead}(G)\)
- decoder-only: put the input and the answer in one sequence
- prefill: run the whole prompt once, store K and V of every block in the cache
- repeat: take the last row of logits, pick a token, run only that token through the blocks, append its K, V to the cache
- encoder–decoder: keep the input and the answer in two sequences
- run the encoder once to get \(Z\); compute the cross-attention K, V from \(Z\) once per decoder block
- start the decoder with \(\langle s \rangle\); repeat: run one token through the decoder blocks (causal self-attention with its own cache, then cross-attention to \(Z\)), pick the next token
- encoder-only: no loop; one forward pass gives all outputs // it cannot write new text by itself
The PyTorch version below builds all three families from one Block class and checks the key property of each: changing the last input token changes the first output of the encoder and of the encoder–decoder, but not of the decoder.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) class Block(nn.Module): """One Pre-LN block. causal: mask the future. cross: add cross-attention to an encoder output.""" def __init__(self, d, h, causal=False, cross=False): super().__init__() self.causal = causal self.n1, self.att = nn.LayerNorm(d), nn.MultiheadAttention(d, h, batch_first=True) self.cross = cross if cross: self.nc, self.xatt = nn.LayerNorm(d), nn.MultiheadAttention(d, h, batch_first=True) self.n2 = nn.LayerNorm(d) self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) def forward(self, x, mem=None): n = x.shape[1] mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1) if self.causal else None # True = blocked a = self.n1(x) x = x + self.att(a, a, a, attn_mask=mask, need_weights=False)[0] # (23.1) if self.cross: c = self.nc(x) x = x + self.xatt(c, mem, mem, need_weights=False)[0] # (23.3): Q from x, K and V from mem return x + self.ffn(self.n2(x)) # (23.2) class Stack(nn.Module): def __init__(self, V, d, h, L, causal, cross=False, lm_head=True): super().__init__() self.emb, self.pos = nn.Embedding(V, d), nn.Embedding(128, d) self.blocks = nn.ModuleList(Block(d, h, causal, cross) for _ in range(L)) self.norm = nn.LayerNorm(d) self.head = nn.Linear(d, V, bias=False) if lm_head else None def forward(self, ids, mem=None): x = self.emb(ids) + self.pos(torch.arange(ids.shape[1])) for b in self.blocks: x = b(x, mem) x = self.norm(x) return self.head(x) if self.head is not None else x V, d, h, L = 100, 64, 4, 2 encoder = Stack(V, d, h, L, causal=False) # BERT-like: logits for masked positions decoder = Stack(V, d, h, L, causal=True) # GPT-like ed_enc = Stack(V, d, h, L, causal=False, lm_head=False) # T5-like: encoder gives vectors ... ed_dec = Stack(V, d, h, L, causal=True, cross=True) # ... the decoder reads them by cross-attention for m in (encoder, decoder, ed_enc, ed_dec): m.eval() x = torch.randint(0, V, (1, 10)) y = torch.randint(0, V, (1, 6)) print(encoder(x).shape, decoder(x).shape, ed_dec(y, ed_enc(x)).shape) # Who sees the future? Change only the last input token and compare the outputs at position 0. x2 = x.clone(); x2[0, -1] = (x[0, -1] + 1) % V with torch.no_grad(): print("encoder pos 0 changed:", not torch.allclose(encoder(x)[0, 0], encoder(x2)[0, 0])) # True print("decoder pos 0 changed:", not torch.allclose(decoder(x)[0, 0], decoder(x2)[0, 0])) # False # In the encoder-decoder, EVERY target position sees the whole source (cross-attention is not causal) print("enc-dec pos 0 changed:", not torch.allclose(ed_dec(y, ed_enc(x))[0, 0], ed_dec(y, ed_enc(x2))[0, 0])) # True # Parameters inside the blocks: about 12 d^2 per block, 16 d^2 with cross-attention (23.10) per = lambda s: sum(p.numel() for p in s.blocks[0].parameters()) print(per(decoder), 12 * d * d, per(ed_dec), 16 * d * d) # 49984 49152 66752 65536 (the rest: biases, norms) # The three losses: masked LM (23.7), causal LM (23.8), sequence-to-sequence (23.9) MASK = 0 pos = torch.tensor([2, 7]) # positions we hide xm = x.clone(); xm[0, pos] = MASK loss_mlm = F.cross_entropy(encoder(xm)[0, pos], x[0, pos]) loss_clm = F.cross_entropy(decoder(x)[0, :-1], x[0, 1:]) # position t predicts token t+1 y_in = torch.cat([torch.zeros(1, 1, dtype=torch.long), y[:, :-1]], 1) # shift right, start token 0 loss_s2s = F.cross_entropy(ed_dec(y_in, ed_enc(x))[0], y[0]) print([round(v.item(), 3) for v in (loss_mlm, loss_clm, loss_s2s)], round(torch.log(torch.tensor(V * 1.0)).item(), 3))
With PyTorch 2.8 on CPU this prints the three shapes [1, 10, 100], [1, 10, 100], [1, 6, 100], then True, False, True, then 49984 49152 66752 65536. The untrained losses are about 3.4, 4.8 and 4.8, near \(\ln 100 = 4.6\) as expected for random guessing (the masked loss is an average over only 2 positions, so it is noisier). In a real decoder-only model you would use F.scaled_dot_product_attention(..., is_causal=True), which applies the same mask with a fast kernel.
Choosing a family, and why decoder-only won
A practical guide
- You need a fixed-size output for each input (a label, a score, a vector) and low cost per request: start with an encoder. A 100M-parameter encoder answers in one pass and is cheap to fine-tune. Typical: spam and intent classification, named entities, reranking, embeddings for search.
- You need to write open-ended text, follow instructions, chat, write code, use tools, or do many tasks with one model: use a decoder-only LLM. Prompting replaces task-specific training.
- You map one well-defined input to one output, the input is long or of another kind (audio, another language), and you train for that task: an encoder–decoder is often the most efficient choice, as Whisper and many production translation systems show. [Likely]
How to read Figure 6
Look at the reason line. Encoders win when the output is small and fixed and latency or cost per request matter. Decoders win when the output is open-ended text or when one model must do many tasks. Encoder–decoders win when a different kind of input (sound, a long source text) must be turned into text. The video tasks show that most video systems combine families: an encoder for indexing, a decoder for writing.
Why decoder-only won at scale
Around 2019, encoders ruled understanding benchmarks and encoder–decoders (T5) did well on many tasks. By 2023 almost every frontier LLM was decoder-only. Several reasons work together:
- One objective that uses all the data. Next-token prediction needs no labels and gives a loss at every position (Figure 5). Raw text from the web is the training set.
- Generation is built in, and generation can do every task. GPT-2 and GPT-3 showed that a large enough decoder can do translation, question answering and classification just from a prompt, with no new head or fine-tuning (few-shot, in-context learning). [Certain] An encoder cannot write; an encoder–decoder needs the task split into “input” and “output”.
- Simplicity scales. One stack, one mask, one loss, one set of weights for prompt and answer. Every tool (parallelism, kernels, quantization, KV caching, serving engines like vLLM) is built for this one shape. [Likely]
- Chat fits naturally. A conversation is one growing sequence. A decoder caches all earlier turns and only processes the new ones. In an encoder–decoder, a growing input must be re-encoded because the encoder is bidirectional: adding a token changes every earlier encoder vector. [Certain]
- Zero-shot results. Wang et al. (2022) trained models of over 5B parameters on over 170B tokens with each architecture. After pure self-supervised pretraining, the causal decoder trained on next-token prediction had the best zero-shot generalization; after multitask fine-tuning, a model with non-causal (bidirectional) input visibility and a masked objective did best. [Certain] So the answer depends on the stage, but the “no extra fine-tuning needed” setting favors the decoder.
The bidirectional advantage did not disappear. It moved: modern decoders read the full prompt before answering, so the last position has seen everything, and at billions of parameters this is enough for most understanding tasks. Encoders are still the best value at small sizes and are used heavily for embeddings, classification and reranking; ModernBERT (2024) is a recent example. [Certain] And many new embedding models are fine-tuned from decoder-only LLMs. [Likely]
What about vision-language models?
A typical VLM (session 177) mixes families. The image goes through a vision encoder (a ViT or SigLIP, which is an encoder-only transformer over image patches). A small connector turns the patch vectors into tokens, and a decoder-only LLM reads them as part of its prompt and writes the answer (LLaVA, Qwen2.5-VL). [Certain] Flamingo instead added cross-attention layers into a frozen decoder, so the text side reads image features the way an encoder–decoder reads \(Z\). [Certain] Speech works the same way: Whisper’s encoder is often reused as the audio encoder of omni models. [Likely] The book covers these designs in chapter 6 (Noyan et al., Vision Language Models).
Where this is used
| Application | Usual family | Why |
|---|---|---|
| Sentiment, intent, spam, toxicity classification | Encoder (BERT-like), fine-tuned | One pass, small model, fixed label set, very high throughput |
| Semantic search and RAG retrieval | Encoder (bi-encoder embeddings), cross-encoder reranker | Vectors are computed once per document; bidirectional context gives good vectors (session 150) |
| Chat assistants, code completion, agents | Decoder-only | Open-ended generation, prompting, KV cache across turns |
| Machine translation, summarization | Encoder–decoder or decoder-only | Clear input-to-output task; large decoders now do both well |
| Speech recognition | Encoder–decoder (Whisper) | Audio in, text out; the encoder is shaped for audio |
| Text-to-video retrieval | Two encoders (video encoder and text encoder, CLIP-style) | Index millions of clips once as vectors; queries need only a fast text pass (session 208) |
| Action recognition, video classification | Encoder (video transformer) with a classification head | Fixed label set over a clip; bidirectional attention over space and time |
| Key-moment detection, temporal grounding | Encoder over frame features, sometimes a decoder-only Video-LLM | Scores per frame need full context; a Video-LLM can answer “when does X happen?” in text |
| Video fingerprinting, near-duplicate detection | Encoder (frame or clip embeddings) | Compact vectors compared at scale; no text output needed |
| Video captioning, video question answering, Video-RAG | Vision encoder + decoder-only LLM | Must write free text about what is seen (session 205) |
Common mistakes
- Mixing up the two meanings of “decoder”. A GPT block has no cross-attention. A T5 decoder block does. Count parameters accordingly (12d² vs 16d²).
- Thinking BERT can generate text like GPT. BERT has no causal mask and no next-token training. You can make it fill in
[MASK]slots, but it has no natural left-to-right generation loop. - Using a decoder’s first-token or mean vector as a sentence embedding without fine-tuning. In a causal model the first token has seen only itself; the last token is the one that has seen the whole input.
- Forgetting the shift. In causal LM, position \(t\) predicts token \(t+1\): logits
[:, :-1]against labels[:, 1:]. In an encoder–decoder, the decoder input is the target shifted right with a start token. - Computing MLM loss on all positions. Only the chosen ~15% get a loss; the others are ignored (label −100 in Hugging Face).
- Comparing families at unequal budgets. “T5 beat GPT” or the reverse means little unless parameters, compute per token and training tokens are stated.
- Re-running the encoder at every decoding step. In an encoder–decoder, the encoder output and the cross-attention keys and values are computed once per input, not once per generated token.
Interview questions
What is the difference between BERT and GPT?basic
What exactly makes a model encoder-only, decoder-only or encoder–decoder?basic
Why does BERT mask only 15% of tokens, and why the 80/10/10 split?core
[MASK] never appears in real inputs, so the model must also produce useful vectors for ordinary tokens and cannot assume that only [MASK] positions matter.Why can a decoder train on all positions in parallel but must generate one token at a time?core
How many parameters does one transformer block have? With cross-attention?core
Why are almost all modern LLMs decoder-only?core
When would you still pick an encoder–decoder?core
What is a prefix LM and how does it relate to an encoder–decoder?advanced
Your product needs sentence embeddings for search over 100 million documents. Which family, and why?advanced
Why must an encoder–decoder re-encode a growing input, while a decoder can cache it?advanced
Exercises
- Count the allowed (query, key) pairs for \(n = 10\) under a bidirectional mask, a causal mask, and a prefix mask with \(p = 6\).
- An encoder–decoder has 6 encoder and 6 decoder blocks with \(d = 512\) and FFN width \(4d\). Estimate its block parameters with formula (23.10). Compare with the 2017 base model’s 65M.
- For a 1,024-token sequence, how many tokens get a loss under masked LM at 15%, causal LM, and T5 span corruption at 15% with mean span 3? How long is the T5 decoder target?
- Llama 3.1 8B has \(d = 4096\), 32 query heads and 8 KV heads of size 128, and a SwiGLU FFN of width 14,336. Compute the parameters of one block (ignore norms).
- A chat has 2,000 tokens of history and the user adds 50 tokens. How many input tokens must the decoder process for the new turn (with a KV cache)? How many would a bidirectional encoder have to re-encode?
- Why does using the vector of the first token of a decoder-only model as a sentence embedding work poorly? Which position would you use instead?
Answerscheck
Key takeaways
- A full transformer is one block (23.1, 23.2) repeated \(L\) times, plus embeddings and a head. The 2017 original was an encoder–decoder, joined by cross-attention (23.3).
- The attention mask (23.4, 23.5) defines the family: bidirectional (\(n^2\) pairs), causal (\(n(n+1)/2\)), or prefix; an encoder–decoder has the prefix pattern split over three attention layers.
- Encoder-only (BERT) learns by masked LM (23.7), 15% of positions, and gives strong vectors in one pass: classification, tagging, embeddings.
- Decoder-only (GPT, Llama, Qwen) learns by next-token prediction (23.8) on every position, generates with a KV cache, and at scale does most tasks by prompting.
- Encoder–decoder (T5, Whisper) maps an input to an output (23.9): best for translation, summarization and speech, where the input is long or of another kind.
- Blocks cost about \(12d^2\), or \(16d^2\) with cross-attention (23.10). Decoder-only won through one simple objective, universal prompting, easy scaling and natural chat caching.
Key formulas
References for this session
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Brown et al. (2020). Language Models are Few-Shot Learners
- Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 6: Transformers
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 7: Transformers and Pretraining
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Brown et al. (2020). Language Models are Few-Shot Learners
- Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 6: Transformers
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 7: Transformers and Pretraining
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 9: Masked Language Models
- Vaswani et al. (2017). Attention Is All You Need
- Radford, Narasimhan, Salimans, Sutskever (2018). Improving Language Understanding by Generative Pre-Training
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners
- Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Qwen Team (2024). Qwen2.5 Technical Report
- Warner et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
- Wang, Roberts, Hesslow, Le Scao, Chung, Beltagy, Launay, Raffel (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
- Tay et al. (2022). UL2: Unifying Language Learning Paradigms
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language