LLM & VLM courseBlog
23Track 2: Attention and transformers · session 8 of 15

The full transformer and its three families

The last sessions built every part of a transformer block. Now we put the blocks together and see that one design gives three families: encoder-only (BERT), decoder-only (GPT, Llama, Qwen) and encoder–decoder (T5, Whisper). They differ in who may look at whom, in what they are trained to predict, and in what jobs they do well. We end with how to choose one for a task, and why decoder-only won at large scale.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 16 to 22 (attention, masks, cross-attention, the FFN, the residual stream)

What you will learn

  • How a full transformer is built from one repeated block, and how the original 2017 model joined an encoder stack and a decoder stack with cross-attention
  • Why the attention mask, not the block, decides the family: bidirectional, causal, prefix and encoder–decoder visibility, with exact pair counts
  • Encoder-only models (BERT): masked language modeling, the 15% rule, the [CLS] vector, and why they are still used for classification and embeddings
  • Decoder-only models (GPT, Llama, Qwen): the causal LM loss, why every token is a training signal, and how one stack both reads and writes
  • Encoder–decoder models (T5, Whisper): a reader and a writer joined by cross-attention, for input-to-output tasks such as translation and speech
  • Parameter counts per family (12d² and 16d² per block), checked against BERT-base, T5-base, GPT-3 and Llama 3 8B
  • How to pick a family for a task, and the main reasons decoder-only won at scale

Why this matters

In the last seven sessions we built every part of a transformer block: attention, masks, multiple heads, cross-attention, the feed-forward network, the residual stream and normalization. This session puts the blocks together into a full model. The surprise is that one block design gives three different families of models, and the main thing that separates them is a small matrix of zeros and minus-infinities: the attention mask.

The three families are:

  • Encoder-only (BERT, RoBERTa, ModernBERT). Every token sees every other token. The model is trained to fill in hidden words. It is good at understanding: classification, tagging, and embedding vectors for search.
  • Decoder-only (GPT, Llama, Qwen, Gemma, DeepSeek). Each token sees only the tokens before it. The model is trained to predict the next token. It is good at generation, and at large scale it does almost everything else too. Nearly all modern LLMs and VLMs are decoder-only. [Certain]
  • Encoder–decoder (the original 2017 Transformer, T5, BART, Whisper). One stack reads the input with full attention; a second stack writes the output one token at a time and reads the first stack through cross-attention. It fits input-to-output tasks: translation, summarization, speech recognition.

This choice matters in real systems. A search team that serves 10,000 queries per second needs a small, fast encoder that turns text into vectors. A chat product needs a decoder. A speech-to-text service on a phone often runs Whisper, an encoder–decoder. A video platform may use all three: a vision encoder to index frames, a text encoder for search queries, and a decoder-only VLM to write captions. If you understand the three families, you can read any new model card and place it in a few seconds.

Plan: first the full original transformer, then the mask that defines each family, then each family with its loss and real sizes, then parameter counts, and last the question of how to choose, and why decoder-only won at scale.

From one block to a full model

Recall the block from session 22. A modern block has two sublayers, each wrapped in a residual connection with a normalization in front (the Pre-LN layout). Let \(H\) be the matrix of token vectors entering the block, one row per token:

\[ H' = H + \operatorname{SelfAttn}\big(\operatorname{Norm}(H)\big) \tag{23.1} \]
\[ H'' = H' + \operatorname{FFN}\big(\operatorname{Norm}(H')\big) \tag{23.2} \]

Attention lets tokens exchange information; the FFN then processes each token on its own (session 21). A full model is just this block repeated \(L\) times, with an embedding layer in front and a task-specific head (a final linear layer) at the end. Input and output of every block have the same shape \(n \times d\) (\(n\) tokens, model width \(d\)), so blocks stack like bricks.

The original Transformer (Vaswani et al., 2017) was built for translation, so it had two stacks. The encoder stack read the English sentence. The decoder stack wrote the German sentence. Each decoder block had a third sublayer between the two above: cross-attention, where the queries come from the decoder and the keys and values come from the final encoder output \(Z\) (session 20):

\[ H' \leftarrow H' + \operatorname{CrossAttn}\big(Q = \operatorname{Norm}(H') W_Q,\; K = Z W_K,\; V = Z W_V\big) \tag{23.3} \]
Read the symbols
\(H, H', H''\)
the token vectors before the block, after attention, and after the FFN; each is \(n \times d\)
\(\operatorname{Norm}\)
LayerNorm or RMSNorm, applied to each token vector
\(Z\)
the output of the last encoder block, one vector per input token (\(m \times d\) for an input of \(m\) tokens)
\(W_Q, W_K, W_V\)
the cross-attention projection matrices; they are separate from the self-attention ones

The base model of 2017 had 6 encoder blocks and 6 decoder blocks, \(d = 512\), 8 heads, an FFN of width 2,048, and about 65 million parameters. [Certain] (The original paper put the norm after each residual add, the Post-LN layout; modern models put it before.)

Figure 1 shows the three ways to wire the block. Change the family, the width \(d\) and the depth \(L\) and watch the parameter count.

self-attention (bidirectional or causal)cross-attention (reads the encoder)feed-forward networknorm and residual add
Figure 1. The three transformer families built from the same block. The parameter count uses formula (23.10): about \(12d^2\) per encoder or decoder-only block and \(16d^2\) per decoder block that has cross-attention, with an FFN of width \(4d\). Embeddings, norms and biases are not counted.
How to read Figure 1

Each tall box is a stack of \(L\) identical blocks; only one block is drawn inside. In encoder-only mode the head sits on top of every token (or on the [CLS] token for classification). In decoder-only mode the head is the LM head that scores the next token. In encoder–decoder mode the orange arrow carries the encoder output \(Z\) into the cross-attention sublayer of every decoder block. Try \(d = 768, L = 12\): about 85 million block parameters for BERT-base or GPT-2 small, and about 198 million for a T5-base-like encoder–decoder with 12 + 12 blocks. Try \(d = 12288, L = 96\) in decoder-only mode: about 174 billion, close to GPT-3’s 175B.

Figure 1 shows the wiring. Figure 2 runs it. It is a real tiny model in your browser: three tokens “the cat sat”, width \(d = 4\), one attention head, an FFN of width 8 with ReLU, RMSNorm without a gain, and a vocabulary of six words. The weights are fixed pseudo-random numbers (the model is not trained, so its guess for the next word means nothing). Follow one token and watch its vector move through formulas (23.1) and (23.2), block after block. Its attention uses the keys and values of the other tokens, which travel through the same blocks in parallel.

residual stream \(h\)normattentionFFNresidual addstage being computed
  1. \(h \leftarrow E[x_i] + P_i\) // token embedding + position
  2. for \(\ell = 1, \dots, L\):
  3. \(u \leftarrow \operatorname{Norm}(h) = h / \operatorname{rms}(h)\)
  4. \(a_j \leftarrow \softmax_j\big(q_i \cdot k_j / \sqrt{d} + M_{ij}\big)\); \(o \leftarrow \big(\sum_j a_j v_j\big)\, W_O\) // (23.4)
  5. \(h \leftarrow h + o\) // (23.1) residual add
  6. \(u \leftarrow \operatorname{Norm}(h)\); \(f \leftarrow \operatorname{ReLU}(u W_1)\, W_2\) // FFN, this token only
  7. \(h \leftarrow h + f\) // (23.2) residual add
  8. \(p \leftarrow \softmax\big(\operatorname{Norm}(h)\, E^\top\big)\) // LM head, tied to \(E\)
Figure 2. One token’s vector \(h\) (four numbers, drawn as four bars) through a full Pre-LN model, computed live. Each row is one block: Norm, attention, residual add, Norm, FFN, residual add. The panel below shows the attention weights of the followed token in the current block, and at the end the next-word probabilities from the LM head.
How to read Figure 2

Press Play, or use Step and Back. The orange frame marks the stage being computed; the box on the right of the code shows the formula with the live numbers. Follow “sat” with the causal mask: in block 1 its weights are 0.278, 0.167, 0.554 on “the”, “cat”, “sat”. Switch to the bidirectional mask: block 1 does not change at all, because the last token sees every token under both masks. Block 2 does change (weights 0.313, 0.108, 0.579 instead of 0.283, 0.114, 0.603), because “the” and “cat” now saw the future in block 1, so their keys and values are different. Follow “the” with the causal mask: it can only read itself, so its weight is exactly 1. Notice that the residual adds change \(h\) only a little: each sublayer writes a small update on top of the stream.

The mask decides the family

All three families use the same attention formula from session 17, with a mask matrix \(M\) added to the scores before the softmax (session 18):

\[ \operatorname{Attn}(Q, K, V) = \softmax\!\left( \frac{Q K^\top}{\sqrt{d_k}} + M \right) V, \qquad M_{ij} = \begin{cases} 0 & \text{if token } i \text{ may see token } j \\ -\infty & \text{otherwise} \end{cases} \tag{23.4} \]

Since \(e^{-\infty} = 0\), a blocked pair gets attention weight exactly 0. The families differ only in which entries are blocked:

\[ \begin{aligned} \text{bidirectional (encoder):}\quad & M_{ij} = 0 \ \text{ for all } i, j \\ \text{causal (decoder):}\quad & M_{ij} = 0 \ \text{ only if } j \le i \\ \text{prefix LM, prefix length } p:\quad & M_{ij} = 0 \ \text{ if } j \le p \ \text{ or } j \le i \end{aligned} \tag{23.5} \]

A prefix LM is a decoder-only model where the first \(p\) tokens (the prompt) see each other in both directions, and only the rest is causal. The T5 paper and UL2 studied it. [Certain] It is a useful bridge: its visibility pattern is exactly the pattern of an encoder–decoder, as Figure 3 shows.

How many (query, key) pairs are allowed? For \(n\) tokens:

\[ \text{bidirectional: } n^2, \qquad \text{causal: } \frac{n(n+1)}{2}, \qquad \text{prefix: } p^2 + \frac{n(n+1)}{2} - \frac{p(p+1)}{2} \tag{23.6} \]

For \(n = 8\): 64, 36, and (with \(p = 4\)) 42. The causal mask throws away almost half of the pairs. That is the price of being able to generate: a token cannot use information from tokens that do not exist yet.

Figure 3. Who may look at whom. Rows are the tokens that ask (queries), columns are the tokens they read (keys). A filled cell means “allowed”. Click a row label or a cell to pick a query token.
How to read Figure 3

Encoder-only fills all 64 cells. Decoder-only fills the lower triangle: 36 cells. Prefix LM fills a full \(4 \times 4\) square for the prompt and a triangle for the rest: 42 cells. Now press Encoder–decoder: the filled cells are the same 42 as the prefix LM, but they are colored in three parts that live in three different attention layers with different weights: encoder self-attention (16 pairs), cross-attention (16 pairs) and causal decoder self-attention (10 pairs). Note that in an encoder–decoder the source tokens never see the target tokens.

Encoder-only: BERT

BERT (Devlin et al., 2018) is a stack of encoder blocks with a bidirectional mask. Every output vector depends on the whole input, left and right. This is great for understanding (“bank” in “river bank” gets a different vector from “bank” in “bank account”), but it creates a training problem. If you train it to predict the next token, the answer is visible: the model can just look at it. So BERT uses a different task.

Masked language modeling

Masked language modeling (MLM) hides some tokens and asks the model to recover them. BERT picks 15% of the token positions. Of those, 80% are replaced by a special [MASK] token, 10% by a random token, and 10% are left unchanged. [Certain] The last two cases exist because [MASK] never appears at fine-tuning time, so the model must also learn to give good vectors for normal tokens. Let \(\mathcal{M}\) be the set of chosen positions and \(\tilde{x}\) the corrupted input:

\[ \mathcal{L}_{\text{MLM}} = -\frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \log p_\theta\big(x_i \mid \tilde{x}\big) \tag{23.7} \]
Read the symbols
\(\mathcal{M}\)
the positions picked for prediction (about 15% of all positions)
\(\tilde{x}\)
the whole input after the 80/10/10 corruption; the model sees all of it, both sides
\(p_\theta(x_i \mid \tilde{x})\)
the probability the model gives to the true token at position \(i\), from a softmax over the vocabulary

Only the chosen positions produce a loss. With 512 tokens, that is 77 predictions per sequence, not 512. Figure 5 compares this with the other families.

BERT also had a second task, next sentence prediction (NSP): is sentence B the real next sentence after A? RoBERTa (2019) showed it was not needed and dropped it, training longer on more data instead. [Certain]

Using an encoder

BERT adds a special [CLS] token at the start and [SEP] between and after sentences. After fine-tuning, the final vector of [CLS] (or the mean of all token vectors) is used for whole-text tasks such as sentiment. Each token’s own vector is used for token tasks such as named-entity tagging. For search, two texts are turned into vectors and compared with cosine similarity (Sentence-BERT, E5; see session 60). An encoder does all of this in one forward pass, with no generation loop. Deeper details are in session 33.

ModelBlocks\(d\)HeadsFFN widthParametersContext
BERT-base12768123,072110M512
BERT-large241,024164,096340M512
ModernBERT-base (2024)22768121,152 (gated)about 150M [Likely]8,192

Sizes come from the papers and the Hugging Face config.json files. [Certain] BERT uses a WordPiece vocabulary of 30,522 tokens and was pretrained on BooksCorpus (800M words) and English Wikipedia (2,500M words). [Certain]

Worked example: where BERT-base’s 110M parameters live

Embeddings: \(30{,}522 \times 768 = 23{,}440{,}896\) for tokens, plus \(512 \times 768\) for positions and \(2 \times 768\) for the two segment types.

One block, exactly: four attention matrices \(4 \times (768^2 + 768)\), the FFN \(768 \times 3{,}072 + 3{,}072 + 3{,}072 \times 768 + 768\), and two LayerNorms \(4 \times 768\). Total \(7{,}087{,}872\). The shortcut \(12d^2 = 7{,}077{,}888\) is within 0.15%.

Twelve blocks plus embeddings plus the small pooler (\(768^2 + 768\)) give \(109{,}482{,}240\): the “110M” of the paper. The token table alone is 21% of the model, a large share for a small model.

Decoder-only: GPT, Llama, Qwen

A decoder-only model is a stack of blocks with a causal mask and no cross-attention. GPT-1 (2018) was the first well-known one, a few months before BERT. [Certain] It is trained with the causal language modeling loss from session 6: predict each token from the ones before it.

\[ \mathcal{L}_{\text{CLM}} = -\frac{1}{n - 1} \sum_{t=2}^{n} \log p_\theta\big(x_t \mid x_1, \dots, x_{t-1}\big) \tag{23.8} \]

Two properties make this loss powerful.

  • Every position is a training example. Thanks to the causal mask, the output at position \(t - 1\) cannot see \(x_t\), so it can be trained to predict it. One forward pass over \(n\) tokens gives \(n - 1\) predictions, all in parallel. With 512 tokens: 511 predictions, against 77 for BERT.
  • Training and use are the same task. At inference, the model generates by predicting a token, appending it and repeating. Because past tokens never look at future ones, their keys and values never change, so they can be stored in a KV cache and reused; each new token costs one block pass, not a re-read of everything (session 80).

The price is that each token’s vector is built only from its left context. In a decoder, the vector for “bank” in “the bank of the river” cannot use “river”. For understanding tasks, this is a real weakness at small scale. At large scale, the model works around it: it reads the whole prompt first, and the last token’s vector has seen everything.

ModelBlocks\(d\)Query heads / KV headsFFN widthVocabularyParameters
GPT-2 small (2019)1276812 / 123,07250,257124M
GPT-3 (2020)9612,28896 / 9649,15250,257 [Likely]175B
Llama 3.1 8B (2024)324,09632 / 814,336 (gated)128,2568.03B
Qwen2.5-7B (2024)283,58428 / 418,944 (gated)152,0647.62B

Numbers from the GPT-3 paper and the config.json files; the Llama and Qwen totals are computed from their configs. [Certain] Modern decoders change the block in small ways (RMSNorm, RoPE, SwiGLU, grouped-query attention with fewer KV heads; see session 26), but the family is the same as GPT-1.

Worked example: GPT-3 from one formula

With \(12d^2\) per block (formula 23.10 below): \(12 \times 96 \times 12{,}288^2 = 173{,}946{,}175{,}488\), about 174B. The token embedding adds \(50{,}257 \times 12{,}288 \approx 0.62\)B. Together about 174.6B, the “175B” of the paper. At this size, the blocks are 99.6% of the weights.

Encoder–decoder: T5 and Whisper

An encoder–decoder keeps both stacks of the 2017 design. The encoder reads the input \(x\) (\(m\) tokens) once, with a bidirectional mask, and produces \(Z\). The decoder generates the output \(y\) with a causal mask, and each of its blocks reads \(Z\) by cross-attention (23.3). The loss is the causal loss on the output only, given the whole input:

\[ \mathcal{L}_{\text{seq2seq}} = -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta\big(y_t \mid y_{\lt t},\, x\big) \tag{23.9} \]

During training the decoder gets the target shifted right by one (a start token in front) and predicts all \(T\) target tokens in parallel; this is teacher forcing, the same trick as in session 7.

The generation loop, with cross-attention

At inference there is no target to shift, so the decoder writes one token per step. The key point of the data flow: the encoder runs once, and the cross-attention keys and values \(K = ZW_K\), \(V = ZW_V\) are computed once and reused by every decoding step. Only the query changes from step to step. Figure 4 runs this loop on a toy translator with hand-set weights, so that you can see each number. Each source vector \(z_j\) holds the word (a one-hot over 7 source words) and its position \(j\) as an angle, \((\cos j\theta, \sin j\theta)\) with \(\theta = 36°\). \(W_K\) keeps only the position, so \(k_j = (\cos j\theta, \sin j\theta)\); \(W_V\) keeps only the word. At step \(t\) the query is \(q_t = \kappa(\cos t\theta, \sin t\theta)\), so the score is

\[ s_{tj} = \frac{q_t \cdot k_j}{\sqrt{d_k}} = \frac{\kappa \cos\big((t - j)\theta\big)}{\sqrt{2}}, \]

which is largest when \(j = t\). The context \(c_t = \sum_j a_{tj} v_j\) is then mostly the word at position \(t\), and the LM head (logits \(= 6\, c_t\), a word-to-word dictionary) turns it into the French word. A real model learns all of this, including word order and agreement; the toy only translates word by word, and its decoder skips the causal self-attention sublayer because it does not need it.

decoder query \(q_t\)cached key \(k_j\) and attention weightvalue \(v_j\) (the source word)token picked
  1. \(Z \leftarrow \operatorname{Encoder}(x)\) // bidirectional, runs once
  2. \(K \leftarrow Z W_K\); \(V \leftarrow Z W_V\) // once, cached for all steps
  3. \(y_0 \leftarrow \langle s \rangle\); for \(t = 1, 2, \dots\):
  4. \(q_t \leftarrow\) query of the decoder at step \(t\)
  5. \(s_{tj} \leftarrow q_t \cdot k_j / \sqrt{d_k}\) for every \(j\) // no mask on the source
  6. \(a_t \leftarrow \softmax(s_t)\); \(c_t \leftarrow \sum_j a_{tj} v_j\) // (23.3)
  7. \(p \leftarrow \softmax(6\, c_t)\); \(y_t \leftarrow \argmax p\)
  8. if \(y_t = \langle /s \rangle\): stop
Figure 4. Encoder–decoder generation, computed live. Left: the encoder output, one row per source token, with its cached key (two numbers) and value (its word). Right: the decoder’s output so far, the query of the current step, and the next-word probabilities. The orange lines carry the attention weights from the query to the cached keys.
How to read Figure 4

Press Play. Steps 1 and 2 run once: the encoder and the K, V cache. Then each decoding step has four frames: the query, the scores, the weights and context, and the pick. With \(\kappa = 20\) at step 2, the scores are 11.441, 14.142, 11.441, 4.370, −4.370, the weights are 0.059, 0.882, 0.059, 0.000, 0.000, and “chat” gets probability 0.967. Lower \(\kappa\) to 4: the attention spreads over neighbors (0.250, 0.429, 0.250, …) and the probability of “chat” falls to 0.493. The counter at the bottom shows the point of the cache: 5 encoder token passes in total, while re-running the encoder at every step would cost 5 per step.

T5: every task is text-to-text

T5 (Raffel et al., 2019) casts every task as “text in, text out”: the input translate English to German: That is good. gives the output Das ist gut.; sst2 sentence: a lovely film gives positive. Pretraining uses span corruption: about 15% of tokens are removed in short spans (mean length 3), each span is replaced by one sentinel token, and the decoder writes the missing spans, each after its sentinel. [Certain] Sizes go from 60M to 11B. [Certain] T5-base has 12 + 12 blocks with \(d = 768\), about 220M parameters. Session 24 covers span corruption and BART’s denoising in detail.

Whisper: audio in, text out

Whisper (Radford et al., 2022) is an encoder–decoder for speech, trained on 680,000 hours of audio with transcripts. [Certain] The audio is cut into 30-second chunks and turned into a log-Mel spectrogram (a picture of sound energy over time and frequency). Two small convolution layers turn it into 1,500 frame vectors, which the encoder reads with full attention. The decoder then writes the text (up to 448 tokens), using cross-attention to look at the right part of the audio for each word. [Certain] Whisper large-v3 has 32 encoder and 32 decoder blocks with \(d = 1{,}280\), about 1.55B parameters. [Certain]

Whisper shows the real strength of this family: the input and the output can be different kinds of things (sound and text, or two languages) with very different lengths. The encoder can be shaped for the input (here, convolutions over audio), and the decoder for the output.

How the training signal differs

The three losses (23.7), (23.8) and (23.9) use the data very differently. Figure 5 shows a 24-token sentence and which positions produce a prediction under each objective.

hidden or corrupted in the inputposition that is predicted (gets a loss)sentinel token
Figure 5. Which tokens give a training signal. Masked positions are chosen with a fixed random seed; T5 spans have mean length 3. The readout also gives the counts for a 512-token sequence.
How to read Figure 5

In masked LM mode, only the orange tokens are predicted: 4 of 24 at 15%. In causal mode, every token except the first is a target: 23 of 24, and the corruption slider has no effect. In span-corruption mode, the input shrinks (each span becomes one sentinel) and the decoder target is short: the hidden tokens, one sentinel per span, and an end token. Raise the rate and watch the masked-LM count grow; but the more you hide, the less context is left to predict from.

Worked example: predictions per 512-token sequence

Masked LM at 15%: \(\operatorname{round}(0.15 \times 512) = 77\) predicted tokens.

Causal LM: \(512 - 1 = 511\) predicted tokens, 6.6 times more per forward pass.

T5 span corruption at 15%, mean span 3: 77 noise tokens in \(\operatorname{round}(77/3) = 26\) spans. The encoder input becomes \(512 - 77 + 26 + 1 = 462\) tokens (each span replaced by a sentinel, plus an end token). The decoder target is \(77 + 26 + 1 = 104\) tokens (the noise tokens, a sentinel before each span, and an end token).

So per token of compute, causal LM gets the most direct supervision. This is not the whole story (an MLM prediction uses context from both sides, so each one can be more informative), but it is one reason decoders learn efficiently from huge text collections. [Likely]

Counting parameters per family

With FFN width \(4d\) and ignoring biases and norms, self-attention has four \(d \times d\) matrices (\(4d^2\)) and the FFN has two \(d \times 4d\) matrices (\(8d^2\)). Cross-attention adds four more \(d \times d\) matrices:

\[ P_{\text{enc block}} = P_{\text{dec-only block}} \approx 12 d^2, \qquad P_{\text{decoder block with cross-attn}} \approx 16 d^2 \tag{23.10} \]

So for the same \(d\) and \(L\) blocks per stack, an encoder–decoder has \(L(12 + 16)d^2 = 28 L d^2\) block parameters, \(2.33\) times a decoder-only model. Comparisons between families must say what is held fixed: total parameters, or compute per token. An encoder–decoder with \(L\) + \(L\) blocks has about twice the parameters of an \(L\)-block decoder, but each token passes through only one of the two stacks, so the compute per token is similar. [Certain] The exact counting, including embeddings, GQA and SwiGLU, is the topic of session 25.

ModelFormulaBlock parametersReported total
BERT-base (enc, \(d = 768\), \(L = 12\))\(12 L d^2\)84.9M110M (with 23.8M of embeddings)
T5-base (enc–dec, 12 + 12)\(28 L d^2\)198.2Mabout 220M (computed 222.9M with the 32,128-token table)
Whisper large-v3 (32 + 32, \(d = 1{,}280\))\(28 L d^2\)1.47B1.55B
GPT-3 (dec, \(d = 12{,}288\), \(L = 96\))\(12 L d^2\)173.9B175B

The three families as algorithms

Algorithm 1: forward pass of each family
  1. input: token IDs \(x_1..x_n\) (and target IDs \(y_1..y_T\) for encoder–decoder)
  2. \(H \leftarrow \operatorname{Embed}(x) + \text{positions}\)
  3. if encoder-only: for \(\ell = 1..L\): \(H \leftarrow \operatorname{Block}(H, \text{mask} = \text{none})\); return \(\operatorname{Head}(H)\) // one vector per token, or [CLS]
  4. if decoder-only: for \(\ell = 1..L\): \(H \leftarrow \operatorname{Block}(H, \text{mask} = \text{causal})\); return \(\operatorname{LMHead}(H)\) // row t scores token t+1
  5. if encoder–decoder:
  6.   \(Z \leftarrow H\); for \(\ell = 1..L_{\text{enc}}\): \(Z \leftarrow \operatorname{Block}(Z, \text{mask} = \text{none})\) // run once
  7.   \(G \leftarrow \operatorname{Embed}(\langle s \rangle, y_1..y_{T-1})\) // target shifted right
  8.   for \(\ell = 1..L_{\text{dec}}\): \(G \leftarrow \operatorname{Block}(G, \text{mask} = \text{causal}, \text{cross} = Z)\) // formula (23.3) in every block
  9.   return \(\operatorname{LMHead}(G)\)
Algorithm 2: generating an answer to an input
  1. decoder-only: put the input and the answer in one sequence
  2.   prefill: run the whole prompt once, store K and V of every block in the cache
  3.   repeat: take the last row of logits, pick a token, run only that token through the blocks, append its K, V to the cache
  4. encoder–decoder: keep the input and the answer in two sequences
  5.   run the encoder once to get \(Z\); compute the cross-attention K, V from \(Z\) once per decoder block
  6.   start the decoder with \(\langle s \rangle\); repeat: run one token through the decoder blocks (causal self-attention with its own cache, then cross-attention to \(Z\)), pick the next token
  7. encoder-only: no loop; one forward pass gives all outputs // it cannot write new text by itself

The PyTorch version below builds all three families from one Block class and checks the key property of each: changing the last input token changes the first output of the encoder and of the encoder–decoder, but not of the decoder.

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

class Block(nn.Module):
    """One Pre-LN block. causal: mask the future. cross: add cross-attention to an encoder output."""
    def __init__(self, d, h, causal=False, cross=False):
        super().__init__()
        self.causal = causal
        self.n1, self.att = nn.LayerNorm(d), nn.MultiheadAttention(d, h, batch_first=True)
        self.cross = cross
        if cross:
            self.nc, self.xatt = nn.LayerNorm(d), nn.MultiheadAttention(d, h, batch_first=True)
        self.n2 = nn.LayerNorm(d)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x, mem=None):
        n = x.shape[1]
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1) if self.causal else None  # True = blocked
        a = self.n1(x)
        x = x + self.att(a, a, a, attn_mask=mask, need_weights=False)[0]          # (23.1)
        if self.cross:
            c = self.nc(x)
            x = x + self.xatt(c, mem, mem, need_weights=False)[0]                # (23.3): Q from x, K and V from mem
        return x + self.ffn(self.n2(x))                                           # (23.2)

class Stack(nn.Module):
    def __init__(self, V, d, h, L, causal, cross=False, lm_head=True):
        super().__init__()
        self.emb, self.pos = nn.Embedding(V, d), nn.Embedding(128, d)
        self.blocks = nn.ModuleList(Block(d, h, causal, cross) for _ in range(L))
        self.norm = nn.LayerNorm(d)
        self.head = nn.Linear(d, V, bias=False) if lm_head else None

    def forward(self, ids, mem=None):
        x = self.emb(ids) + self.pos(torch.arange(ids.shape[1]))
        for b in self.blocks:
            x = b(x, mem)
        x = self.norm(x)
        return self.head(x) if self.head is not None else x

V, d, h, L = 100, 64, 4, 2
encoder = Stack(V, d, h, L, causal=False)                  # BERT-like: logits for masked positions
decoder = Stack(V, d, h, L, causal=True)                   # GPT-like
ed_enc  = Stack(V, d, h, L, causal=False, lm_head=False)   # T5-like: encoder gives vectors ...
ed_dec  = Stack(V, d, h, L, causal=True, cross=True)       # ... the decoder reads them by cross-attention
for m in (encoder, decoder, ed_enc, ed_dec):
    m.eval()

x = torch.randint(0, V, (1, 10))
y = torch.randint(0, V, (1, 6))
print(encoder(x).shape, decoder(x).shape, ed_dec(y, ed_enc(x)).shape)

# Who sees the future? Change only the last input token and compare the outputs at position 0.
x2 = x.clone(); x2[0, -1] = (x[0, -1] + 1) % V
with torch.no_grad():
    print("encoder pos 0 changed:", not torch.allclose(encoder(x)[0, 0], encoder(x2)[0, 0]))   # True
    print("decoder pos 0 changed:", not torch.allclose(decoder(x)[0, 0], decoder(x2)[0, 0]))   # False
    # In the encoder-decoder, EVERY target position sees the whole source (cross-attention is not causal)
    print("enc-dec pos 0 changed:", not torch.allclose(ed_dec(y, ed_enc(x))[0, 0], ed_dec(y, ed_enc(x2))[0, 0]))  # True

# Parameters inside the blocks: about 12 d^2 per block, 16 d^2 with cross-attention (23.10)
per = lambda s: sum(p.numel() for p in s.blocks[0].parameters())
print(per(decoder), 12 * d * d, per(ed_dec), 16 * d * d)      # 49984 49152 66752 65536 (the rest: biases, norms)

# The three losses: masked LM (23.7), causal LM (23.8), sequence-to-sequence (23.9)
MASK = 0
pos = torch.tensor([2, 7])                          # positions we hide
xm = x.clone(); xm[0, pos] = MASK
loss_mlm = F.cross_entropy(encoder(xm)[0, pos], x[0, pos])
loss_clm = F.cross_entropy(decoder(x)[0, :-1], x[0, 1:])          # position t predicts token t+1
y_in = torch.cat([torch.zeros(1, 1, dtype=torch.long), y[:, :-1]], 1)   # shift right, start token 0
loss_s2s = F.cross_entropy(ed_dec(y_in, ed_enc(x))[0], y[0])
print([round(v.item(), 3) for v in (loss_mlm, loss_clm, loss_s2s)], round(torch.log(torch.tensor(V * 1.0)).item(), 3))

With PyTorch 2.8 on CPU this prints the three shapes [1, 10, 100], [1, 10, 100], [1, 6, 100], then True, False, True, then 49984 49152 66752 65536. The untrained losses are about 3.4, 4.8 and 4.8, near \(\ln 100 = 4.6\) as expected for random guessing (the masked loss is an average over only 2 positions, so it is noisier). In a real decoder-only model you would use F.scaled_dot_product_attention(..., is_causal=True), which applies the same mask with a fast kernel.

Choosing a family, and why decoder-only won

A practical guide

  • You need a fixed-size output for each input (a label, a score, a vector) and low cost per request: start with an encoder. A 100M-parameter encoder answers in one pass and is cheap to fine-tune. Typical: spam and intent classification, named entities, reranking, embeddings for search.
  • You need to write open-ended text, follow instructions, chat, write code, use tools, or do many tasks with one model: use a decoder-only LLM. Prompting replaces task-specific training.
  • You map one well-defined input to one output, the input is long or of another kind (audio, another language), and you train for that task: an encoder–decoder is often the most efficient choice, as Whisper and many production translation systems show. [Likely]
Figure 6. A task picker. Click a task to see which family is the usual first choice, a reasonable alternative, and why. These are common defaults, not laws.
How to read Figure 6

Look at the reason line. Encoders win when the output is small and fixed and latency or cost per request matter. Decoders win when the output is open-ended text or when one model must do many tasks. Encoder–decoders win when a different kind of input (sound, a long source text) must be turned into text. The video tasks show that most video systems combine families: an encoder for indexing, a decoder for writing.

Why decoder-only won at scale

Around 2019, encoders ruled understanding benchmarks and encoder–decoders (T5) did well on many tasks. By 2023 almost every frontier LLM was decoder-only. Several reasons work together:

  1. One objective that uses all the data. Next-token prediction needs no labels and gives a loss at every position (Figure 5). Raw text from the web is the training set.
  2. Generation is built in, and generation can do every task. GPT-2 and GPT-3 showed that a large enough decoder can do translation, question answering and classification just from a prompt, with no new head or fine-tuning (few-shot, in-context learning). [Certain] An encoder cannot write; an encoder–decoder needs the task split into “input” and “output”.
  3. Simplicity scales. One stack, one mask, one loss, one set of weights for prompt and answer. Every tool (parallelism, kernels, quantization, KV caching, serving engines like vLLM) is built for this one shape. [Likely]
  4. Chat fits naturally. A conversation is one growing sequence. A decoder caches all earlier turns and only processes the new ones. In an encoder–decoder, a growing input must be re-encoded because the encoder is bidirectional: adding a token changes every earlier encoder vector. [Certain]
  5. Zero-shot results. Wang et al. (2022) trained models of over 5B parameters on over 170B tokens with each architecture. After pure self-supervised pretraining, the causal decoder trained on next-token prediction had the best zero-shot generalization; after multitask fine-tuning, a model with non-causal (bidirectional) input visibility and a masked objective did best. [Certain] So the answer depends on the stage, but the “no extra fine-tuning needed” setting favors the decoder.

The bidirectional advantage did not disappear. It moved: modern decoders read the full prompt before answering, so the last position has seen everything, and at billions of parameters this is enough for most understanding tasks. Encoders are still the best value at small sizes and are used heavily for embeddings, classification and reranking; ModernBERT (2024) is a recent example. [Certain] And many new embedding models are fine-tuned from decoder-only LLMs. [Likely]

encoder-onlydecoder-onlyencoder–decoder
Figure 7. Some well-known models by family and year of first release. All three families started within about a year of each other; the decoder-only lane is where the largest models went.

What about vision-language models?

A typical VLM (session 177) mixes families. The image goes through a vision encoder (a ViT or SigLIP, which is an encoder-only transformer over image patches). A small connector turns the patch vectors into tokens, and a decoder-only LLM reads them as part of its prompt and writes the answer (LLaVA, Qwen2.5-VL). [Certain] Flamingo instead added cross-attention layers into a frozen decoder, so the text side reads image features the way an encoder–decoder reads \(Z\). [Certain] Speech works the same way: Whisper’s encoder is often reused as the audio encoder of omni models. [Likely] The book covers these designs in chapter 6 (Noyan et al., Vision Language Models).

Where this is used

ApplicationUsual familyWhy
Sentiment, intent, spam, toxicity classificationEncoder (BERT-like), fine-tunedOne pass, small model, fixed label set, very high throughput
Semantic search and RAG retrievalEncoder (bi-encoder embeddings), cross-encoder rerankerVectors are computed once per document; bidirectional context gives good vectors (session 150)
Chat assistants, code completion, agentsDecoder-onlyOpen-ended generation, prompting, KV cache across turns
Machine translation, summarizationEncoder–decoder or decoder-onlyClear input-to-output task; large decoders now do both well
Speech recognitionEncoder–decoder (Whisper)Audio in, text out; the encoder is shaped for audio
Text-to-video retrievalTwo encoders (video encoder and text encoder, CLIP-style)Index millions of clips once as vectors; queries need only a fast text pass (session 208)
Action recognition, video classificationEncoder (video transformer) with a classification headFixed label set over a clip; bidirectional attention over space and time
Key-moment detection, temporal groundingEncoder over frame features, sometimes a decoder-only Video-LLMScores per frame need full context; a Video-LLM can answer “when does X happen?” in text
Video fingerprinting, near-duplicate detectionEncoder (frame or clip embeddings)Compact vectors compared at scale; no text output needed
Video captioning, video question answering, Video-RAGVision encoder + decoder-only LLMMust write free text about what is seen (session 205)

Common mistakes

  • Mixing up the two meanings of “decoder”. A GPT block has no cross-attention. A T5 decoder block does. Count parameters accordingly (12d² vs 16d²).
  • Thinking BERT can generate text like GPT. BERT has no causal mask and no next-token training. You can make it fill in [MASK] slots, but it has no natural left-to-right generation loop.
  • Using a decoder’s first-token or mean vector as a sentence embedding without fine-tuning. In a causal model the first token has seen only itself; the last token is the one that has seen the whole input.
  • Forgetting the shift. In causal LM, position \(t\) predicts token \(t+1\): logits [:, :-1] against labels [:, 1:]. In an encoder–decoder, the decoder input is the target shifted right with a start token.
  • Computing MLM loss on all positions. Only the chosen ~15% get a loss; the others are ignored (label −100 in Hugging Face).
  • Comparing families at unequal budgets. “T5 beat GPT” or the reverse means little unless parameters, compute per token and training tokens are stated.
  • Re-running the encoder at every decoding step. In an encoder–decoder, the encoder output and the cross-attention keys and values are computed once per input, not once per generated token.

Interview questions

What is the difference between BERT and GPT?basic
Both are stacks of transformer blocks. BERT is encoder-only: a bidirectional mask, so every token sees the whole input, trained with masked language modeling (predict ~15% hidden tokens). GPT is decoder-only: a causal mask, so each token sees only the past, trained with next-token prediction. BERT gives strong representations for classification and embeddings in one pass; GPT can generate text and, at scale, does most tasks by prompting.
What exactly makes a model encoder-only, decoder-only or encoder–decoder?basic
The attention pattern and the stacks. Encoder-only: one stack, all pairs visible (\(n^2\) pairs). Decoder-only: one stack, causal mask (\(n(n+1)/2\) pairs), no cross-attention. Encoder–decoder: two stacks; the encoder is bidirectional over the input, the decoder is causal over the output and has a cross-attention sublayer in every block that reads the final encoder output. The blocks themselves (attention, FFN, residuals, norms) are the same.
Why does BERT mask only 15% of tokens, and why the 80/10/10 split?core
Masking more leaves too little context to predict from, and masking less gives too few training signals per pass; 15% was the paper’s choice. The 80/10/10 split (mask token, random token, unchanged) reduces the mismatch between pretraining and fine-tuning: [MASK] never appears in real inputs, so the model must also produce useful vectors for ordinary tokens and cannot assume that only [MASK] positions matter.
Why can a decoder train on all positions in parallel but must generate one token at a time?core
In training the whole true sequence is known, and the causal mask ensures that position \(t\) only uses tokens \(\le t\), so all \(n-1\) next-token predictions are computed in one forward pass. In generation, token \(t+1\) does not exist until it is sampled from the output at position \(t\), so steps are sequential. The KV cache makes each step cheap: past keys and values never change because past tokens never attend to the future.
How many parameters does one transformer block have? With cross-attention?core
With FFN width \(4d\) and no biases: attention \(4d^2\) (Q, K, V, O), FFN \(8d^2\), total \(12d^2\). A decoder block with cross-attention adds another \(4d^2\): \(16d^2\). Check: GPT-3 has \(12 \times 96 \times 12288^2 \approx 174\)B in its blocks; BERT-base \(12 \times 12 \times 768^2 \approx 85\)M plus 24M of embeddings \(\approx\) 110M. With SwiGLU and GQA the numbers change (Llama 3 8B: about 218M per block at \(d = 4096\)).
Why are almost all modern LLMs decoder-only?core
(1) Next-token prediction uses every token of unlabeled text as a target. (2) Generation is built in, and with scale, prompting turns generation into a universal interface for many tasks (GPT-3 few-shot). (3) One stack and one mask are simple to scale, parallelize and serve; the whole tool ecosystem targets it. (4) Multi-turn chat fits a single growing sequence with a KV cache; a bidirectional encoder would need to re-encode. (5) Wang et al. 2022 found causal decoders best for zero-shot generalization right after pretraining. Encoders and encoder–decoders still win in specific niches.
When would you still pick an encoder–decoder?core
When the task is a clear mapping from an input to an output, especially when the input is long or a different modality: speech recognition (Whisper), translation, summarization with a fixed format. The encoder reads the input once with full attention, can be shaped for the input (e.g. audio convolutions), and its keys and values are reused by every decoding step. At equal compute per token it can hold about twice the parameters of a decoder-only model.
What is a prefix LM and how does it relate to an encoder–decoder?advanced
A prefix LM is a single decoder stack whose mask is bidirectional over the first \(p\) tokens (the input) and causal after. Its visibility pattern is exactly that of an encoder–decoder: input sees input, output sees all input and its own past. The difference is weight sharing: the prefix LM uses one set of weights for both roles in one stack, while the encoder–decoder has separate encoder weights, separate decoder weights and separate cross-attention layers. T5 and UL2 studied both.
Your product needs sentence embeddings for search over 100 million documents. Which family, and why?advanced
A bi-encoder: an encoder-only model (or an embedding model fine-tuned contrastively, possibly from a decoder backbone) maps each document once to a vector; queries are encoded at request time and matched with approximate nearest-neighbor search. Bidirectional attention gives each vector the full context. A generative decoder is unnecessary and much more expensive per document. Add a cross-encoder reranker on the top results if quality matters more than latency.
Why must an encoder–decoder re-encode a growing input, while a decoder can cache it?advanced
In a bidirectional encoder every token attends to every other token, so adding a new input token changes the vectors of all earlier tokens in every layer; the old encoder output is no longer valid. In a causal decoder, earlier tokens never attend to later ones, so their keys and values are unchanged when new tokens arrive; they can be cached and only the new tokens are processed. This is a major reason chat systems use decoders.

Exercises

  1. Count the allowed (query, key) pairs for \(n = 10\) under a bidirectional mask, a causal mask, and a prefix mask with \(p = 6\).
  2. An encoder–decoder has 6 encoder and 6 decoder blocks with \(d = 512\) and FFN width \(4d\). Estimate its block parameters with formula (23.10). Compare with the 2017 base model’s 65M.
  3. For a 1,024-token sequence, how many tokens get a loss under masked LM at 15%, causal LM, and T5 span corruption at 15% with mean span 3? How long is the T5 decoder target?
  4. Llama 3.1 8B has \(d = 4096\), 32 query heads and 8 KV heads of size 128, and a SwiGLU FFN of width 14,336. Compute the parameters of one block (ignore norms).
  5. A chat has 2,000 tokens of history and the user adds 50 tokens. How many input tokens must the decoder process for the new turn (with a KV cache)? How many would a bidirectional encoder have to re-encode?
  6. Why does using the vector of the first token of a decoder-only model as a sentence embedding work poorly? Which position would you use instead?
Answerscheck
1. \(100\); \(10 \cdot 11/2 = 55\); \(36 + 55 - 21 = 70\). 2. \(6 \times 12 \times 512^2 + 6 \times 16 \times 512^2 = 18{,}874{,}368 + 25{,}165{,}824 = 44{,}040{,}192\), about 44M in the blocks; the rest of the 65M is mostly the shared token embedding (about 37,000 tokens \(\times\) 512 \(\approx\) 19M) [Likely]. 3. \(\operatorname{round}(153.6) = 154\); \(1{,}023\); 154 noise tokens in \(\operatorname{round}(154/3) = 51\) spans, so the target is \(154 + 51 + 1 = 206\) tokens. 4. Attention: \(Q\) and \(O\) are \(4096 \times 4096\) each, \(K\) and \(V\) are \(4096 \times 1024\) each: \(2 \times 16{,}777{,}216 + 2 \times 4{,}194{,}304 = 41{,}943{,}040\). FFN: \(3 \times 4096 \times 14{,}336 = 176{,}160{,}768\). Block: \(218{,}103{,}808\), about 218M. 5. Decoder: only the 50 new tokens (the 2,000 are in the cache). Encoder: all 2,050, because every old vector changes. 6. With a causal mask the first token has attended only to itself, so it knows nothing about the rest of the sentence. Use the last token (or fine-tune the model for embeddings, often with a bidirectional mask or mean pooling).

Key takeaways

  • A full transformer is one block (23.1, 23.2) repeated \(L\) times, plus embeddings and a head. The 2017 original was an encoder–decoder, joined by cross-attention (23.3).
  • The attention mask (23.4, 23.5) defines the family: bidirectional (\(n^2\) pairs), causal (\(n(n+1)/2\)), or prefix; an encoder–decoder has the prefix pattern split over three attention layers.
  • Encoder-only (BERT) learns by masked LM (23.7), 15% of positions, and gives strong vectors in one pass: classification, tagging, embeddings.
  • Decoder-only (GPT, Llama, Qwen) learns by next-token prediction (23.8) on every position, generates with a KV cache, and at scale does most tasks by prompting.
  • Encoder–decoder (T5, Whisper) maps an input to an output (23.9): best for translation, summarization and speech, where the input is long or of another kind.
  • Blocks cost about \(12d^2\), or \(16d^2\) with cross-attention (23.10). Decoder-only won through one simple objective, universal prompting, easy scaling and natural chat caching.

Key formulas

\[ \softmax\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V, \quad \mathcal{L}_{\text{MLM}} = -\tfrac{1}{|\mathcal{M}|}\textstyle\sum_{i\in\mathcal{M}} \log p(x_i \mid \tilde{x}), \quad \mathcal{L}_{\text{CLM}} = -\tfrac{1}{n-1}\textstyle\sum_{t} \log p(x_t \mid x_{\lt t}) \]
\[ \mathcal{L}_{\text{seq2seq}} = -\tfrac{1}{T}\textstyle\sum_{t} \log p(y_t \mid y_{\lt t}, x), \qquad P_{\text{block}} \approx 12d^2 \ \ (16d^2 \text{ with cross-attention}) \]

References for this session

  1. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  2. Brown et al. (2020). Language Models are Few-Shot Learners
  3. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  4. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 6: Transformers
  5. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 7: Transformers and Pretraining
  6. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  7. Brown et al. (2020). Language Models are Few-Shot Learners
  8. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  9. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 6: Transformers
  10. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 7: Transformers and Pretraining
  11. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 9: Masked Language Models
  12. Vaswani et al. (2017). Attention Is All You Need
  13. Radford, Narasimhan, Salimans, Sutskever (2018). Improving Language Understanding by Generative Pre-Training
  14. Radford et al. (2019). Language Models are Unsupervised Multitask Learners
  15. Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach
  16. Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  17. Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
  18. Llama Team, Meta (2024). The Llama 3 Herd of Models
  19. Qwen Team (2024). Qwen2.5 Technical Report
  20. Warner et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
  21. Wang, Roberts, Hesslow, Le Scao, Chung, Beltagy, Launay, Raffel (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
  22. Tay et al. (2022). UL2: Unifying Language Learning Paradigms
  23. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language