Masks: causal, padding, bidirectional, prefix and packing masks
Attention lets every token look at every other token. A mask decides which of those looks are allowed. This session shows how one simple trick, adding minus infinity to a score before the softmax, gives the causal mask of GPT, the padding mask for batches, the full view of BERT, the prefix-LM view used by T5 and PaliGemma, and the document mask that keeps packed samples apart.
What you will learn
- How a mask works: add 0 or minus infinity to each attention score, so blocked pairs get exactly zero weight
- The causal mask: why a language model must not see the future during training, and how this lets it learn from all positions at once
- Padding masks for batches of different lengths, left versus right padding, and the separate loss mask
- Bidirectional (BERT), causal (GPT) and prefix-LM (T5, UniLM, PaliGemma) attention patterns, and what each is good for
- Sequence packing and the document mask that stops packed samples from seeing each other
- How real code expresses masks (boolean versus additive, the True means keep or block trap, NaN rows) and how fast kernels skip blocked tiles
- How the same masks appear in VLMs and video models
Why this matters
In session 17 every query got a score with every key, and softmax turned each row of scores into weights. Nothing stopped a token from looking anywhere. Real models need rules: “do not look at the future”, “do not look at padding”, “do not look at a different document”. A mask is the tool that writes these rules into attention.
A mask is a table with one entry for each (query, key) pair. The entry says “allowed” or “blocked”. Blocked pairs get exactly zero attention weight. That is the whole idea, and it is surprisingly powerful. The same mechanism, with different tables, gives:
- The causal mask of GPT-style models: each token sees only itself and the tokens before it. This is what makes next-token training honest, and it is why a KV cache works.
- The padding mask: sequences in a batch have different lengths, so we fill the short ones with padding tokens and tell attention to ignore them.
- The bidirectional (full) view of BERT, the causal view of GPT, and the prefix-LM view in between, used by T5 experiments, UniLM and PaliGemma.
- The document mask: when we pack many short samples into one long training row, it stops one sample from reading another.
Masks look like a small detail, but many real bugs live here: a model that “cheats” in training and fails at generation, a batch whose outputs change when you add padding, a fine-tune that learns from padding tokens, NaN losses from a row where everything was blocked. Interviewers know this. Typical questions: “Why do we need a causal mask?”, “Left or right padding for generation, and why?”, “What is the difference between an attention mask and a loss mask?”, “What goes wrong when you pack sequences without a document mask?”. By the end of this session you can answer each one with a formula, a picture and a few lines of code.
One mechanism: add minus infinity before the softmax
Recall the attention of one head. The queries \(Q\), keys \(K\) and values \(V\) have one row per token. The scores are \(QK^\top/\sqrt{d_k}\), one number per (query, key) pair. A mask is a matrix \(M\) of the same shape, added to the scores before the softmax:
- Read the symbols
- \(Q, K, V\)
- queries, keys and values, one row per token (shape \(T \times d_k\) for \(T\) tokens)
- \(d_k\)
- the size of one query or key vector; dividing by \(\sqrt{d_k}\) keeps scores at a sane size (session 17)
- \(M\)
- the mask, a \(T \times T\) matrix of 0 and \(-\infty\). Row \(i\) is the rule for query \(i\); column \(j\) is key \(j\).
- softmax
- applied to each row separately, so each query gets weights that sum to 1
Why does adding \(-\infty\) work? Write \(s_{ij}\) for the scaled score. The weight of key \(j\) for query \(i\) is:
- Read the symbols
- \(a_{ij}\)
- the attention weight of key \(j\) in the output of query \(i\)
- \(\mathcal{A}_i\)
- the set of keys that query \(i\) is allowed to see (where \(M_{ij} = 0\))
- \(e^{-\infty} = 0\)
- a blocked key adds nothing to the sum and gets weight exactly 0
So a masked softmax is just an ordinary softmax over the allowed keys only. The blocked keys do not get a small weight; they get zero, and their values never reach the output. Their gradient is zero too: no learning signal flows through a blocked pair.
A sequence has 4 tokens. Query 2 has the scaled scores \(s_{2,\cdot} = (2.0,\ 1.0,\ 0.5,\ 3.0)\) for keys 1 to 4. Without a mask, softmax gives the weights \((0.232,\ 0.085,\ 0.052,\ 0.631)\): most of the weight goes to key 4, a token that comes later in the sentence.
With the causal mask, row 2 is \(M_{2,\cdot} = (0, 0, -\infty, -\infty)\). Only keys 1 and 2 remain: \(e^{2}/(e^{2} + e^{1}) = 7.389/10.107 = 0.731\) and \(e^{1}/10.107 = 0.269\). The weights are \((0.731,\ 0.269,\ 0,\ 0)\). They still sum to 1, and the future has no effect.
Two ways to store a mask are common. An additive mask is a float matrix of 0 and \(-\infty\) (or a very negative number), added exactly as in (18.1). A boolean mask is a matrix of True and False that the library turns into the additive form. Be careful: libraries do not agree on what True means. We come back to this in the section on code.
Figure 1 runs this computation for real, one query row at a time, on 8 tokens. The scores are a fixed table of numbers. For each row it builds the mask row, adds it, takes the maximum over the allowed keys, exponentiates, and divides by the sum: exactly formula (18.2), done the numerically safe way of Algorithm 3. Switch the mask at any moment and watch which cells survive.
- \(S \leftarrow QK^\top / \sqrt{d_k}\) // one score per \((i, j)\)
- for each query row \(i = 1, \dots, 8\):
- \(M_{ij} \leftarrow 0\) if allowed\((i, j)\) else \(-\infty\)
- \(m \leftarrow \max_j \,(s_{ij} + M_{ij})\) // only allowed keys can win
- \(e_j \leftarrow \exp(s_{ij} + M_{ij} - m)\); \(Z \leftarrow \sum_j e_j\) // \(e^{-\infty} = 0\)
- \(a_{ij} \leftarrow e_j / Z\) // (18.2): the row sums to 1
- return \(A\) // then the output is \(AV\)
How to read Figure 1
Press Play, or use Step and Back. Each row takes four steps: build the mask row (line 3), find the maximum over the allowed keys (line 4), exponentiate and sum (line 5), divide (line 6). In Causal mode, row 4 (“on”) keeps keys 1 to 4; the maximum is \(m = 0.4\) (key 2), the values are \(e = (0.202, 1, 0.202, 0.202)\), \(Z = 1.606\), and the weights are \((0.126, 0.623, 0.126, 0.126)\). Now switch to Bidirectional without stopping: the same row also keeps keys 5 to 8, the maximum moves to key 5 (1.2), and key 5 takes 0.458 of the weight. In Packed mode row 4 is the first token of document 2 (“hi”), so it may see only itself: weight 1. In Causal + padding the red columns 7 and 8 are the pads. The right matrix is always zero in every red cell, and every finished row sums to 1.
The causal mask: no peeking at the future
Why a language model needs it
A language model is trained to predict each token from the tokens before it (session 6). The probability of a whole text is a product of these one-step predictions:
During training we feed the full text at once and ask, at every position \(t\), for the next token \(x_{t+1}\). This gives \(T\) training examples from one forward pass, which is the main reason transformers train so fast. But there is a trap. If position \(t\) can attend to position \(t+1\), the answer is sitting in its input. The model learns to copy it, the training loss drops close to zero, and the model is useless at generation time, when the future does not exist yet.
The causal mask (also called the look-ahead mask or autoregressive mask) closes this door. Query \(i\) may see key \(j\) only when \(j\) is not after \(i\):
In the matrix picture, the allowed entries are the diagonal and everything below it: a lower triangle. Row 1 sees one key, row 2 sees two keys, and row \(T\) sees all \(T\). The total number of allowed pairs is
For \(T = 8\) that is 36 of 64 pairs. For large \(T\) it approaches half of \(T^2\); for \(T = 1024\) it is 524,800 of 1,048,576, which is 50.0%. A kernel that skips the blocked half can save almost half of the attention work (Figure 5).
What the causal mask buys at inference
The causal mask has a second, practical effect. Because token \(i\) never looks at later tokens, its hidden states in every layer do not change when new tokens are appended. So during generation we can compute the keys and values of old tokens once and keep them. This is the KV cache (session 80). A bidirectional model cannot do this: every new token would change the states of all earlier tokens. Figure 3 shows this “nothing in the past changes” property directly.
How to read Figure 2
Start in Causal mode with row 4 (the query “on”). It may see keys 1 to 4, and the weights are (0.126, 0.623, 0.126, 0.126). Switch to Bidirectional: the same row now also sees keys 5 to 8, and almost half of its weight (0.458) goes to key 5, a word that comes later. That is exactly the leak the causal mask prevents. In Prefix-LM mode the dashed square is fully colored: the first \(P\) tokens see each other in both directions; the rest is causal. In Packed mode the sequence holds three documents (“the cat sat”, “hi there”, “a dog ran”); each one is a small causal triangle and nothing crosses between them: 15 allowed pairs instead of 36. In Causal + padding the last two columns are grey for every row, so no real token reads the padding.
The padding mask: batches with different lengths
Why padding exists
GPUs work on rectangular tensors. A batch of \(B\) sequences must have shape \(B \times T\), but real samples have different lengths. The usual fix is padding: add a special pad token (see session 9) to the short sequences until all have the same length. Together with the token IDs the tokenizer returns an attention mask \(m\): 1 for a real token, 0 for padding.
Padding tokens are not text. If real tokens attend to them, the output depends on how much padding the batch happened to have: the same sentence gives a different answer in a different batch. So we block every key that is padding:
- Read the symbols
- \(m_j\)
- the attention-mask bit of token \(j\) from the tokenizer: 1 = real token, 0 = padding
- \(j \le i\)
- the causal part; drop it for a bidirectional encoder such as BERT
Notice what is blocked: the padding columns (keys). The padding rows (queries) are still computed; they produce some output, but nobody reads it. This is fine because of a second, separate mask.
The loss mask is a different mask
The attention mask controls what tokens see. The loss mask controls which positions count in the loss. Padding positions must not be trained on: otherwise the model learns to predict “pad after pad”, and the loss is diluted by easy fake targets. With per-position losses \(\ell_t\) and weights \(w_t \in \{0, 1\}\):
In PyTorch and Hugging Face code this is done by setting the label of an ignored position to -100; F.cross_entropy skips it and divides by the number of remaining targets. [Certain] The same tool also removes the loss on the prompt in supervised fine-tuning, so the model is trained only on the answer (session 54).
A batch has four samples with 5, 12, 3 and 8 tokens. Padding to the longest gives a \(4 \times 12\) tensor: 48 slots for \(5 + 12 + 3 + 8 = 28\) real tokens. The padding fraction is
So 41.7% of the compute in the linear layers is spent on tokens that are then thrown away. Attention is worse, because its cost grows with the square of the row length: \(4 \times 12^2 = 576\) score entries, while the real samples need only \(5^2 + 12^2 + 3^2 + 8^2 = 242\). Krell et al. (2021) report that up to 50% of all tokens can be padding in common NLP datasets, and up to 89% for one GLUE task at length 128. [Certain]
Left or right padding?
For training, right padding (pads at the end) is the usual choice. For batched generation with a decoder-only model, most libraries want left padding (pads at the start). The reason: generation appends the next token at the end of every row. With right padding, the new token of a short row would land after its pads, far from its last real token, and the model would be asked to continue a text that ends in padding. With left padding, the last position of every row is a real token, so all rows can be extended in lockstep. [Certain]
Left padding has one catch: positions. If row 1 is pad pad the cat, the word “the” should still be position 0, not position 2. Libraries compute position IDs from the attention mask, for example the cumulative sum of \(m\) minus 1. The PyTorch code below shows it. With rotary positions (session 14), wrong position IDs here silently hurt quality.
Bidirectional, causal and prefix-LM
The causal mask is right for generation, but it throws away information when the whole input is already known. To classify a sentence or to tag every word, token 1 would like to see token 8. Three patterns cover most models; T5 (Raffel et al., 2019) calls them “fully-visible”, “causal” and “causal with prefix”. [Certain]
- Bidirectional (fully-visible). \(M = 0\) everywhere (except padding). Every token sees every token. Used by encoders such as BERT, which is trained by hiding 15% of the input tokens and predicting them (masked language modeling). [Certain] Careful with words: the [MASK] token of BERT is a change to the input, not an attention mask. BERT has no causal mask at all.
- Causal. Formula (18.4). Used by decoder-only models (GPT, Llama, Qwen) and by the decoder of encoder–decoder models.
- Prefix-LM (causal with prefix). The first \(P\) tokens, the prefix (the input or prompt), see each other in both directions. The rest, the target, is causal and also sees the whole prefix:
- Read the symbols
- \(P\)
- the prefix length; positions are counted from 0, so the prefix is positions \(0, \dots, P-1\)
- \(j \lt P\)
- every query may see every prefix key, even prefix keys to its right
- \(j \le i\)
- outside the prefix the usual causal rule holds
A prefix-LM is one stack of layers that behaves like an encoder on the prefix and like a decoder on the target. The loss is put only on the target tokens (a loss mask again). UniLM (Dong et al., 2019) trained one network with all three masks, switching per batch, to serve both understanding and generation tasks. [Certain] PaliGemma uses the prefix-LM pattern for images: the image tokens and the text prompt form the prefix with full attention, and the answer (the suffix) is generated causally. [Certain] This lets image tokens look at the question, and the question look at all of the image.
Counting allowed pairs for the prefix-LM: the \(P\) prefix rows see \(P\) keys each, and row \(i \ge P\) (counting from 0) sees \(i + 1\) keys:
For \(T = 8\) and \(P = 3\): \(9 + (72 - 12)/2 = 39\) pairs, three more than causal. Those three are the pairs inside the prefix that look to the right.
| Pattern | Who sees what | Typical models | Good for | Cost |
|---|---|---|---|---|
| Bidirectional | everyone sees everyone | BERT, RoBERTa, ViT, the T5 encoder, CLIP image tower | understanding: classification, tagging, embeddings | cannot generate left to right; no KV cache reuse |
| Causal | only self and past | GPT, Llama, Qwen, Mistral, decoders | generation; training on every position at once | prompt tokens never see later prompt tokens |
| Prefix-LM | prefix: full; target: causal and sees the prefix | UniLM, T5 “prefix LM” variant, PaliGemma | conditional generation: answer a question about an input | loss only on the target; prefix KV must be built with the full mask |
| Encoder–decoder | encoder: full; decoder: causal self-attention plus cross-attention to the encoder | T5, BART, Whisper | translation, summarization, speech | two stacks of weights (session 20) |
Packing and document masks
Packing: no more padding
Pretraining text is a stream of documents of very different lengths. Instead of padding each one, we pack them: concatenate many documents, separated by an end-of-text token, and cut the stream into rows of exactly \(T\) tokens. For fine-tuning data, where each sample must stay whole, we use a bin-packing method instead: fill each row with whole samples that fit, for example with first-fit decreasing (sort samples from long to short; put each into the first row with enough room). Krell et al. (2021) report a 2× speed-up for BERT phase-2 pretraining with packing. [Certain]
Cross-contamination and the document mask
Packing creates a new problem. With a plain causal mask, the first token of document 2 can attend to all of document 1. The model then learns from a context that is unrelated text, and in fine-tuning a sample can “see” the answer of another sample. Krell et al. call this cross-contamination. The fix is a document mask (also called intra-document causal mask): give every token a document ID \(\mathrm{doc}(i)\) and allow only pairs inside the same document:
The mask is block-diagonal: one small causal triangle per document, and nothing between them. With document lengths \(L_1, \dots, L_D\) the number of allowed pairs is
To make a packed row exactly equal to separate rows, also reset the position IDs at every document start, so each document begins at position 0. With both, the outputs are identical to running each document alone; the PyTorch code below checks this.
A row of \(T = 1024\) tokens holds four documents with 384, 128, 320 and 192 tokens. The plain causal mask allows \(1024 \cdot 1025/2 = 524{,}800\) pairs. The document mask allows \(384 \cdot 385/2 + 128 \cdot 129/2 + 320 \cdot 321/2 + 192 \cdot 193/2 = 73{,}920 + 8{,}256 + 51{,}360 + 18{,}528 = 152{,}064\) pairs, only 14.5% of the full \(1024^2\) matrix. A kernel that can skip blocked tiles does far less work (Figure 5).
Does the document mask matter for quality? Llama 3 uses “an attention mask that prevents self-attention between different documents within the same sequence”. [Certain] The same report says this had limited impact in standard pretraining but mattered for continued pretraining on very long sequences. [Likely] Zhao et al. (2024) found that intra-document causal masking improves language modeling and downstream results compared to plain causal packing. [Certain] Some pretraining pipelines still use plain causal packing and rely on the end-of-text token to separate documents. [Likely]
How to read Figure 3
This is the test you should run on any attention code. In Causal mode, edit token 4: outputs 4 to 8 change and outputs 1 to 3 stay exactly the same (\(\Delta = 0\)), so the past cannot see the future. In Bidirectional mode every output changes. In Prefix-LM mode, editing a prefix token (1 to 3) changes everything, but editing token 6 changes only 6 to 8. In Packed mode, editing “hi” (token 4) changes only 4 and 5: the other two documents do not notice. In Causal + padding mode, edit a pad token (7 or 8): only its own output changes, and that position is ignored by the loss. If any real output moved, your padding mask would be broken.
How to read Figure 4
With 16 samples, plain padding wastes 65.3% of the token slots: one long sample in a batch forces 7 short ones to be padded to its length. Sorting puts similar lengths together and cuts the waste to 55.0% (28.2% with 32 samples, where there are more similar lengths to group). Packing needs only 4 rows for the 16 samples and wastes 1.5%. The last line of the readout is about attention: a packed row costs a full \(512^2\) score matrix if you use a plain mask, but the real samples need only \(\sum L_i^2\). With 32 samples, packed rows without a document-aware kernel do more attention work (2,359,296) than sorted padding (2,113,496). Packing wins clearly only when the kernel skips the blocked tiles (Figure 5).
Masks in real code and fast kernels
Boolean, additive, and the “True means what?” trap
Three conventions are in daily use, and they disagree:
| API | Mask form | Meaning of True / 1 |
|---|---|---|
F.scaled_dot_product_attention(..., attn_mask=) | bool or float, broadcast to \((B, H, T_q, T_k)\) | bool True = may attend; float is added to the scores |
nn.MultiheadAttention(..., key_padding_mask=, attn_mask=) | bool \((B, T_k)\) and \((T_q, T_k)\) | True = ignore (blocked) |
Hugging Face tokenizer attention_mask | int \((B, T)\) of 0 and 1 | 1 = real token; the model builds the 4-D mask |
We checked the first two in PyTorch 2.8: in nn.MultiheadAttention a True in key_padding_mask gave that key weight 0, while in scaled_dot_product_attention a True means the pair takes part. [Certain] Passing a mask from one API to the other without flipping it gives a model that attends only to padding and the future. It still trains, so this bug can hide for a long time.
A few more practical rules:
- Use
is_causal=Truewhen you can. Kernels such as FlashAttention then never build the \(T \times T\) mask and skip the upper triangle. But note: in PyTorch SDPA,is_causalis aligned to the top-left corner. With 1 query and 5 cached keys, it lets the query see only key 1. For decoding with a KV cache you need the bottom-right version (the new query sees all cached keys), or no mask at all for a single new token. [Certain] - Large negative, not always \(-\infty\). Many codes use the most negative number of the data type, for example
torch.finfo(dtype).min, to avoid \(-\infty - (-\infty)\) = NaN in some kernels. Then a fully blocked row becomes uniform instead of NaN, which hides the bug instead of showing it. - Mask in the right dtype. In float16 the most negative value is −65,504. Adding it to a score and then adding another bias can overflow to \(-\infty\). Build masks in the same dtype as the scores. [Likely]
- Variable-length kernels. For packed rows, FlashAttention offers a “varlen” interface: you pass the documents back to back plus the cumulative lengths (
cu_seqlens), and the kernel applies the document mask (18.11) without ever building it. [Certain]
Block masks: how kernels skip work
Fast attention kernels (session 124) do not work on single (query, key) pairs. They cut the score matrix into tiles of, say, \(128 \times 128\) and process one tile at a time. For each tile, the mask has three cases:
- Full: every pair in the tile is allowed. Compute it with no mask at all.
- Partial: some pairs are allowed. Compute it and apply the mask inside the tile.
- Empty: no pair is allowed. Skip the tile entirely: no load, no multiply.
This is how a causal mask saves about half the work, and a document mask much more. PyTorch’s FlexAttention (Dong et al., 2024) lets you write the mask as a small Python function of \((b, h, i, j)\); it then builds this tile table (a “block mask”) and generates a kernel that skips empty tiles. [Certain] The tile size matters: a partial tile costs as much as a full one, so smaller tiles follow the shape of the mask more closely.
- for each query block \(Q_a\) (\(B\) rows):
- \(m \leftarrow -\infty,\ \ell \leftarrow 0,\ O \leftarrow 0\) // running max, sum, output
- for each key block \(K_b\) (\(B\) columns):
- \(n \leftarrow\) allowed pairs in tile \((a, b)\) // from the block mask
- if \(n = 0\): skip // no load, no multiply
- elif \(n = B^2\): \(S \leftarrow Q_a K_b^\top\) // full: no mask needed
- else: \(S \leftarrow Q_a K_b^\top + M_{ab}\) // partial: mask inside
- update \(m, \ell, O\) with \(S\) and \(V_b\) // online softmax
- write \(O / \ell\) for the rows of \(Q_a\)
How to read Figure 5
Press Play (or All tiles to see the result at once). The thick frame is the tile the kernel is looking at, and the box on the right shows its \(n\) and the branch it takes. With the causal mask and tiles of 128, the result is 28 full, 8 partial (the diagonal) and 28 skipped tiles: the kernel does 56.3% of the dense work for a mask that allows 50.0% of the pairs. Smaller tiles get closer: 53.1% at tile 64, 62.5% at tile 256. Now switch on the document mask: only 14.5% of pairs are allowed. Watch query block 4 (queries 384 to 511, the second document): its tiles against keys 0 to 383 are below the diagonal, but \(n = 0\), so they are skipped. Query block 7 (queries 768 to 895) has \(n = 8{,}192\) of 16,384 against keys 512 to 639, because a document starts at 832: a partial tile. At tile 128 the result is 4 full, 11 partial and 49 skipped (23.4% of the dense work). At tile 64 the kernel does 17.6% (29 full, 16 partial, 211 skipped), but at tile 256 it does 37.5%, because no tile of 256 is fully inside one document. Document boundaries that do not line up with tile edges create many partial tiles.
Masks in VLMs and video models
Multimodal models put image or video tokens into the same sequence as text, and the mask decides how the two kinds of tokens talk (session 188).
- Image tokens as a prefix. PaliGemma treats the image tokens and the prompt as a prefix with full attention, and the answer as a causal suffix, formula (18.9). [Certain] Many LLaVA-style models instead keep a plain causal mask over the whole sequence, so image token 10 cannot see image token 11. [Likely] Gemma 3 uses bidirectional attention among the tokens of one image inside an otherwise causal model. [Likely]
- Loss mask on images. Image tokens are inputs, not things to predict. Their labels are set to −100, just like padding and the prompt.
- Packing many images and samples. When VLM training packs several samples into one row, each sample gets its own document ID, and the image tokens of sample 1 must not be visible to sample 2. The document mask (18.11) does this.
- Video and streaming. A natural mask for streaming video is block-causal: tokens of one frame see each other fully, and see all earlier frames, but no later frame. It is the prefix-LM idea applied frame by frame, and it keeps the KV cache property, so a live video model can process frame after frame without recomputing the past. [Likely]
Where this is used
| Application | Mask | Why | Link |
|---|---|---|---|
| Pretraining a decoder-only LLM | causal + document mask, packed rows | train on every position at once, no padding waste, no cross-document reading | session 31 |
| Supervised fine-tuning (chat) | causal attention; loss mask on the prompt, padding and other samples | learn only the assistant’s answer | session 54 |
| Text classification, NER, embeddings | bidirectional + padding mask | each token needs the whole sentence | session 33 |
| Batched generation and serving | left padding, or no padding with variable-length kernels and the KV cache | append tokens to all rows at once; never recompute the past | session 135 |
| VLM answering about an image (PaliGemma) | prefix-LM: image + prompt full, answer causal | image and question see each other; answer is generated | session 182 |
| Video-LLMs on long or live video | block-causal by frame; document masks when packing clips | streaming with a KV cache, many clips per training row | session 205 |
| Video retrieval and action recognition encoders | bidirectional over all frame and patch tokens, padding mask for clips of different lengths | a clip embedding should use the whole clip | session 208 |
| Long-context models | sliding-window mask (each token sees the last \(w\) tokens) | cost grows like \(T w\) instead of \(T^2\); Mistral 7B uses \(w = 4096\) [Certain] | session 27 |
Pseudo-code
- input: attention bits \(m \in \{0,1\}^{B \times T}\), document IDs \(\mathrm{doc} \in \mathbb{N}^{B \times T}\) (or none), prefix lengths \(P_b\) (or none), mode
- for \(b = 1\) to \(B\), \(i = 1\) to \(T\), \(j = 1\) to \(T\) do
- ok \(\leftarrow m_{b,j} = 1\) // padding: block pad keys, formula (18.6)
- if mode is causal then ok \(\leftarrow\) ok and \(j \le i\) // (18.4)
- if mode is prefix-LM then ok \(\leftarrow\) ok and (\(j \le P_b\) or \(j \le i\)) // (18.9), positions from 1 here
- if doc is given then ok \(\leftarrow\) ok and \(\mathrm{doc}_{b,i} = \mathrm{doc}_{b,j}\) // (18.11)
- \(M_{b,i,j} \leftarrow 0\) if ok else \(-\infty\)
- assert every row of a real query has at least one allowed key // avoids NaN rows
- return \(M\) // broadcast over heads: shape (B, 1, T, T)
- input: samples with lengths \(L_1, \dots, L_n\) (each \(\le T\)), row length \(T\)
- sort samples by length, longest first
- for each sample \(s\) do
- put \(s\) in the first row whose used length \(+ L_s \le T\); if none, open a new row
- for each row do
- concatenate its samples; doc IDs \(\leftarrow 0, 0, \dots, 1, 1, \dots\) // one ID per sample
- position IDs restart at 0 at every sample start
- labels \(\leftarrow -100\) at padding and at the first token of each sample // do not predict across a boundary
- fill the rest of the row with padding, attention bit 0
- return token IDs, doc IDs, position IDs, labels // mask: Algorithm 1 or a varlen kernel
- input: \(Q, K, V \in \R^{T \times d_k}\), mask \(M \in \{0, -\infty\}^{T \times T}\)
- \(S \leftarrow QK^\top / \sqrt{d_k} + M\) // formula (18.1)
- for each row \(i\): \(S_{i,:} \leftarrow S_{i,:} - \max_j S_{ij}\) // numerical safety; the max is over allowed keys
- \(A \leftarrow \exp(S)\) row-normalized // blocked entries are exactly 0, formula (18.2)
- return \(A V\)
The same ideas in PyTorch. Every block checks one claim from this session (tested with PyTorch 2.8 on CPU):
import torch import torch.nn.functional as F torch.manual_seed(0) def causal_mask(T): # True = may attend (SDPA convention) return torch.ones(T, T, dtype=torch.bool).tril() def prefix_mask(T, P): # prefix sees itself fully, rest is causal m = causal_mask(T) m[:, :P] = True # every query sees the whole prefix return m def document_mask(doc_ids): # doc_ids: (T,) like [0,0,0,1,1,2,2,2] same = doc_ids[:, None] == doc_ids[None, :] return same & causal_mask(len(doc_ids)) def padding_mask(attn): # attn: (B, T), 1 = real token, 0 = pad return attn[:, None, None, :].bool() # (B, 1, 1, T): blocks pad KEYS only # 1) allowed pairs for T = 8, formulas (18.5), (18.10), (18.12) print(causal_mask(8).sum().item(), prefix_mask(8, 3).sum().item(), document_mask(torch.tensor([0,0,0,1,1,2,2,2])).sum().item()) # 36 39 15 # 2) leak test: change token 6, outputs 1..5 must not move under the causal mask T, d = 8, 16 x = torch.randn(1, 1, T, d) y = F.scaled_dot_product_attention(x, x, x, attn_mask=causal_mask(T)) x2 = x.clone(); x2[..., 5, :] += 1.0 y2 = F.scaled_dot_product_attention(x2, x2, x2, attn_mask=causal_mask(T)) print((y - y2).abs().amax(-1)[0, 0] > 1e-6) # False x5, then True x3 print(torch.allclose(F.scaled_dot_product_attention(x, x, x, is_causal=True), y)) # True # 3) packing with a document mask == running each document alone docs = torch.tensor([0, 0, 0, 1, 1, 2, 2, 2]) yp = F.scaled_dot_product_attention(x, x, x, attn_mask=document_mask(docs)) parts, s = [], 0 for L in (3, 2, 3): xs = x[..., s:s+L, :] parts.append(F.scaled_dot_product_attention(xs, xs, xs, is_causal=True)); s += L print(torch.allclose(yp, torch.cat(parts, dim=2), atol=1e-6)) # True # 4) padding: a padded row gives the same outputs as the short sequence alone attn = torch.tensor([[1, 1, 1, 1, 1, 1, 0, 0]]) m = causal_mask(T)[None, None] & padding_mask(attn) ypad = F.scaled_dot_product_attention(x, x, x, attn_mask=m) xs = x[..., :6, :] print(torch.allclose(ypad[..., :6, :], F.scaled_dot_product_attention(xs, xs, xs, is_causal=True), atol=1e-6)) # True # 5) a row with every key blocked: -inf gives NaN, finfo.min gives a uniform row s0 = torch.zeros(2) print(torch.softmax(s0 + torch.tensor([float("-inf")] * 2), -1)) # tensor([nan, nan]) print(torch.softmax(s0 + torch.finfo(torch.float32).min, -1)) # tensor([0.5000, 0.5000]) # 6) loss mask, formula (18.7): no loss on padding, and (prefix-LM) none on the prefix ids = torch.tensor([[5, 9, 2, 7, 3, 4, 0, 0]]) labels = ids.clone() labels[attn == 0] = -100 # cross_entropy ignores -100 labels[:, :3] = -100 # no loss on the 3 prefix tokens logits = torch.randn(1, T, 10) loss = F.cross_entropy(logits[:, :-1].reshape(-1, 10), labels[:, 1:].reshape(-1)) print(int((labels[:, 1:] != -100).sum())) # 3 targets count # 7) left padding: position ids from the attention mask left = torch.tensor([[0, 0, 1, 1, 1], [1, 1, 1, 1, 1]]) pos = (left.cumsum(-1) - 1).clamp(min=0) print(pos.tolist()) # [[0, 0, 0, 1, 2], [0, 1, 2, 3, 4]] # 8) is_causal is top-left aligned: wrong for 1 new query against a KV cache q, k = torch.randn(1, 1, 1, d), torch.randn(1, 1, 5, d) a = F.scaled_dot_product_attention(q, k, k, is_causal=True) # sees key 1 only b = F.scaled_dot_product_attention(q, k, k) # sees all 5 keys print(torch.allclose(a, k[..., :1, :]), torch.allclose(a, b)) # True False
Common mistakes
- No causal mask in a decoder. The training loss looks wonderful (the model copies the next token from its input), and generation is garbage. Always run the leak test of Figure 3 on new attention code.
- Flipped boolean convention.
nn.MultiheadAttentionuses True = blocked; SDPA uses True = allowed; Hugging Face uses 1 = real token. Passing one where another is expected makes the model attend to exactly the wrong tokens, and it still trains. - Masking padding in attention but not in the loss (or the reverse). The attention mask stops tokens from reading pads; the loss mask (−100 labels) stops the model from learning to predict pads. You need both.
- Right padding for batched generation. New tokens are appended after the pads, and short rows continue from padding. Use left padding (with correct position IDs) or a server that handles variable lengths.
- Packing without a document mask or without resetting positions. Samples read each other (cross-contamination), and position IDs keep counting across samples, so a short sample at the end of a row is seen at positions it would never have alone.
- Rows where every key is blocked. \(-\infty\) everywhere gives NaN; a finite “minimum” value gives a silent uniform average of forbidden tokens. This happens with left padding plus a causal mask (the pad query at position 1 can only see itself, which is padding) and with empty samples. Make sure every real query sees at least itself; ignore pad rows in the loss.
- Using
is_causal=Truewith a KV cache. In PyTorch SDPA it is top-left aligned: one new query would see only the first cached key. Use no mask for a single new token, or a bottom-right causal mask for several. - Confusing BERT’s [MASK] with an attention mask. Masked language modeling replaces input tokens; BERT’s attention stays fully bidirectional (except for padding).
Interview questions
What does an attention mask do, mathematically?basic
Why does a decoder-only LM need a causal mask during training but the problem does not appear at inference?basic
How does the causal mask make the KV cache possible?core
Explain padding mask versus loss mask. Do you need both?core
Left or right padding for batched generation? What else must change?core
Compare bidirectional, causal and prefix-LM attention. When would you use each?core
What is sequence packing, and what goes wrong without a document mask?core
Your packed training run gets NaN loss after you added a document mask with left padding. What is a likely cause?advanced
How do fast attention kernels exploit masks? Why does tile size matter?advanced
In PyTorch, what is the difference between True in nn.MultiheadAttention’s key_padding_mask and True in scaled_dot_product_attention’s attn_mask?advanced
nn.MultiheadAttention, True means “ignore this key”. In F.scaled_dot_product_attention, a boolean True means “this pair takes part in attention”. Hugging Face tokenizers return 1 for real tokens. Mixing them up makes the model attend to exactly the wrong set; it can still train, so always test with a leak test and a padding-invariance test (outputs for real tokens must not change when you add padding).How are masks used in a VLM like PaliGemma, and for streaming video?advanced
Exercises
- A query has scaled scores \((1.0,\ 3.0,\ 0.0)\) for keys 1 to 3 and the mask row \((0,\ -\infty,\ 0)\). Compute the attention weights.
- How many (query, key) pairs does a causal mask allow for \(T = 4096\)? What fraction of \(T^2\) is that?
- For \(T = 10\) and a prefix of \(P = 4\), how many pairs does the prefix-LM mask allow? How many more than causal?
- A batch has samples of 100, 400, 50 and 250 tokens, padded to the longest. What fraction of the token slots is padding?
- A packed row of \(T = 12\) has documents of 5, 4 and 3 tokens. Write the document ID of each position, the reset position IDs, and count the allowed pairs under the document mask.
- In Figure 3, Causal + padding mode, why does editing a pad token change its own output but no other output? Would that still hold without the padding mask?
- Write the boolean mask (True = may attend, SDPA convention) for a 2-row batch with attention bits
[1,1,1,0]and[1,1,1,1]and a causal mask. Which rows of row 1 are “fully padding”?
Answerscheck
Key takeaways
- A mask adds 0 or \(-\infty\) to each attention score before the softmax (18.1). Blocked keys get weight and gradient exactly zero; the softmax renormalizes over the allowed keys (18.2).
- The causal mask (18.4) lets a model train on all \(T\) positions at once without seeing the answers, and it is what makes the KV cache possible. It allows \(T(T+1)/2\) pairs (18.5).
- The padding mask blocks pad keys (18.6); the loss mask (labels −100) removes positions from the loss (18.7). They are different masks and you need both. Use left padding for batched generation, with position IDs from the mask.
- Bidirectional (BERT) sees everything, causal (GPT) sees the past, prefix-LM (T5 variant, UniLM, PaliGemma) is bidirectional on the prefix and causal on the target (18.9).
- Packing removes padding; a block-diagonal document mask (18.11) and reset position IDs make a packed row equal to separate rows and stop cross-contamination.
- In code, check what True means, avoid fully blocked rows, and prefer
is_causal, varlen kernels or block masks, which skip blocked tiles instead of computing and discarding them.
Key formulas
References for this session
- Raschka, Build a Large Language Model (From Scratch), chapter 3: Coding attention mechanisms
- Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Raschka, Build a Large Language Model (From Scratch), chapter 3: Coding attention mechanisms
- Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Vaswani et al. (2017). Attention Is All You Need
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Dong, Yang, Wang, Wei, Liu, Wang, Gao, Zhou, Hon (2019). Unified Language Model Pre-training for Natural Language Understanding and Generation
- Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
- Krell, Kosec, Perez, Fitzgibbon (2021). Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance
- Zhao, Qu, Staniszewski, Tworkowski, Liu, Miłoś, Wu, Minervini (2024). Analysing The Impact of Sequence Composition on Language Model Pre-Training
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Dong, Feng, Guessous, Liang, He (2024). Flex Attention: A Programming Model for Generating Optimized Attention Kernels
- Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Jiang et al. (2023). Mistral 7B