LLM & VLM courseBlog
18Track 2: Attention and transformers · session 3 of 15

Masks: causal, padding, bidirectional, prefix and packing masks

Attention lets every token look at every other token. A mask decides which of those looks are allowed. This session shows how one simple trick, adding minus infinity to a score before the softmax, gives the causal mask of GPT, the padding mask for batches, the full view of BERT, the prefix-LM view used by T5 and PaliGemma, and the document mask that keeps packed samples apart.

Time: 2 to 3 hours (reading 1.5 hours, exercises 1 hour)Level: BeginnerBefore this: Session 17 (scaled dot-product attention) and session 9 (special tokens and padding)

What you will learn

  • How a mask works: add 0 or minus infinity to each attention score, so blocked pairs get exactly zero weight
  • The causal mask: why a language model must not see the future during training, and how this lets it learn from all positions at once
  • Padding masks for batches of different lengths, left versus right padding, and the separate loss mask
  • Bidirectional (BERT), causal (GPT) and prefix-LM (T5, UniLM, PaliGemma) attention patterns, and what each is good for
  • Sequence packing and the document mask that stops packed samples from seeing each other
  • How real code expresses masks (boolean versus additive, the True means keep or block trap, NaN rows) and how fast kernels skip blocked tiles
  • How the same masks appear in VLMs and video models

Why this matters

In session 17 every query got a score with every key, and softmax turned each row of scores into weights. Nothing stopped a token from looking anywhere. Real models need rules: “do not look at the future”, “do not look at padding”, “do not look at a different document”. A mask is the tool that writes these rules into attention.

A mask is a table with one entry for each (query, key) pair. The entry says “allowed” or “blocked”. Blocked pairs get exactly zero attention weight. That is the whole idea, and it is surprisingly powerful. The same mechanism, with different tables, gives:

  • The causal mask of GPT-style models: each token sees only itself and the tokens before it. This is what makes next-token training honest, and it is why a KV cache works.
  • The padding mask: sequences in a batch have different lengths, so we fill the short ones with padding tokens and tell attention to ignore them.
  • The bidirectional (full) view of BERT, the causal view of GPT, and the prefix-LM view in between, used by T5 experiments, UniLM and PaliGemma.
  • The document mask: when we pack many short samples into one long training row, it stops one sample from reading another.

Masks look like a small detail, but many real bugs live here: a model that “cheats” in training and fails at generation, a batch whose outputs change when you add padding, a fine-tune that learns from padding tokens, NaN losses from a row where everything was blocked. Interviewers know this. Typical questions: “Why do we need a causal mask?”, “Left or right padding for generation, and why?”, “What is the difference between an attention mask and a loss mask?”, “What goes wrong when you pack sequences without a document mask?”. By the end of this session you can answer each one with a formula, a picture and a few lines of code.

One mechanism: add minus infinity before the softmax

Recall the attention of one head. The queries \(Q\), keys \(K\) and values \(V\) have one row per token. The scores are \(QK^\top/\sqrt{d_k}\), one number per (query, key) pair. A mask is a matrix \(M\) of the same shape, added to the scores before the softmax:

\[ \operatorname{Attn}(Q, K, V) = \softmax\!\Big(\frac{QK^\top}{\sqrt{d_k}} + M\Big)\, V, \qquad M_{ij} = \begin{cases} 0 & \text{if query } i \text{ may see key } j \\ -\infty & \text{otherwise} \end{cases} \tag{18.1} \]
Read the symbols
\(Q, K, V\)
queries, keys and values, one row per token (shape \(T \times d_k\) for \(T\) tokens)
\(d_k\)
the size of one query or key vector; dividing by \(\sqrt{d_k}\) keeps scores at a sane size (session 17)
\(M\)
the mask, a \(T \times T\) matrix of 0 and \(-\infty\). Row \(i\) is the rule for query \(i\); column \(j\) is key \(j\).
softmax
applied to each row separately, so each query gets weights that sum to 1

Why does adding \(-\infty\) work? Write \(s_{ij}\) for the scaled score. The weight of key \(j\) for query \(i\) is:

\[ a_{ij} = \frac{e^{\,s_{ij} + M_{ij}}}{\sum_{l} e^{\,s_{il} + M_{il}}} = \begin{cases} \dfrac{e^{\,s_{ij}}}{\sum_{l \in \mathcal{A}_i} e^{\,s_{il}}} & j \in \mathcal{A}_i \\[2mm] 0 & j \notin \mathcal{A}_i \end{cases} \tag{18.2} \]
Read the symbols
\(a_{ij}\)
the attention weight of key \(j\) in the output of query \(i\)
\(\mathcal{A}_i\)
the set of keys that query \(i\) is allowed to see (where \(M_{ij} = 0\))
\(e^{-\infty} = 0\)
a blocked key adds nothing to the sum and gets weight exactly 0

So a masked softmax is just an ordinary softmax over the allowed keys only. The blocked keys do not get a small weight; they get zero, and their values never reach the output. Their gradient is zero too: no learning signal flows through a blocked pair.

Worked example: one masked row

A sequence has 4 tokens. Query 2 has the scaled scores \(s_{2,\cdot} = (2.0,\ 1.0,\ 0.5,\ 3.0)\) for keys 1 to 4. Without a mask, softmax gives the weights \((0.232,\ 0.085,\ 0.052,\ 0.631)\): most of the weight goes to key 4, a token that comes later in the sentence.

With the causal mask, row 2 is \(M_{2,\cdot} = (0, 0, -\infty, -\infty)\). Only keys 1 and 2 remain: \(e^{2}/(e^{2} + e^{1}) = 7.389/10.107 = 0.731\) and \(e^{1}/10.107 = 0.269\). The weights are \((0.731,\ 0.269,\ 0,\ 0)\). They still sum to 1, and the future has no effect.

Two ways to store a mask are common. An additive mask is a float matrix of 0 and \(-\infty\) (or a very negative number), added exactly as in (18.1). A boolean mask is a matrix of True and False that the library turns into the additive form. Be careful: libraries do not agree on what True means. We come back to this in the section on code.

Figure 1 runs this computation for real, one query row at a time, on 8 tokens. The scores are a fixed table of numbers. For each row it builds the mask row, adds it, takes the maximum over the allowed keys, exponentiates, and divides by the sum: exactly formula (18.2), done the numerically safe way of Algorithm 3. Switch the mask at any moment and watch which cells survive.

allowed: \(s + 0\)blocked: \(s + (-\infty) = -\infty\)row maximum \(m\)weight \(a\) (darker = larger)
  1. \(S \leftarrow QK^\top / \sqrt{d_k}\) // one score per \((i, j)\)
  2. for each query row \(i = 1, \dots, 8\):
  3. \(M_{ij} \leftarrow 0\) if allowed\((i, j)\) else \(-\infty\)
  4. \(m \leftarrow \max_j \,(s_{ij} + M_{ij})\) // only allowed keys can win
  5. \(e_j \leftarrow \exp(s_{ij} + M_{ij} - m)\); \(Z \leftarrow \sum_j e_j\) // \(e^{-\infty} = 0\)
  6. \(a_{ij} \leftarrow e_j / Z\) // (18.2): the row sums to 1
  7. return \(A\) // then the output is \(AV\)
Figure 1. The masked softmax, computed live. Left: the scaled scores \(s_{ij}\) (a fixed seeded table, the same one as in Figure 2). When a row is processed, its blocked cells become \(-\infty\) (red). Right: the attention weights \(a_{ij}\), filled in row by row; the orange numbers are the values \(e_j\) before the division by \(Z\). The box shows the line of the algorithm that runs now, and the formula with the live numbers.
How to read Figure 1

Press Play, or use Step and Back. Each row takes four steps: build the mask row (line 3), find the maximum over the allowed keys (line 4), exponentiate and sum (line 5), divide (line 6). In Causal mode, row 4 (“on”) keeps keys 1 to 4; the maximum is \(m = 0.4\) (key 2), the values are \(e = (0.202, 1, 0.202, 0.202)\), \(Z = 1.606\), and the weights are \((0.126, 0.623, 0.126, 0.126)\). Now switch to Bidirectional without stopping: the same row also keeps keys 5 to 8, the maximum moves to key 5 (1.2), and key 5 takes 0.458 of the weight. In Packed mode row 4 is the first token of document 2 (“hi”), so it may see only itself: weight 1. In Causal + padding the red columns 7 and 8 are the pads. The right matrix is always zero in every red cell, and every finished row sums to 1.

The causal mask: no peeking at the future

Why a language model needs it

A language model is trained to predict each token from the tokens before it (session 6). The probability of a whole text is a product of these one-step predictions:

\[ \log p(x_1, \dots, x_T) = \sum_{t=1}^{T} \log p\big(x_t \mid x_1, \dots, x_{t-1}\big) \tag{18.3} \]

During training we feed the full text at once and ask, at every position \(t\), for the next token \(x_{t+1}\). This gives \(T\) training examples from one forward pass, which is the main reason transformers train so fast. But there is a trap. If position \(t\) can attend to position \(t+1\), the answer is sitting in its input. The model learns to copy it, the training loss drops close to zero, and the model is useless at generation time, when the future does not exist yet.

The causal mask (also called the look-ahead mask or autoregressive mask) closes this door. Query \(i\) may see key \(j\) only when \(j\) is not after \(i\):

\[ M^{\text{causal}}_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j \gt i \end{cases} \tag{18.4} \]

In the matrix picture, the allowed entries are the diagonal and everything below it: a lower triangle. Row 1 sees one key, row 2 sees two keys, and row \(T\) sees all \(T\). The total number of allowed pairs is

\[ N_{\text{causal}} = 1 + 2 + \dots + T = \frac{T(T+1)}{2} \tag{18.5} \]

For \(T = 8\) that is 36 of 64 pairs. For large \(T\) it approaches half of \(T^2\); for \(T = 1024\) it is 524,800 of 1,048,576, which is 50.0%. A kernel that skips the blocked half can save almost half of the attention work (Figure 5).

What the causal mask buys at inference

The causal mask has a second, practical effect. Because token \(i\) never looks at later tokens, its hidden states in every layer do not change when new tokens are appended. So during generation we can compute the keys and values of old tokens once and keep them. This is the KV cache (session 80). A bidirectional model cannot do this: every new token would change the states of all earlier tokens. Figure 3 shows this “nothing in the past changes” property directly.

Figure 2. The mask explorer. An 8-token sequence, a fixed table of raw scores (seeded), and five masks. Rows are queries, columns are keys. Colored cells are allowed (\(M_{ij} = 0\)), grey cells are blocked (\(M_{ij} = -\infty\)). The selected row shows the attention weights after the masked softmax (18.2). Click any cell to select its row.
How to read Figure 2

Start in Causal mode with row 4 (the query “on”). It may see keys 1 to 4, and the weights are (0.126, 0.623, 0.126, 0.126). Switch to Bidirectional: the same row now also sees keys 5 to 8, and almost half of its weight (0.458) goes to key 5, a word that comes later. That is exactly the leak the causal mask prevents. In Prefix-LM mode the dashed square is fully colored: the first \(P\) tokens see each other in both directions; the rest is causal. In Packed mode the sequence holds three documents (“the cat sat”, “hi there”, “a dog ran”); each one is a small causal triangle and nothing crosses between them: 15 allowed pairs instead of 36. In Causal + padding the last two columns are grey for every row, so no real token reads the padding.

The padding mask: batches with different lengths

Why padding exists

GPUs work on rectangular tensors. A batch of \(B\) sequences must have shape \(B \times T\), but real samples have different lengths. The usual fix is padding: add a special pad token (see session 9) to the short sequences until all have the same length. Together with the token IDs the tokenizer returns an attention mask \(m\): 1 for a real token, 0 for padding.

Padding tokens are not text. If real tokens attend to them, the output depends on how much padding the batch happened to have: the same sentence gives a different answer in a different batch. So we block every key that is padding:

\[ M_{ij} = \begin{cases} 0 & j \le i \ \text{ and } \ m_j = 1 \\ -\infty & \text{otherwise} \end{cases} \tag{18.6} \]
Read the symbols
\(m_j\)
the attention-mask bit of token \(j\) from the tokenizer: 1 = real token, 0 = padding
\(j \le i\)
the causal part; drop it for a bidirectional encoder such as BERT

Notice what is blocked: the padding columns (keys). The padding rows (queries) are still computed; they produce some output, but nobody reads it. This is fine because of a second, separate mask.

The loss mask is a different mask

The attention mask controls what tokens see. The loss mask controls which positions count in the loss. Padding positions must not be trained on: otherwise the model learns to predict “pad after pad”, and the loss is diluted by easy fake targets. With per-position losses \(\ell_t\) and weights \(w_t \in \{0, 1\}\):

\[ \mathcal{L} = \frac{\sum_{t} w_t\, \ell_t}{\sum_{t} w_t} \tag{18.7} \]

In PyTorch and Hugging Face code this is done by setting the label of an ignored position to -100; F.cross_entropy skips it and divides by the number of remaining targets. [Certain] The same tool also removes the loss on the prompt in supervised fine-tuning, so the model is trained only on the answer (session 54).

Worked example: how much of a batch is padding?

A batch has four samples with 5, 12, 3 and 8 tokens. Padding to the longest gives a \(4 \times 12\) tensor: 48 slots for \(5 + 12 + 3 + 8 = 28\) real tokens. The padding fraction is

\[ 1 - \frac{\sum_b L_b}{B \cdot \max_b L_b} = 1 - \frac{28}{48} = 0.417 \tag{18.8} \]

So 41.7% of the compute in the linear layers is spent on tokens that are then thrown away. Attention is worse, because its cost grows with the square of the row length: \(4 \times 12^2 = 576\) score entries, while the real samples need only \(5^2 + 12^2 + 3^2 + 8^2 = 242\). Krell et al. (2021) report that up to 50% of all tokens can be padding in common NLP datasets, and up to 89% for one GLUE task at length 128. [Certain]

Left or right padding?

For training, right padding (pads at the end) is the usual choice. For batched generation with a decoder-only model, most libraries want left padding (pads at the start). The reason: generation appends the next token at the end of every row. With right padding, the new token of a short row would land after its pads, far from its last real token, and the model would be asked to continue a text that ends in padding. With left padding, the last position of every row is a real token, so all rows can be extended in lockstep. [Certain]

Left padding has one catch: positions. If row 1 is pad pad the cat, the word “the” should still be position 0, not position 2. Libraries compute position IDs from the attention mask, for example the cumulative sum of \(m\) minus 1. The PyTorch code below shows it. With rotary positions (session 14), wrong position IDs here silently hurt quality.

Bidirectional, causal and prefix-LM

The causal mask is right for generation, but it throws away information when the whole input is already known. To classify a sentence or to tag every word, token 1 would like to see token 8. Three patterns cover most models; T5 (Raffel et al., 2019) calls them “fully-visible”, “causal” and “causal with prefix”. [Certain]

  • Bidirectional (fully-visible). \(M = 0\) everywhere (except padding). Every token sees every token. Used by encoders such as BERT, which is trained by hiding 15% of the input tokens and predicting them (masked language modeling). [Certain] Careful with words: the [MASK] token of BERT is a change to the input, not an attention mask. BERT has no causal mask at all.
  • Causal. Formula (18.4). Used by decoder-only models (GPT, Llama, Qwen) and by the decoder of encoder–decoder models.
  • Prefix-LM (causal with prefix). The first \(P\) tokens, the prefix (the input or prompt), see each other in both directions. The rest, the target, is causal and also sees the whole prefix:
\[ M^{\text{prefix}}_{ij} = \begin{cases} 0 & j \lt P \ \text{ or } \ j \le i \\ -\infty & \text{otherwise} \end{cases} \tag{18.9} \]
Read the symbols
\(P\)
the prefix length; positions are counted from 0, so the prefix is positions \(0, \dots, P-1\)
\(j \lt P\)
every query may see every prefix key, even prefix keys to its right
\(j \le i\)
outside the prefix the usual causal rule holds

A prefix-LM is one stack of layers that behaves like an encoder on the prefix and like a decoder on the target. The loss is put only on the target tokens (a loss mask again). UniLM (Dong et al., 2019) trained one network with all three masks, switching per batch, to serve both understanding and generation tasks. [Certain] PaliGemma uses the prefix-LM pattern for images: the image tokens and the text prompt form the prefix with full attention, and the answer (the suffix) is generated causally. [Certain] This lets image tokens look at the question, and the question look at all of the image.

Counting allowed pairs for the prefix-LM: the \(P\) prefix rows see \(P\) keys each, and row \(i \ge P\) (counting from 0) sees \(i + 1\) keys:

\[ N_{\text{prefix}} = P^2 + \sum_{i=P}^{T-1} (i+1) = P^2 + \frac{T(T+1) - P(P+1)}{2} \tag{18.10} \]

For \(T = 8\) and \(P = 3\): \(9 + (72 - 12)/2 = 39\) pairs, three more than causal. Those three are the pairs inside the prefix that look to the right.

PatternWho sees whatTypical modelsGood forCost
Bidirectionaleveryone sees everyoneBERT, RoBERTa, ViT, the T5 encoder, CLIP image towerunderstanding: classification, tagging, embeddingscannot generate left to right; no KV cache reuse
Causalonly self and pastGPT, Llama, Qwen, Mistral, decodersgeneration; training on every position at onceprompt tokens never see later prompt tokens
Prefix-LMprefix: full; target: causal and sees the prefixUniLM, T5 “prefix LM” variant, PaliGemmaconditional generation: answer a question about an inputloss only on the target; prefix KV must be built with the full mask
Encoder–decoderencoder: full; decoder: causal self-attention plus cross-attention to the encoderT5, BART, Whispertranslation, summarization, speechtwo stacks of weights (session 20)

Packing and document masks

Packing: no more padding

Pretraining text is a stream of documents of very different lengths. Instead of padding each one, we pack them: concatenate many documents, separated by an end-of-text token, and cut the stream into rows of exactly \(T\) tokens. For fine-tuning data, where each sample must stay whole, we use a bin-packing method instead: fill each row with whole samples that fit, for example with first-fit decreasing (sort samples from long to short; put each into the first row with enough room). Krell et al. (2021) report a 2× speed-up for BERT phase-2 pretraining with packing. [Certain]

Cross-contamination and the document mask

Packing creates a new problem. With a plain causal mask, the first token of document 2 can attend to all of document 1. The model then learns from a context that is unrelated text, and in fine-tuning a sample can “see” the answer of another sample. Krell et al. call this cross-contamination. The fix is a document mask (also called intra-document causal mask): give every token a document ID \(\mathrm{doc}(i)\) and allow only pairs inside the same document:

\[ M^{\text{doc}}_{ij} = \begin{cases} 0 & \mathrm{doc}(i) = \mathrm{doc}(j) \ \text{ and } \ j \le i \\ -\infty & \text{otherwise} \end{cases} \tag{18.11} \]

The mask is block-diagonal: one small causal triangle per document, and nothing between them. With document lengths \(L_1, \dots, L_D\) the number of allowed pairs is

\[ N_{\text{doc}} = \sum_{k=1}^{D} \frac{L_k (L_k + 1)}{2} \tag{18.12} \]

To make a packed row exactly equal to separate rows, also reset the position IDs at every document start, so each document begins at position 0. With both, the outputs are identical to running each document alone; the PyTorch code below checks this.

Worked example: how much attention a document mask removes

A row of \(T = 1024\) tokens holds four documents with 384, 128, 320 and 192 tokens. The plain causal mask allows \(1024 \cdot 1025/2 = 524{,}800\) pairs. The document mask allows \(384 \cdot 385/2 + 128 \cdot 129/2 + 320 \cdot 321/2 + 192 \cdot 193/2 = 73{,}920 + 8{,}256 + 51{,}360 + 18{,}528 = 152{,}064\) pairs, only 14.5% of the full \(1024^2\) matrix. A kernel that can skip blocked tiles does far less work (Figure 5).

Does the document mask matter for quality? Llama 3 uses “an attention mask that prevents self-attention between different documents within the same sequence”. [Certain] The same report says this had limited impact in standard pretraining but mattered for continued pretraining on very long sequences. [Likely] Zhao et al. (2024) found that intra-document causal masking improves language modeling and downstream results compared to plain causal packing. [Certain] Some pretraining pipelines still use plain causal packing and rely on the end-of-text token to separate documents. [Likely]

Figure 3. The leak test. One real attention layer (random seeded weights, \(d = 4\)) runs on 8 tokens. Click a token to replace its vector with a different one (marked *). The boxes below show which outputs \(\mathbf{y}_1, \dots, \mathbf{y}_8\) changed, with the size of the change \(\Delta = \lVert \mathbf{y}_i - \mathbf{y}'_i \rVert\).
How to read Figure 3

This is the test you should run on any attention code. In Causal mode, edit token 4: outputs 4 to 8 change and outputs 1 to 3 stay exactly the same (\(\Delta = 0\)), so the past cannot see the future. In Bidirectional mode every output changes. In Prefix-LM mode, editing a prefix token (1 to 3) changes everything, but editing token 6 changes only 6 to 8. In Packed mode, editing “hi” (token 4) changes only 4 and 5: the other two documents do not notice. In Causal + padding mode, edit a pad token (7 or 8): only its own output changes, and that position is ignored by the loss. If any real output moved, your padding mask would be broken.

Figure 4. Padding versus packing. Up to 32 samples with seeded lengths between 24 and 512 tokens (log-uniform). “Pad per batch” takes batches of 8 in the given order and pads each to its longest sample. “Sort, then pad” sorts by length first. “Pack” fills rows of \(T = 512\) with whole samples, first-fit decreasing. Colored bars are samples (hover for the length), grey is padding.
How to read Figure 4

With 16 samples, plain padding wastes 65.3% of the token slots: one long sample in a batch forces 7 short ones to be padded to its length. Sorting puts similar lengths together and cuts the waste to 55.0% (28.2% with 32 samples, where there are more similar lengths to group). Packing needs only 4 rows for the 16 samples and wastes 1.5%. The last line of the readout is about attention: a packed row costs a full \(512^2\) score matrix if you use a plain mask, but the real samples need only \(\sum L_i^2\). With 32 samples, packed rows without a document-aware kernel do more attention work (2,359,296) than sorted padding (2,113,496). Packing wins clearly only when the kernel skips the blocked tiles (Figure 5).

Masks in real code and fast kernels

Boolean, additive, and the “True means what?” trap

Three conventions are in daily use, and they disagree:

APIMask formMeaning of True / 1
F.scaled_dot_product_attention(..., attn_mask=)bool or float, broadcast to \((B, H, T_q, T_k)\)bool True = may attend; float is added to the scores
nn.MultiheadAttention(..., key_padding_mask=, attn_mask=)bool \((B, T_k)\) and \((T_q, T_k)\)True = ignore (blocked)
Hugging Face tokenizer attention_maskint \((B, T)\) of 0 and 11 = real token; the model builds the 4-D mask

We checked the first two in PyTorch 2.8: in nn.MultiheadAttention a True in key_padding_mask gave that key weight 0, while in scaled_dot_product_attention a True means the pair takes part. [Certain] Passing a mask from one API to the other without flipping it gives a model that attends only to padding and the future. It still trains, so this bug can hide for a long time.

A few more practical rules:

  • Use is_causal=True when you can. Kernels such as FlashAttention then never build the \(T \times T\) mask and skip the upper triangle. But note: in PyTorch SDPA, is_causal is aligned to the top-left corner. With 1 query and 5 cached keys, it lets the query see only key 1. For decoding with a KV cache you need the bottom-right version (the new query sees all cached keys), or no mask at all for a single new token. [Certain]
  • Large negative, not always \(-\infty\). Many codes use the most negative number of the data type, for example torch.finfo(dtype).min, to avoid \(-\infty - (-\infty)\) = NaN in some kernels. Then a fully blocked row becomes uniform instead of NaN, which hides the bug instead of showing it.
  • Mask in the right dtype. In float16 the most negative value is −65,504. Adding it to a score and then adding another bias can overflow to \(-\infty\). Build masks in the same dtype as the scores. [Likely]
  • Variable-length kernels. For packed rows, FlashAttention offers a “varlen” interface: you pass the documents back to back plus the cumulative lengths (cu_seqlens), and the kernel applies the document mask (18.11) without ever building it. [Certain]

Block masks: how kernels skip work

Fast attention kernels (session 124) do not work on single (query, key) pairs. They cut the score matrix into tiles of, say, \(128 \times 128\) and process one tile at a time. For each tile, the mask has three cases:

  1. Full: every pair in the tile is allowed. Compute it with no mask at all.
  2. Partial: some pairs are allowed. Compute it and apply the mask inside the tile.
  3. Empty: no pair is allowed. Skip the tile entirely: no load, no multiply.

This is how a causal mask saves about half the work, and a document mask much more. PyTorch’s FlexAttention (Dong et al., 2024) lets you write the mask as a small Python function of \((b, h, i, j)\); it then builds this tile table (a “block mask”) and generates a kernel that skips empty tiles. [Certain] The tile size matters: a partial tile costs as much as a full one, so smaller tiles follow the shape of the mask more closely.

  1. for each query block \(Q_a\) (\(B\) rows):
  2. \(m \leftarrow -\infty,\ \ell \leftarrow 0,\ O \leftarrow 0\) // running max, sum, output
  3. for each key block \(K_b\) (\(B\) columns):
  4. \(n \leftarrow\) allowed pairs in tile \((a, b)\) // from the block mask
  5. if \(n = 0\): skip // no load, no multiply
  6. elif \(n = B^2\): \(S \leftarrow Q_a K_b^\top\) // full: no mask needed
  7. else: \(S \leftarrow Q_a K_b^\top + M_{ab}\) // partial: mask inside
  8. update \(m, \ell, O\) with \(S\) and \(V_b\) // online softmax
  9. write \(O / \ell\) for the rows of \(Q_a\)
Figure 5. A tiled kernel walks over the mask. A packed row of \(T = 1024\) tokens with four documents of 384, 128, 320 and 192 tokens (dashed squares). The kernel visits the tiles one query block at a time. For each tile it counts the allowed pairs \(n\) and decides: full (computed without a mask), partial (computed, then masked) or skipped. Tiles not visited yet are drawn as outlines.
How to read Figure 5

Press Play (or All tiles to see the result at once). The thick frame is the tile the kernel is looking at, and the box on the right shows its \(n\) and the branch it takes. With the causal mask and tiles of 128, the result is 28 full, 8 partial (the diagonal) and 28 skipped tiles: the kernel does 56.3% of the dense work for a mask that allows 50.0% of the pairs. Smaller tiles get closer: 53.1% at tile 64, 62.5% at tile 256. Now switch on the document mask: only 14.5% of pairs are allowed. Watch query block 4 (queries 384 to 511, the second document): its tiles against keys 0 to 383 are below the diagonal, but \(n = 0\), so they are skipped. Query block 7 (queries 768 to 895) has \(n = 8{,}192\) of 16,384 against keys 512 to 639, because a document starts at 832: a partial tile. At tile 128 the result is 4 full, 11 partial and 49 skipped (23.4% of the dense work). At tile 64 the kernel does 17.6% (29 full, 16 partial, 211 skipped), but at tile 256 it does 37.5%, because no tile of 256 is fully inside one document. Document boundaries that do not line up with tile edges create many partial tiles.

Masks in VLMs and video models

Multimodal models put image or video tokens into the same sequence as text, and the mask decides how the two kinds of tokens talk (session 188).

  • Image tokens as a prefix. PaliGemma treats the image tokens and the prompt as a prefix with full attention, and the answer as a causal suffix, formula (18.9). [Certain] Many LLaVA-style models instead keep a plain causal mask over the whole sequence, so image token 10 cannot see image token 11. [Likely] Gemma 3 uses bidirectional attention among the tokens of one image inside an otherwise causal model. [Likely]
  • Loss mask on images. Image tokens are inputs, not things to predict. Their labels are set to −100, just like padding and the prompt.
  • Packing many images and samples. When VLM training packs several samples into one row, each sample gets its own document ID, and the image tokens of sample 1 must not be visible to sample 2. The document mask (18.11) does this.
  • Video and streaming. A natural mask for streaming video is block-causal: tokens of one frame see each other fully, and see all earlier frames, but no later frame. It is the prefix-LM idea applied frame by frame, and it keeps the KV cache property, so a live video model can process frame after frame without recomputing the past. [Likely]

Where this is used

ApplicationMaskWhyLink
Pretraining a decoder-only LLMcausal + document mask, packed rowstrain on every position at once, no padding waste, no cross-document readingsession 31
Supervised fine-tuning (chat)causal attention; loss mask on the prompt, padding and other sampleslearn only the assistant’s answersession 54
Text classification, NER, embeddingsbidirectional + padding maskeach token needs the whole sentencesession 33
Batched generation and servingleft padding, or no padding with variable-length kernels and the KV cacheappend tokens to all rows at once; never recompute the pastsession 135
VLM answering about an image (PaliGemma)prefix-LM: image + prompt full, answer causalimage and question see each other; answer is generatedsession 182
Video-LLMs on long or live videoblock-causal by frame; document masks when packing clipsstreaming with a KV cache, many clips per training rowsession 205
Video retrieval and action recognition encodersbidirectional over all frame and patch tokens, padding mask for clips of different lengthsa clip embedding should use the whole clipsession 208
Long-context modelssliding-window mask (each token sees the last \(w\) tokens)cost grows like \(T w\) instead of \(T^2\); Mistral 7B uses \(w = 4096\) [Certain]session 27

Pseudo-code

Algorithm 1: build the attention mask for one batch
  1. input: attention bits \(m \in \{0,1\}^{B \times T}\), document IDs \(\mathrm{doc} \in \mathbb{N}^{B \times T}\) (or none), prefix lengths \(P_b\) (or none), mode
  2. for \(b = 1\) to \(B\), \(i = 1\) to \(T\), \(j = 1\) to \(T\) do
  3. ok \(\leftarrow m_{b,j} = 1\) // padding: block pad keys, formula (18.6)
  4. if mode is causal then ok \(\leftarrow\) ok and \(j \le i\) // (18.4)
  5. if mode is prefix-LM then ok \(\leftarrow\) ok and (\(j \le P_b\) or \(j \le i\)) // (18.9), positions from 1 here
  6. if doc is given then ok \(\leftarrow\) ok and \(\mathrm{doc}_{b,i} = \mathrm{doc}_{b,j}\) // (18.11)
  7. \(M_{b,i,j} \leftarrow 0\) if ok else \(-\infty\)
  8. assert every row of a real query has at least one allowed key // avoids NaN rows
  9. return \(M\) // broadcast over heads: shape (B, 1, T, T)
Algorithm 2: pack samples into rows (first-fit decreasing) with document IDs
  1. input: samples with lengths \(L_1, \dots, L_n\) (each \(\le T\)), row length \(T\)
  2. sort samples by length, longest first
  3. for each sample \(s\) do
  4. put \(s\) in the first row whose used length \(+ L_s \le T\); if none, open a new row
  5. for each row do
  6. concatenate its samples; doc IDs \(\leftarrow 0, 0, \dots, 1, 1, \dots\) // one ID per sample
  7. position IDs restart at 0 at every sample start
  8. labels \(\leftarrow -100\) at padding and at the first token of each sample // do not predict across a boundary
  9. fill the rest of the row with padding, attention bit 0
  10. return token IDs, doc IDs, position IDs, labels // mask: Algorithm 1 or a varlen kernel
Algorithm 3: masked attention, one head
  1. input: \(Q, K, V \in \R^{T \times d_k}\), mask \(M \in \{0, -\infty\}^{T \times T}\)
  2. \(S \leftarrow QK^\top / \sqrt{d_k} + M\) // formula (18.1)
  3. for each row \(i\): \(S_{i,:} \leftarrow S_{i,:} - \max_j S_{ij}\) // numerical safety; the max is over allowed keys
  4. \(A \leftarrow \exp(S)\) row-normalized // blocked entries are exactly 0, formula (18.2)
  5. return \(A V\)

The same ideas in PyTorch. Every block checks one claim from this session (tested with PyTorch 2.8 on CPU):

import torch
import torch.nn.functional as F
torch.manual_seed(0)

def causal_mask(T):                       # True = may attend (SDPA convention)
    return torch.ones(T, T, dtype=torch.bool).tril()

def prefix_mask(T, P):                    # prefix sees itself fully, rest is causal
    m = causal_mask(T)
    m[:, :P] = True                       # every query sees the whole prefix
    return m

def document_mask(doc_ids):               # doc_ids: (T,) like [0,0,0,1,1,2,2,2]
    same = doc_ids[:, None] == doc_ids[None, :]
    return same & causal_mask(len(doc_ids))

def padding_mask(attn):                   # attn: (B, T), 1 = real token, 0 = pad
    return attn[:, None, None, :].bool()  # (B, 1, 1, T): blocks pad KEYS only

# 1) allowed pairs for T = 8, formulas (18.5), (18.10), (18.12)
print(causal_mask(8).sum().item(), prefix_mask(8, 3).sum().item(),
      document_mask(torch.tensor([0,0,0,1,1,2,2,2])).sum().item())   # 36 39 15

# 2) leak test: change token 6, outputs 1..5 must not move under the causal mask
T, d = 8, 16
x = torch.randn(1, 1, T, d)
y = F.scaled_dot_product_attention(x, x, x, attn_mask=causal_mask(T))
x2 = x.clone(); x2[..., 5, :] += 1.0
y2 = F.scaled_dot_product_attention(x2, x2, x2, attn_mask=causal_mask(T))
print((y - y2).abs().amax(-1)[0, 0] > 1e-6)   # False x5, then True x3
print(torch.allclose(F.scaled_dot_product_attention(x, x, x, is_causal=True), y))  # True

# 3) packing with a document mask == running each document alone
docs = torch.tensor([0, 0, 0, 1, 1, 2, 2, 2])
yp = F.scaled_dot_product_attention(x, x, x, attn_mask=document_mask(docs))
parts, s = [], 0
for L in (3, 2, 3):
    xs = x[..., s:s+L, :]
    parts.append(F.scaled_dot_product_attention(xs, xs, xs, is_causal=True)); s += L
print(torch.allclose(yp, torch.cat(parts, dim=2), atol=1e-6))       # True

# 4) padding: a padded row gives the same outputs as the short sequence alone
attn = torch.tensor([[1, 1, 1, 1, 1, 1, 0, 0]])
m = causal_mask(T)[None, None] & padding_mask(attn)
ypad = F.scaled_dot_product_attention(x, x, x, attn_mask=m)
xs = x[..., :6, :]
print(torch.allclose(ypad[..., :6, :], F.scaled_dot_product_attention(xs, xs, xs, is_causal=True), atol=1e-6))  # True

# 5) a row with every key blocked: -inf gives NaN, finfo.min gives a uniform row
s0 = torch.zeros(2)
print(torch.softmax(s0 + torch.tensor([float("-inf")] * 2), -1))     # tensor([nan, nan])
print(torch.softmax(s0 + torch.finfo(torch.float32).min, -1))       # tensor([0.5000, 0.5000])

# 6) loss mask, formula (18.7): no loss on padding, and (prefix-LM) none on the prefix
ids = torch.tensor([[5, 9, 2, 7, 3, 4, 0, 0]])
labels = ids.clone()
labels[attn == 0] = -100                  # cross_entropy ignores -100
labels[:, :3] = -100                      # no loss on the 3 prefix tokens
logits = torch.randn(1, T, 10)
loss = F.cross_entropy(logits[:, :-1].reshape(-1, 10), labels[:, 1:].reshape(-1))
print(int((labels[:, 1:] != -100).sum()))                           # 3 targets count

# 7) left padding: position ids from the attention mask
left = torch.tensor([[0, 0, 1, 1, 1], [1, 1, 1, 1, 1]])
pos = (left.cumsum(-1) - 1).clamp(min=0)
print(pos.tolist())                       # [[0, 0, 0, 1, 2], [0, 1, 2, 3, 4]]

# 8) is_causal is top-left aligned: wrong for 1 new query against a KV cache
q, k = torch.randn(1, 1, 1, d), torch.randn(1, 1, 5, d)
a = F.scaled_dot_product_attention(q, k, k, is_causal=True)        # sees key 1 only
b = F.scaled_dot_product_attention(q, k, k)                        # sees all 5 keys
print(torch.allclose(a, k[..., :1, :]), torch.allclose(a, b))      # True False

Common mistakes

  • No causal mask in a decoder. The training loss looks wonderful (the model copies the next token from its input), and generation is garbage. Always run the leak test of Figure 3 on new attention code.
  • Flipped boolean convention. nn.MultiheadAttention uses True = blocked; SDPA uses True = allowed; Hugging Face uses 1 = real token. Passing one where another is expected makes the model attend to exactly the wrong tokens, and it still trains.
  • Masking padding in attention but not in the loss (or the reverse). The attention mask stops tokens from reading pads; the loss mask (−100 labels) stops the model from learning to predict pads. You need both.
  • Right padding for batched generation. New tokens are appended after the pads, and short rows continue from padding. Use left padding (with correct position IDs) or a server that handles variable lengths.
  • Packing without a document mask or without resetting positions. Samples read each other (cross-contamination), and position IDs keep counting across samples, so a short sample at the end of a row is seen at positions it would never have alone.
  • Rows where every key is blocked. \(-\infty\) everywhere gives NaN; a finite “minimum” value gives a silent uniform average of forbidden tokens. This happens with left padding plus a causal mask (the pad query at position 1 can only see itself, which is padding) and with empty samples. Make sure every real query sees at least itself; ignore pad rows in the loss.
  • Using is_causal=True with a KV cache. In PyTorch SDPA it is top-left aligned: one new query would see only the first cached key. Use no mask for a single new token, or a bottom-right causal mask for several.
  • Confusing BERT’s [MASK] with an attention mask. Masked language modeling replaces input tokens; BERT’s attention stays fully bidirectional (except for padding).

Interview questions

What does an attention mask do, mathematically?basic
It adds a matrix \(M\) of 0 and \(-\infty\) to the scaled scores before the softmax (18.1). Because \(e^{-\infty} = 0\), blocked keys get weight exactly zero and the softmax renormalizes over the allowed keys only (18.2). Blocked pairs also get zero gradient. In code the mask is often boolean and converted to this additive form by the library.
Why does a decoder-only LM need a causal mask during training but the problem does not appear at inference?basic
In training we feed the whole text and predict \(x_{t+1}\) at every position \(t\) in one pass. Without the mask, position \(t\) could attend to \(x_{t+1}\) and simply copy it; the loss would collapse and nothing useful would be learned. At inference, future tokens do not exist yet, so the model would face inputs it never saw in training. The causal mask (18.4) makes training match inference: each position sees only the past, exactly as in generation.
How does the causal mask make the KV cache possible?core
With a causal mask, the hidden state of token \(i\) in every layer depends only on tokens \(1, \dots, i\). Appending new tokens does not change it, so its keys and values can be computed once and stored. Each new token only computes its own query, key and value and attends to the cache. In a bidirectional model every new token would change all earlier states, so the past would have to be recomputed at every step.
Explain padding mask versus loss mask. Do you need both?core
The padding (attention) mask blocks pad tokens as keys, so real tokens never read them and outputs do not depend on how much padding the batch has (18.6). The loss mask removes positions from the loss (18.7), done with label −100 in PyTorch: pads, and in fine-tuning also prompt tokens. You need both: attention mask without loss mask trains the model to predict pads; loss mask without attention mask lets real outputs depend on padding. Pad queries are still computed, but nobody reads them.
Left or right padding for batched generation? What else must change?core
Left padding for decoder-only generation: then the last position of every row is a real token, and each step appends one token to all rows in lockstep. With right padding, short rows would continue after their pads. The position IDs must be computed from the attention mask (cumulative sum minus 1), so the first real token is position 0. The attention mask must also block the pad keys at every decoding step. Right padding is fine for training.
Compare bidirectional, causal and prefix-LM attention. When would you use each?core
Bidirectional: every token sees all tokens; best representations for understanding (BERT, ViT, retrieval encoders), but no left-to-right generation and no KV cache reuse. Causal: each token sees itself and the past; needed for generation and gives \(T\) training targets per pass (GPT, Llama). Prefix-LM: the prefix (input, prompt, image) sees itself fully, the target is causal and sees the whole prefix (18.9); good for conditional generation with one stack of weights (UniLM, T5’s prefix-LM variant, PaliGemma). The loss goes only on the target. For \(T = 8, P = 3\) it allows 39 pairs versus 36 for causal.
What is sequence packing, and what goes wrong without a document mask?core
Packing concatenates several samples into one row of length \(T\) to avoid padding (Krell et al. report up to 50% padding in common NLP datasets, and a 2× speed-up for BERT pretraining with packing). With a plain causal mask, tokens of sample 2 attend to sample 1: cross-contamination. The model conditions on unrelated text, and in fine-tuning it can see other samples’ answers. Fix: a block-diagonal document mask (18.11), position IDs reset per sample, and no loss on the first token of a sample predicted from the previous one. Then a packed row is exactly equal to separate rows.
Your packed training run gets NaN loss after you added a document mask with left padding. What is a likely cause?advanced
A query row where every key is blocked. For example, a pad token at the start of a row with its own document ID 0 and attention bit 0: causal leaves only itself, the padding mask blocks itself, so the row is all \(-\infty\), and softmax gives \(0/0\) = NaN, which then spreads through the backward pass. Fixes: let pad queries see at least one key (for example themselves) since their outputs are ignored anyway, use a backend that returns 0 for empty rows, or drop pads with a varlen kernel. Do not just switch to a finite minimum value without thinking: that turns the bug into a silent uniform average.
How do fast attention kernels exploit masks? Why does tile size matter?advanced
Kernels like FlashAttention process the score matrix in tiles. A tile can be full (no masking needed), partial (mask applied inside) or empty (skipped: no loads, no math). Causal masks skip about half the tiles; document masks skip many more. FlexAttention builds this block table from a mask function and generates a kernel that skips empty tiles. A partial tile costs as much as a full one, so large tiles waste work: in Figure 5, for four documents in 1,024 tokens with 14.5% allowed pairs, the kernel does 17.6% of dense work with 64-token tiles but 37.5% with 256-token tiles.
In PyTorch, what is the difference between True in nn.MultiheadAttention’s key_padding_mask and True in scaled_dot_product_attention’s attn_mask?advanced
Opposite meanings. In nn.MultiheadAttention, True means “ignore this key”. In F.scaled_dot_product_attention, a boolean True means “this pair takes part in attention”. Hugging Face tokenizers return 1 for real tokens. Mixing them up makes the model attend to exactly the wrong set; it can still train, so always test with a leak test and a padding-invariance test (outputs for real tokens must not change when you add padding).
How are masks used in a VLM like PaliGemma, and for streaming video?advanced
PaliGemma uses a prefix-LM mask: image tokens plus the text prompt form a fully visible prefix, and the answer is generated causally while seeing the whole prefix. Image tokens get label −100 (no loss). When samples are packed, each needs its own document ID so images do not leak between samples. For streaming video, a block-causal mask by frame (full inside a frame, causal across frames) keeps the KV cache property, so new frames can be added without recomputing old ones.

Exercises

  1. A query has scaled scores \((1.0,\ 3.0,\ 0.0)\) for keys 1 to 3 and the mask row \((0,\ -\infty,\ 0)\). Compute the attention weights.
  2. How many (query, key) pairs does a causal mask allow for \(T = 4096\)? What fraction of \(T^2\) is that?
  3. For \(T = 10\) and a prefix of \(P = 4\), how many pairs does the prefix-LM mask allow? How many more than causal?
  4. A batch has samples of 100, 400, 50 and 250 tokens, padded to the longest. What fraction of the token slots is padding?
  5. A packed row of \(T = 12\) has documents of 5, 4 and 3 tokens. Write the document ID of each position, the reset position IDs, and count the allowed pairs under the document mask.
  6. In Figure 3, Causal + padding mode, why does editing a pad token change its own output but no other output? Would that still hold without the padding mask?
  7. Write the boolean mask (True = may attend, SDPA convention) for a 2-row batch with attention bits [1,1,1,0] and [1,1,1,1] and a causal mask. Which rows of row 1 are “fully padding”?
Answerscheck
1. Key 2 is blocked. Softmax over keys 1 and 3: \(e^{1}/(e^{1} + e^{0}) = 2.718/3.718 = 0.731\) and \(0.269\). Weights: \((0.731,\ 0,\ 0.269)\). 2. \(4096 \cdot 4097/2 = 8{,}390{,}656\) pairs, out of \(4096^2 = 16{,}777{,}216\): 50.01%. 3. \(16 + (110 - 20)/2 = 16 + 45 = 61\) pairs; causal gives 55, so 6 more (the pairs inside the prefix that look to the right: \(P(P-1)/2 = 6\)). 4. Real tokens: 800; slots \(4 \times 400 = 1600\); padding fraction 50%. 5. Doc IDs: 0 0 0 0 0 1 1 1 1 2 2 2. Positions: 0 1 2 3 4 0 1 2 3 0 1 2. Allowed pairs: \(15 + 10 + 6 = 31\), versus 78 for plain causal. 6. Its column is blocked, so no other query reads its key or value; only its own query changes, and its own row is ignored by the loss. Without the padding mask, every later position would read the pad token, so their outputs would change too. 7. Row 1: the causal lower triangle with column 4 set to False: [[T,F,F,F],[T,T,F,F],[T,T,T,F],[T,T,T,F]]. Row 2: the plain lower triangle. No row of row 1 is fully blocked; the pad query at position 4 still sees keys 1 to 3 (its output is ignored). With left padding instead, the first query would see only itself, a pad, and its row would be fully blocked.

Key takeaways

  • A mask adds 0 or \(-\infty\) to each attention score before the softmax (18.1). Blocked keys get weight and gradient exactly zero; the softmax renormalizes over the allowed keys (18.2).
  • The causal mask (18.4) lets a model train on all \(T\) positions at once without seeing the answers, and it is what makes the KV cache possible. It allows \(T(T+1)/2\) pairs (18.5).
  • The padding mask blocks pad keys (18.6); the loss mask (labels −100) removes positions from the loss (18.7). They are different masks and you need both. Use left padding for batched generation, with position IDs from the mask.
  • Bidirectional (BERT) sees everything, causal (GPT) sees the past, prefix-LM (T5 variant, UniLM, PaliGemma) is bidirectional on the prefix and causal on the target (18.9).
  • Packing removes padding; a block-diagonal document mask (18.11) and reset position IDs make a packed row equal to separate rows and stop cross-contamination.
  • In code, check what True means, avoid fully blocked rows, and prefer is_causal, varlen kernels or block masks, which skip blocked tiles instead of computing and discarding them.

Key formulas

\[ \operatorname{Attn} = \softmax\!\Big(\frac{QK^\top}{\sqrt{d_k}} + M\Big) V, \qquad M^{\text{causal}}_{ij} = 0 \text{ iff } j \le i \]
\[ M^{\text{prefix}}_{ij} = 0 \text{ iff } (j \lt P \text{ or } j \le i), \qquad M^{\text{doc}}_{ij} = 0 \text{ iff } (\mathrm{doc}(i) = \mathrm{doc}(j) \text{ and } j \le i) \]
\[ N_{\text{causal}} = \frac{T(T+1)}{2}, \qquad N_{\text{doc}} = \sum_k \frac{L_k(L_k+1)}{2}, \qquad \mathcal{L} = \frac{\sum_t w_t \ell_t}{\sum_t w_t} \]

References for this session

  1. Raschka, Build a Large Language Model (From Scratch), chapter 3: Coding attention mechanisms
  2. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  3. Raschka, Build a Large Language Model (From Scratch), chapter 3: Coding attention mechanisms
  4. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  5. Vaswani et al. (2017). Attention Is All You Need
  6. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  7. Dong, Yang, Wang, Wei, Liu, Wang, Gao, Zhou, Hon (2019). Unified Language Model Pre-training for Natural Language Understanding and Generation
  8. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
  9. Krell, Kosec, Perez, Fitzgibbon (2021). Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance
  10. Zhao, Qu, Staniszewski, Tworkowski, Liu, Miłoś, Wu, Minervini (2024). Analysing The Impact of Sequence Composition on Language Model Pre-Training
  11. Llama Team, Meta (2024). The Llama 3 Herd of Models
  12. Dong, Feng, Guessous, Liang, He (2024). Flex Attention: A Programming Model for Generating Optimized Attention Kernels
  13. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  14. Jiang et al. (2023). Mistral 7B