LLM & VLM courseBlog
188Track 7: Vision and VLMs · session 26 of 38

Batching and masking for multimodal sequences

A VLM batch mixes text-only chats, single images, tiled documents and videos, so one sample may have 0 visual tokens and the next 3,000. This session shows how such samples become tensors that train fast and correctly: padding and length grouping, packing with document masks and restarted positions (in the language model and, as in NaViT, in the vision encoder), a loss only on the text answer with the right averaging, and balancing tokens and images across GPUs.

Time: 2.5 to 3.5 hours (reading 1.5 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 18 (masks), 54 (SFT in detail) and 177 (anatomy of a VLM); session 44 (data parallelism) helps

What you will learn

  • Why the number of image tokens per sample varies so much (fixed 576, AnyRes tiles, native resolution, video), and how placeholders and masked_scatter put image features into the sequence
  • Padding efficiency, and how length grouping and LLaVA’s modality-grouped sampler remove most padding
  • Packing with a block-diagonal document mask, restarted position ids and cu_seqlens, and how much attention work it saves
  • NaViT: packing images of different sizes into one vision-encoder sequence with masked attention and masked pooling
  • Loss only on answer tokens (labels −100 elsewhere), the shift inside each sample, and token, sample and square averaging (InternVL 2.5)
  • The global-count rule for gradient accumulation and data parallelism
  • Balancing tokens and vision-encoder work across GPUs with a simple cost model

Why this matters

Take four training samples for a vision-language model: a text-only chat of 300 tokens, a LLaVA-1.5 image with a short question (696 tokens), a 1080p screenshot in Qwen2-VL (2,751 tokens) and a document cut into LLaVA-NeXT tiles (2,960 tokens). Put them in one batch the simple way, by padding every row to the longest one, and 43% of the tensor is padding. The GPU does that work and throws it away.

In a text model, sample lengths already vary. In a VLM they vary much more, because the number of visual tokens (the vectors that stand for the image inside the language model) depends on the model design, the image size, the number of images and the number of video frames. One sample can have 0 visual tokens and the next 3,000. This session is about the engineering that turns such a mixed set of samples into tensors that train fast and train correctly:

  • how a multimodal sample becomes one sequence, with a different number of image tokens per sample;
  • how to pack several samples into one row, with attention masks so that samples do not see each other, and the same idea inside the vision encoder (NaViT);
  • how to put the loss only on the text answer, never on image tokens or prompts, and how to average it so that short and long answers get the weight you want;
  • how to get high throughput: sort by length, and balance tokens (and images) per GPU.

We built the masks themselves in session 18 and the text-only version of packing and loss averaging in session 54. Here we add what images change. The book covers batching and the real batch size in VLM training (Noyan et al., chapter 3).

These are not toy problems. NaViT packs images of different sizes into one sequence of the vision encoder [Certain]. Qwen2-VL’s vision encoder runs all images of a batch as one packed sequence with segment boundaries (the cu_seqlens in the Hugging Face code) [Certain, from the transformers code]. InternVL 2.5 packs samples under a limit on both the context length and the number of image tiles, and lets tokens attend only within their own sample [Certain, from the paper]. LLaVA-1.5 trains with --group_by_modality_length True, a sampler that groups samples by modality and length [Certain, from its training script].

One sample, one sequence: a different number of image tokens

A VLM in the unified-sequence style (session 177, session 179) reads one sequence of embeddings. The text goes through the token embedding table. Each image goes through the vision encoder and the connector, which return a matrix of \(v\) vectors. These vectors are placed into the sequence where the image belongs.

The usual way to do this is with a placeholder token. The chat template writes a special token such as <image> where the image goes. Before the model runs, the processor repeats it: one placeholder per visual token, so the input ids already have the final length. Inside the model, the embedding layer produces a vector for every id, and then the rows at the placeholder positions are overwritten with the image features (in PyTorch, masked_scatter). Qwen2-VL does exactly this, and checks first that the number of placeholders equals the number of feature vectors [Certain, from the transformers code]. LLaVA’s original code uses a negative id, IMAGE_TOKEN_INDEX = -200, as the placeholder [Certain, from its constants.py].

For sample \(i\) with images \(j = 1, \dots, J_i\), the length of the sequence is

\[ n_i = V_i + T_i, \qquad V_i = \sum_{j=1}^{J_i} v_{ij} \tag{188.1} \]
Read the symbols
\(n_i\)
the number of positions sample \(i\) takes in the language model
\(T_i\)
the number of text tokens: system prompt, user turns, assistant answers and special tokens
\(v_{ij}\)
the visual tokens of image (or frame) \(j\): 576 for every LLaVA-1.5 image; \((H'/28)(W'/28)\) for Qwen2-VL, which depends on the image size; up to \(5 \times 576 = 2{,}880\) for LLaVA-NeXT tiles
\(J_i\)
the number of images in the sample: 0 for text-only data, several for multi-image or video samples

So \(V_i\) is a random variable with a wide range, and in a typical sample it is far larger than \(T_i\). In the four samples above, 91.7% of the real tokens are visual. This is why VLM batching is harder than text batching: the length is decided mostly by the image, and the image size is decided by the data, not by you.

Two practical rules follow from (188.1). First, you must know \(n_i\) before you batch, so the processor (resize, tile, count tokens) runs in the data loader, not in the model. Second, a limit on the sequence length must be checked on \(n_i\), not on the text length: cutting a sample in the middle of its image tokens breaks the placeholder count and crashes the scatter, and cutting the answer trains on a fragment.

Figure 1. The life of a packed multimodal batch. Top: the images of all samples run through the vision encoder as one sequence; patches attend only within their own image (brackets). Middle: the connector output is scattered into the placeholder positions of one language-model row that holds three samples; a document mask keeps the samples apart. Bottom: the labels; only answer tokens (green) carry a loss, everything else is −100.

Padding, and why sorting helps

The simple way to make a batch is to pad: give each sample its own row and fill the short rows with a pad token up to the longest row in the batch. The padding mask from session 18 makes the pad positions invisible, and the loss ignores them. The result is correct, but the GPU still computes every pad position. The padding efficiency of a batch is the share of real tokens:

\[ \eta = \frac{\sum_{i=1}^{B} n_i}{B \cdot \max_i n_i} \tag{188.2} \]
Read the symbols
\(B\)
the number of samples (rows) in the batch
\(\max_i n_i\)
the longest sample, which sets the width of the tensor
\(\eta\)
1 means no waste; \(1 - \eta\) is the share of compute spent on padding (for the linear layers; for attention the waste is larger, because the cost grows with the square of the width)
Worked example: four mixed samples

Lengths 300 (text only), 696 (576 + 120), 2,751 (2,691 + 60) and 2,960 (2,880 + 80): 6,707 real tokens. Padded to 2,960, the tensor has \(4 \times 2{,}960 = 11{,}840\) positions, so \(\eta = 6{,}707 / 11{,}840 = 56.6\%\): 5,133 positions are padding.

Attention is worse. Each padded row computes a \(2{,}960 \times 2{,}960\) score matrix, \(4 \times 2{,}960^2 = 35.0\) million pairs. The real samples need only \(\sum_i n_i^2 = 16.9\) million, 48.2% of that.

Now pack them instead (next section) into rows of 4,096 with first-fit decreasing (sort long to short, put each sample into the first row with room): row 1 holds 2,960 + 696 + 300 = 3,956 tokens, row 2 holds 2,751. Efficiency \(6{,}707 / 8{,}192 = 81.9\%\), and with a document mask the attention work is exactly the 16.9 million pairs the samples need.

The cheapest fix is to put samples of similar length in the same batch. Length grouping sorts samples by \(n_i\) (usually inside a large random chunk, so that the order of the data stays random) and cuts the sorted list into batches. LLaVA’s sampler does this: it shuffles, cuts the data into “megabatches” of (number of GPUs × batch size) samples, sorts each megabatch from long to short, and splits it over the GPUs [Certain, from llava_trainer.py]. With group_by_modality_length it also keeps image-text samples and text-only samples in separate megabatches, so text-only batches are not padded to image lengths [Certain, from the same file].

Sorting has a cost. Batches of similar length are less random: all the long document samples arrive together, and the gradient of that step sees only one kind of data. Grouping inside megabatches keeps most of the randomness and most of the gain.

visual tokenstext tokenspadding
Figure 2. The same 24 samples (a fixed mix of text-only chats, LLaVA-1.5 images, AnyRes tiles, Qwen2-VL native-resolution images and short videos; 28,333 real tokens) batched three ways. “Attention work” counts score pairs \(n^2\) per row segment and divides by the minimum \(\sum_i n_i^2\).
How to read Figure 2

Each bar is one row of the tensor that goes to the GPU; gaps separate batches. With random order and \(B = 4\), 46.2% of the positions are padding and attention does 2.59× the necessary work. Sort by length: padding falls to 16.1% and attention to 1.38×. Raise \(B\) to 8: padding grows again in both modes, because a larger batch is more likely to contain one very long sample. Packing does not depend on \(B\): the only waste is the empty end of each row (13.5% here), and with a document mask attention does exactly the necessary work. The row-length slider switches between rows of 4,096 and 8,192 tokens.

Packing, with masks so samples do not see each other

Packing puts several whole samples one after the other into one row. A row of 8,192 positions might hold one video sample, two images and a text chat. There is almost no padding left. But now there is a new danger: in one row, the causal mask lets a token of sample 3 attend to all of sample 1 and sample 2. The model would learn to use context that does not exist at test time, and the loss of later samples would look better than it should.

The document mask

The fix is a block-diagonal (document) mask: a query may only see keys of its own sample. Give each position \(t\) a sample id \(s(t)\) and an image flag. The mask is

\[ M_{tu} = \begin{cases} 0 & s(t) = s(u) \ \text{ and } \ \big(u \le t \ \text{ or } \ t, u \in \text{same image}\big) \\ -\infty & \text{otherwise} \end{cases} \tag{188.3} \]
Read the symbols
\(t, u\)
the query position and the key position in the packed row
\(s(t)\)
which sample position \(t\) belongs to
\(u \le t\)
the usual causal rule inside a sample
same image
an optional exception: visual tokens of one image see each other in both directions. PaliGemma goes further and lets the image and the text prefix attend fully (a prefix-LM, session 18) [Certain]; the Hugging Face Gemma 3 code makes image tokens bidirectional [Likely]. Most LLaVA-style models keep the plain causal rule for image tokens too.

With the mask, a packed row computes the same outputs as separate rows, only faster. The code at the end of this session checks this: the difference is about \(7 \times 10^{-7}\), float rounding.

Positions restart in every sample

The mask is not enough. Position ids must also restart at 0 for every sample, because the model was trained to see the image near position 0, not at position 5,000. If \(c_k\) is the start of sample \(k\) in the row,

\[ c_0 = 0, \quad c_k = \sum_{i=1}^{k} n_i, \qquad p_t = t - c_{s(t)} \tag{188.4} \]
Read the symbols
\(c_k\)
the cumulative lengths; the vector \((c_0, c_1, \dots, c_K)\) is called cu_seqlens in FlashAttention and Hugging Face code
\(p_t\)
the position id of token \(t\) inside its own sample, used by RoPE or by a position table

For lengths 13, 4 and 13, cu_seqlens is \((0, 13, 17, 30)\). In Qwen2-VL, positions are three numbers per token (time, height, width, called M-RoPE), and the text after an image continues from the largest image position plus one (session 180). The rule is the same: compute these ids per sample, then concatenate. In the code below, a packed row without restarted positions differs from the separate rows by 3.37 in the logits: a different model.

Kernels: you rarely build the mask

A dense \(N \times N\) boolean mask costs memory, and most of it is \(-\infty\). Fast kernels take cu_seqlens instead. FlashAttention’s variable-length function (flash_attn_varlen_func) runs attention on each segment separately inside one launch (session 124). Hugging Face’s DataCollatorWithFlattening concatenates samples into one row and passes position_ids so FlashAttention-2 can find the boundaries; its authors report up to 2× training throughput on FLAN and 1.4× on OrcaMath, with up to 20% lower peak memory [Certain, from the Hugging Face blog, August 2024]. PyTorch FlexAttention can build the block-diagonal rule from a small Python function of the sample ids. The cost of attention with a document mask is

\[ \frac{\text{pairs with a document mask}}{\text{pairs without}} = \frac{\sum_{k} n_k^2}{\big(\sum_{k} n_k\big)^2} \tag{188.5} \]

For the four samples of the worked example in one row of 6,707, this ratio is 37.6%: a kernel that skips the masked blocks does less than half the attention work of a plain causal row of the same length. A dense kernel with a mask computes everything and then throws most of it away.

Algorithm 1: multimodal collate with packing
  1. input: samples with images and chat turns; row length \(L_{\max}\); tile or image limit
  2. for each sample: run the image processor, count \(v_{ij}\), write \(v_{ij}\) placeholders, compute \(n_i\) (188.1) // in the data loader
  3. build labels: answer tokens keep their ids, all else \(-100\); shift by one inside the sample
  4. drop or split samples with \(n_i \gt L_{\max}\) at a turn boundary, never inside an image
  5. first-fit decreasing: sort by \(n_i\), put each sample into the first row with room (also checking the image limit)
  6. for each row: concatenate ids and labels, compute cu_seqlens and restarted positions (188.4), sample ids, image flags
  7. concatenate the pixel tensors of all images in the row, with their own cu_seqlens for the vision encoder
  8. return ids, labels, positions, cu_seqlens, pixels, image grid sizes

Figure 3 runs Algorithm 1 on five tiny interleaved samples, one line at a time, so you can watch the three tensors (ids, positions, labels) being written. Sample A has text, then an image, then more text; sample D has two images between text turns; B and E are text-only. Each image becomes \(v\) placeholder tokens. Switch between padding and packing and compare the waste.

image placeholderprompt tokenanswer tokenpaddingsample boundary
Algorithm 1 in the toy
    Figure 3. Algorithm 1 computed live on five samples (A: 2 prompt tokens, an image, 1 prompt token, 3 answer tokens; B: 3 prompt, 2 answer; C: an image, 1 prompt, 2 answer; D: prompt, image, prompt, image, prompt, 1 answer; E: 2 prompt, 4 answer). Top: the queue, in data order for padding and sorted by \(n_i\) for packing. Middle: the current sample before and after its <image> placeholders are repeated \(v\) times. Bottom: the batch tensor. In each row, the first line is the ids (letter = sample), the second the position ids, the third the labels (✓ = this position has a loss, × = −100).
    How to read Figure 3

    Each sample takes three steps: expand the images and count \(n_i\) with (188.1), build its labels with the shift inside the sample, and place it. With \(v = 3\) the lengths are A 9, B 5, C 6, D 10 and E 6, 36 real tokens. In pad mode every sample gets its own row and the last step pads all rows to \(W = 10\): 50 positions, \(\eta = 72.0\%\) (188.2), and 500 attention pairs where the samples need \(\sum_i n_i^2 = 278\). In pack mode the queue is D, A, C, E, B. Watch the first-fit checks in the formula panel: A fits after D (10 + 9 = 19 ≤ 20), C does not (25 \(\gt\) 20) and opens row 2, then E and B go to row 2. Two rows, 40 positions, \(\eta = 90.0\%\), cu_seqlens \((0, 10, 19)\) and \((0, 6, 12, 17)\), and the positions restart at 0 after every red wall (188.4). With a variable-length kernel the attention work is exactly the 278 pairs. Try \(v = 4\) with \(L = 20\): packing needs 3 rows and its efficiency falls to 66.7%, the same as padding. With only five samples the empty end of each row is a large share; with thousands of samples per step it becomes small (Figure 2). In both modes 12 positions carry a loss.

    The same idea inside the vision encoder: NaViT

    The vision encoder has its own batching problem. A standard ViT resizes every image to one size, say 224 × 224, so that every image gives the same number of patches. NaViT (“Patch n’ Pack”, Dehghani et al., 2023) keeps each image at its own resolution and aspect ratio, and packs the patch sequences of several images into one sequence. Three changes make this work [Certain, from the paper]:

    • masked self-attention, so that patches of one image do not attend to another image (the vision version of (188.3), without the causal part);
    • masked pooling, so that each image gets its own pooled vector at the end;
    • factorized position embeddings (separate x and y embeddings, added), so that any grid size works.

    NaViT packs greedily, adding each image to the first sequence with enough space, and reports that typically less than 2% of the tokens are padding [Certain]. Packing also makes token dropping easy (randomly removing a share of the patches of each image during training), and NaViT reports processing about five times more images in training at the same compute, which it names as the main source of its gains [Certain]. In VLMs, the Qwen2-VL family uses the same approach in its vision encoder: one long sequence of patches for all images in the batch, with attention only inside each segment.

    So a multimodal batch usually has two packed sequences: one for the vision encoder (patches, segmented by image) and one for the language model (tokens, segmented by sample). They have different lengths and different segment boundaries, and both must be balanced (last section).

    image tokenprompt tokenanswer token (has a loss)allowed, same sampleallowed, but from another sample
    Building the mask (188.3) and the loss (188.6)
      Figure 4. One packed row of 22 tokens: sample A (4 image, 2 prompt, 3 answer tokens), sample B (text only: 2 prompt, 2 answer) and sample C (6 image, 1 prompt, 2 answer). The animation builds the mask one query row at a time with (188.3): row \(t\) of the grid shows the keys that query \(t\) may attend to. Then it builds the label row with the shift inside each sample and accumulates the loss (188.6) from toy probabilities \(p_\theta\) of the correct next token (0.60, 0.35, 0.80, 0.50, 0.90, 0.40, 0.70 for the seven answer targets, in order); “×” means −100 (no loss).
      How to read Figure 4

      Press Play. Steps 1 to 22 write the 22 rows of the mask; the formula panel shows \(s(t)\), which keys pass the sample check and which pass the causal rule, and the running count of allowed pairs. Steps 23 to 44 write the labels: \(m_t = 1\) only if token \(t+1\) is in the same sample and is an answer token. Click any grid cell or token to jump to that query row, or a label cell to jump to that label. With “causal only”, the 22 tokens have 253 allowed pairs and 153 of them cross a sample boundary (red): sample C’s answer can read sample A’s image. The document mask leaves 100 pairs, all inside a sample. Adding bidirectional attention inside each image adds 21 pairs (6 for A’s 4-token image, 15 for C’s 6-token image). In the label phase only 7 positions carry a loss, one per answer token, and the last token of each sample has no target, so no sample learns to predict the start of the next one. The loss ends at \(\mathcal{L} = 3.855 / 7 = 0.551\); the mask mode does not change the labels.

      Loss only on the text answer

      A VLM is trained with next-token prediction, but only on the tokens it should produce at test time (session 186). Image tokens are not predicted at all: they are continuous vectors from the encoder, there is no vocabulary entry to predict, and the output head would be asked for a placeholder id that means nothing. The system prompt and the user question are also inputs, not outputs. So the labels are \(-100\) (the ignore_index of PyTorch’s cross-entropy; LLaVA calls it IGNORE_INDEX [Certain]) on every image, prompt and pad position, and the loss is

      \[ \mathcal{L} = \frac{\sum_{t} m_t\, \ell_t}{\sum_{t} m_t}, \qquad \ell_t = -\log p_\theta\big(y_{t+1} \mid y_{\le t}, \text{image}\big), \quad m_t = \mathbb{1}\big[y_{t+1} \in \text{answer}\big] \tag{188.6} \]
      Read the symbols
      \(\ell_t\)
      the cross-entropy at position \(t\): the negative log probability of the next token
      \(m_t\)
      1 when the next token is part of an assistant answer (including its end-of-turn token), 0 otherwise
      \(\sum_t m_t\)
      the number of answer tokens that count; the denominator decides how examples are weighted (below)

      Three details cause most bugs. (1) The shift: position \(t\) predicts token \(t+1\). In a packed row, shift inside each sample, so the last token of sample A does not get the first token of sample B as its target. (2) Multi-turn chats have several assistant turns; each gets \(m = 1\), and each user turn between them gets \(m = 0\). (3) The end-of-turn token must be inside the answer, or the model never learns to stop. Print one sample with its labels before training; this is the single cheapest check.

      How to average: token mean, sample mean, square mean

      Formula (188.6) is a token mean: every answer token in the batch has the same weight. A sample with a 400-token document transcription then has 100 times the weight of a sample whose answer is “Yes.”. The sample mean averages inside each sample first, so every sample has the same weight. InternVL 2.5 writes both as one family [Certain, from the paper]:

      \[ \mathcal{L} = \frac{\sum_{s} w_s \sum_{t \in s} \ell_t}{\sum_{s} w_s\, x_s}, \qquad w_s = \frac{1}{x_s^{\,q}} \tag{188.7} \]
      Read the symbols
      \(x_s\)
      the number of answer tokens in sample \(s\)
      \(q\)
      \(q = 0\): token mean; \(q = 1\): sample mean; \(q = 0.5\): the “square averaging” that InternVL 2.5 proposes
      \(w_s x_s\)
      the total weight of sample \(s\), proportional to \(x_s^{1-q}\)

      The paper explains the choice: token averaging biases the gradient toward responses with more tokens, and sample averaging makes every sample count equally but can make the model favor shorter responses [Certain]. VLM data mixes one-word VQA answers with long captions and OCR outputs, so the range of \(x_s\) is huge and this choice matters more than in text SFT.

      Worked example: “Yes.” against a document

      Two samples: a VQA answer of \(x = 4\) tokens and a transcription of \(x = 400\). Total weight is proportional to \(x^{1-q}\):

      • token mean (\(q = 0\)): \(4 : 400\), so 1.0% and 99.0%;
      • square mean (\(q = 0.5\)): \(2 : 20\), so 9.1% and 90.9%;
      • sample mean (\(q = 1\)): 50% and 50%.

      Per token, the short answer’s tokens weigh 1×, 10× and 100× as much as the long answer’s tokens.

      The denominator across micro-batches and GPUs

      With gradient accumulation or data parallelism, the batch is split into micro-batches \(k = 1, \dots, K\) (on one GPU over time, or over many GPUs). If each micro-batch divides by its own token count and the results are averaged, the tokens of a micro-batch with few answer tokens get more weight. The correct token mean divides by the global count:

      \[ \mathcal{L} = \frac{\sum_{k=1}^{K} S_k}{\sum_{k=1}^{K} N_k} \ \ne\ \frac{1}{K} \sum_{k=1}^{K} \frac{S_k}{N_k} \tag{188.8} \]
      Read the symbols
      \(S_k\)
      the sum of the (weighted) token losses in micro-batch \(k\)
      \(N_k\)
      the number of loss tokens in micro-batch \(k\) (the weighted count \(\sum_s w_s x_s\) for 188.7)

      Example: micro-batch 1 has 10 answer tokens with mean loss 2.0, micro-batch 2 has 190 with mean loss 1.0. The right loss is \((20 + 190)/200 = 1.05\); the mean of means is \((2.0 + 1.0)/2 = 1.5\), and every token of micro-batch 1 got 10 times its fair weight. Hugging Face transformers fixed exactly this bug in gradient accumulation in October 2024: the loss is now summed and divided by the total number of items [Certain, from the Hugging Face blog]. With data parallelism, you get the global count with one all-reduce of \(N_k\) before the backward pass, and scale each GPU’s loss by \(K / \sum_k N_k\) (DDP averages gradients over the \(K\) GPUs, session 44). Packing makes this more important, not less: the number of samples and answer tokens per row now varies a lot.

      Figure 5. Five samples of one global batch, with answers of 1, 8, 30, 120 and 400 tokens (a yes/no answer, a short phrase, a caption, a detailed description, a document transcription). Bars: the share of the total loss weight each sample gets under formula (188.7). The first three samples are in micro-batch 1 and the last two in micro-batch 2.
      How to read Figure 5

      At \(q = 0\) (token mean) the transcription takes 71.6% of the weight and the yes/no answer 0.18%. Move \(q\) to 0.5: the shares become 2.5% to 49.7%. At \(q = 1\) every sample gets 20%. Now switch to the bug at \(q = 0\): each micro-batch gets half of the weight whatever its size, so the caption jumps from 5.4% to 38.5%, the same as the transcription. The bug is an accidental and unstable re-weighting: it changes with how the data falls into micro-batches.

      Throughput: sort by length, balance tokens per GPU

      In data-parallel training, every GPU must finish its micro-batch before the gradients are all-reduced. The step time is set by the slowest GPU. If one GPU gets two video samples and another gets four short chats, the second GPU waits. The cost of a sample is not just its length. A useful model is

      \[ c_i = \underbrace{n_i}_{\text{linear layers}} + \underbrace{\frac{n_i^2}{n_\star}}_{\text{attention}} + \underbrace{r\, V_i}_{\text{vision encoder}}, \qquad \text{step} = \max_g \sum_{i \in g} c_i, \qquad U = \frac{\frac{1}{G}\sum_g \sum_{i \in g} c_i}{\text{step}} \tag{188.9} \]
      Read the symbols
      \(c_i\)
      the cost of sample \(i\), in units of “one token through the language model’s linear layers”
      \(n_\star\)
      the length at which attention costs as much as the linear layers. With the layer cost of session 183, \(2n(4d^2 + 3dm) + 4n^2 d\), it is \(n_\star = (4d^2 + 3dm)/(2d) = 24{,}704\) for a 7B Llama shape (\(d = 4096\), \(m = 11008\))
      \(r\)
      the vision-encoder cost per visual token, relative to one language-model token. It depends on the encoder size and on the merge ratio (a 2 × 2 merge means 4 patches per visual token). Measure it on your system
      \(g\), \(G\)
      a GPU and the number of GPUs
      \(U\)
      the utilization: the average GPU’s work divided by the slowest GPU’s work; \(1 - U\) is the share of time GPUs sit idle waiting

      Three strategies, in order of effort:

      1. Equal count, random: every GPU gets the same number of random samples. Simple, and with heavy-tailed lengths, slow.
      2. Sort and balance tokens (LLaVA): sort a megabatch from long to short and give each sample to the GPU with the fewest tokens so far, keeping the count per GPU equal. This is the greedy longest-processing-time-first rule from scheduling.
      3. Balance the full cost: the same greedy rule on \(c_i\) instead of \(n_i\), and without the equal-count rule. GPUs may then get different numbers of samples, which is fine only if the loss uses the global token count (188.8).
      Algorithm 2: cost-balanced assignment of one global batch
      1. input: samples with \(n_i\), \(V_i\); \(G\) GPUs; cost model (188.9)
      2. compute \(c_i\) for every sample // lengths are known from the data loader
      3. sort samples by \(c_i\), largest first
      4. \(\text{load}_g \leftarrow 0\) for all \(g\)
      5. for each sample \(i\): \(g^* \leftarrow \argmin_g \text{load}_g\); give \(i\) to \(g^*\); \(\text{load}_{g^*} \leftarrow \text{load}_{g^*} + c_i\)
      6. on each GPU, pack its samples into rows (Algorithm 1)
      7. all-reduce the number of loss tokens; scale each local loss by \(G / \sum_k N_k\) // formula 188.8
      Worked example: two GPUs, eight samples

      Lengths 3,000, 2,900, 700, 650, 600, 500, 400 and 300 tokens. Split in order, four per GPU: 7,250 and 1,800 tokens. The step waits for 7,250, while the average is 4,525: \(U = 62.4\%\). Sort and fill the lightest GPU: GPU 1 gets 3,000, 650, 500 and 400 (4,550); GPU 2 gets 2,900, 700, 600 and 300 (4,500). \(U = 99.5\%\), and the step is 1.59× faster.

      The vision encoder adds a second axis. Balancing tokens does not balance images: a GPU with one 3,000-token video and a GPU with three 1,000-token text chats have the same token count, but only the first runs the vision encoder. When the encoder is large (InternVL uses a 6B-parameter InternViT in some versions, session 182) or runs at high resolution, \(r\) is large and the cost-aware rule wins. InternVL 2.5’s packing limits both the context length and the number of image tiles of each packed sample for this reason [Certain, from the paper, which does not give the numeric limits]. Another common fix is to run the vision encoder on a balanced split of all images of the step, then send the features to the GPUs that own the samples [Likely].

      language model, linearlanguage model, attentionvision encoder
      Figure 6. One global batch of 64 fixed samples (text-only, single images, tiles, native-resolution images and videos; 80,923 tokens) spread over \(G\) GPUs with formula (188.9), \(n_\star = 24{,}704\). The dashed line is the slowest GPU, which sets the step time. The cost units are “one token through the linear layers”.
      How to read Figure 6

      With 8 GPUs and random equal counts, utilization is 73.4%: about a quarter of the GPU time is waiting. Sorting and balancing tokens brings it to 98.5% and makes the step 1.34× faster. Raise \(r\) to 1.0 (a large vision encoder): the token-balanced split falls to 95.7%, because it ignores where the images are, while balancing the full cost keeps 99.3% (some GPUs now get 7 samples and others 9). Choose 16 GPUs: with only 4 samples per GPU, random assignment drops to 47.2% utilization, and balancing makes the step about 2× faster.

      Code: packing, masks and the loss in PyTorch

      This file builds three samples with 6, 0 and 9 image tokens, packs them into one row with restarted positions and a document mask, scatters fake image features into the placeholders, and checks that the packed row gives the same logits as three separate rows. Then it checks the masked loss, the global-count rule (188.8), and the greedy GPU balance. It runs on a CPU in under a second and is deterministic.

      import torch, torch.nn as nn, torch.nn.functional as F
      torch.manual_seed(0)
      IMG, IGNORE, VOCAB, D = 0, -100, 100, 32       # id 0 is the image placeholder
      
      # ---------- 1) three samples with different numbers of image tokens ----------
      samples = [dict(n_img=6, prompt=[5, 6, 7], answer=[20, 21, 22, 2]),   # 2 = end of turn
                 dict(n_img=0, prompt=[8, 9],    answer=[23, 2]),           # a text-only sample
                 dict(n_img=9, prompt=[10],      answer=[24, 25, 2])]
      for s in samples:
          s["feats"] = torch.randn(s["n_img"], D)                           # output of the vision encoder + connector
      
      def build(s):
          """input ids with one placeholder per visual token; targets already shifted by one."""
          ids = [IMG] * s["n_img"] + s["prompt"] + s["answer"]
          lab = [IGNORE] * (s["n_img"] + len(s["prompt"])) + s["answer"]   # loss only on the answer
          tgt = lab[1:] + [IGNORE]                                           # position t predicts token t+1
          return torch.tensor(ids), torch.tensor(tgt)
      
      # ---------- 2) pack: concatenate, cu_seqlens, restarted positions, sample ids ----------
      parts = [build(s) for s in samples]
      ids = torch.cat([p[0] for p in parts]); tgt = torch.cat([p[1] for p in parts])
      lens = torch.tensor([len(p[0]) for p in parts])
      cu_seqlens = F.pad(lens.cumsum(0), (1, 0))
      seq_id = torch.repeat_interleave(torch.arange(len(lens)), lens)
      pos = torch.arange(len(ids)) - cu_seqlens[seq_id]                      # 0,1,2,... inside each sample
      print(lens.tolist(), cu_seqlens.tolist(), int((tgt != IGNORE).sum()))  # [13, 4, 13] [0, 13, 17, 30] 9
      
      # ---------- 3) a tiny model: embeddings, image scatter, one masked attention layer ----------
      class Tiny(nn.Module):
          def __init__(self):
              super().__init__()
              self.tok, self.pos = nn.Embedding(VOCAB, D), nn.Embedding(64, D)
              self.qkv, self.out = nn.Linear(D, 3 * D), nn.Linear(D, VOCAB)
          def forward(self, ids, feats, pos, mask):
              h = self.tok(ids)
              img = (ids == IMG).unsqueeze(-1)
              assert int(img.sum()) == feats.shape[0]                        # one feature per placeholder
              h = h.masked_scatter(img, feats) + self.pos(pos)               # put visual tokens in place
              q, k, v = self.qkv(h).chunk(3, -1)
              a = F.scaled_dot_product_attention(q[None], k[None], v[None], attn_mask=mask[None])[0]
              return self.out(h + a)
      
      def packed_mask(seq_id, is_img):
          same = seq_id[:, None] == seq_id[None, :]                          # block diagonal: no cross-sample attention
          n = len(seq_id); causal = torch.ones(n, n, dtype=torch.bool).tril()
          both_img = is_img[:, None] & is_img[None, :]                       # optional: bidirectional inside the image
          return same & (causal | both_img)
      
      model = Tiny().eval()
      feats = torch.cat([s["feats"] for s in samples])
      with torch.no_grad():
          packed = model(ids, feats, pos, packed_mask(seq_id, ids == IMG))
          alone = torch.cat([model(p[0], s["feats"], torch.arange(len(p[0])),
                                   packed_mask(torch.zeros(len(p[0]), dtype=torch.long), p[0] == IMG))
                             for p, s in zip(parts, samples)])
          no_restart = model(ids, feats, torch.arange(len(ids)), packed_mask(seq_id, ids == IMG))
          leaky = model(ids, feats, pos, torch.ones(len(ids), len(ids), dtype=torch.bool).tril())
      print(f"{(packed - alone).abs().max():.1e}")                           # 7.2e-07: packing changes nothing
      print(f"{(no_restart - alone).abs().max():.2f}", f"{(leaky - alone).abs().max():.2f}")  # 3.37 1.28: both wrong
      
      # ---------- 4) the loss: sum over answer tokens, divided by the GLOBAL count ----------
      loss_tok = F.cross_entropy(packed, tgt, ignore_index=IGNORE, reduction="none")
      print(f"{loss_tok[tgt != IGNORE].mean():.4f}", f"{F.cross_entropy(packed, tgt, ignore_index=IGNORE):.4f}")  # 4.8876 4.8876
      
      def global_mean(micro):                       # micro: list of (sum of token losses, number of loss tokens)
          return sum(s for s, _ in micro) / sum(n for _, n in micro)
      micro = [(20.0, 10), (190.0, 190)]           # micro-batch 1: 10 tokens at loss 2.0; micro-batch 2: 190 at 1.0
      print(global_mean(micro), sum(s / n for s, n in micro) / len(micro))   # 1.05 (right) versus 1.5 (wrong)
      
      # ---------- 5) balance samples over GPUs (sort, then fill the lightest GPU) ----------
      def balance(lengths, n_gpu, cost=lambda n: n):
          order = sorted(range(len(lengths)), key=lambda i: -cost(lengths[i]))
          load, groups = [0.0] * n_gpu, [[] for _ in range(n_gpu)]
          for i in order:
              g = min(range(n_gpu), key=lambda g: load[g])
              groups[g].append(i); load[g] += cost(lengths[i])
          return groups, load
      L = [3000, 2900, 700, 650, 600, 500, 400, 300]
      print([sum(L[:4]), sum(L[4:])], balance(L, 2)[1])                     # [7250, 1800] [4550.0, 4500.0]

      The dense boolean mask here is for clarity. In real training you pass cu_seqlens (or position_ids) to a variable-length FlashAttention kernel, or a mask function to FlexAttention, and never build the \(N \times N\) matrix. The 3.37 and 1.28 are the two classic bugs: positions that do not restart, and a causal mask that lets samples read each other. Both run without any error message; only a test like this one finds them.

      Where this is used

      SettingTypical choiceWhy
      LLaVA-style SFT (one image per sample, 576 tokens)Padding plus modality and length groupingImage lengths are fixed, so grouping removes most padding with no kernel changes
      Native-resolution VLMs (Qwen2-VL, Qwen2.5-VL)Packed vision-encoder sequence with cu_seqlens; per-sample M-RoPE idsImage token counts vary from a few to thousands per image
      Tiled high-resolution models (InternVL, LLaVA-NeXT)Packing under a token limit and a tile limitBoth the LLM and the large vision encoder must stay balanced
      Vision-encoder pretraining (NaViT, SigLIP-style)Pack images of mixed resolution; masked attention and pooling; token droppingLess than 2% padding and about 5× more images at the same compute (NaViT)
      Video captioning and video QA trainingFew samples per row, cost-aware balancing, frame caps per sampleOne video can be tens of thousands of tokens; it dominates any GPU it lands on (session 206)
      Action recognition and video classificationPacked clips of different lengths and frame rates in the video encoder, masked pooling per clipSame idea as NaViT in time: no resizing of every clip to the same frame count
      Key-moment detection and temporal groundingLoss only on the timestamp answer; long clips packed alone or with short onesThe answer is a few tokens after many visual tokens; token mean would drown it among long captions
      Video retrieval and fingerprinting (embedding training)Packed clips, one pooled vector per clip with masked poolingThe pooled vector must not mix two clips; contrastive batches need many clips per step
      Multimodal serving (prefill)Variable-length kernels with cu_seqlens, continuous batchingThe same packing idea, without labels (session 196)

      Common mistakes

      • Packing with a plain causal mask. Samples read each other, the loss looks too good, and the model learns to depend on context that is absent at test time. Use a document mask or cu_seqlens.
      • Not restarting position ids. The image of the third sample sits at position 5,000; with RoPE or M-RoPE the model sees a different geometry than at test time.
      • Truncating inside an image. The number of placeholders no longer matches the number of feature vectors; the scatter fails, or worse, silently shifts features. Truncate only at turn boundaries, or drop the sample.
      • Loss on image or prompt tokens. Predicting placeholder ids is meaningless, and training on the prompt teaches the model to write questions. Labels must be −100 there.
      • Shifting the labels across the packed row. The last token of one sample then learns to predict the first token of the next. Shift inside each sample.
      • Averaging per-micro-batch means. It re-weights tokens by the size of their micro-batch (formula 188.8). Divide the summed loss by the global count of loss tokens.
      • Balancing only tokens when the vision encoder is heavy. A GPU with all the images is the straggler even if its token count is average.
      • Sorting the whole dataset by length. Every step then sees one kind of sample, and the loss curve has a pattern that is about length, not learning. Sort inside random megabatches.

      Interview questions

      Why does a VLM batch have more padding than a text batch with the same samples’ text?basic
      Because the sequence length is \(n_i = V_i + T_i\) (formula 188.1), and the visual part \(V_i\) varies from 0 (text-only samples) to thousands (tiles, native resolution, video). It is usually much larger than the text part, so the length spread is mostly the image spread. With four samples of 300, 696, 2,751 and 2,960 tokens, padding to the longest gives 56.6% efficiency: 43% of the tensor is padding.
      You pack samples into rows of 8,192 tokens. What else must change for training to stay correct?core
      (1) The attention mask must be block-diagonal: a token attends only to its own sample (pass cu_seqlens to a variable-length kernel). (2) Position ids must restart at 0 in every sample (or M-RoPE ids must be computed per sample). (3) Labels must be shifted inside each sample, so no sample predicts the start of the next. (4) The loss normalization must use the global count of loss tokens, because rows now hold different numbers of samples. (5) In the vision encoder, the images of a row also need their own segments. Without (1) and (2) the packed row is a different model; the test is to compare packed logits against separate rows.
      Why are image tokens excluded from the loss?basic
      The positions hold continuous vectors from the vision encoder, not vocabulary tokens. Their “target” would be the placeholder id, which carries no information; predicting it wastes capacity and adds a large, easy loss that hides the real signal. The model should learn to produce text answers given the image, so only answer tokens (including the end-of-turn token) get labels; image, system and user tokens get −100.
      Token mean, sample mean or square mean: what is the difference and how do you choose?core
      With weight \(w_s = x_s^{-q}\) per token of a sample with \(x_s\) answer tokens (formula 188.7), the total weight of a sample is proportional to \(x_s^{1-q}\). Token mean (\(q = 0\)) gives long answers most of the gradient: a 400-token transcription gets 99% against a 4-token answer. Sample mean (\(q = 1\)) gives every sample equal weight, which over-weights tokens of short answers and, per InternVL 2.5, can push the model toward short responses. InternVL 2.5 uses the middle, \(q = 0.5\) (9.1% / 90.9% in the same example). Choose by looking at the answer-length distribution of your mixture and at the outputs: too short answers suggest lowering \(q\), ignored short-answer tasks suggest raising it.
      Your loss with gradient accumulation of 8 does not match the loss with a batch 8 times larger. Why?core
      Each micro-batch divided its summed loss by its own token count, and the 8 results were averaged. That is a mean of means (formula 188.8), which gives tokens in small micro-batches more weight. With 10 tokens at loss 2.0 and 190 at 1.0, the mean of means is 1.5 while the true token mean is 1.05. Fix: sum token losses, divide by the total number of loss tokens of the whole accumulated batch (count them first, all-reduce across GPUs). Hugging Face fixed exactly this in October 2024.
      How does NaViT handle images of different sizes in one batch?core
      It keeps each image at its own resolution and aspect ratio, turns it into a variable number of patches, and packs the patch sequences of several images into one sequence. Masked self-attention stops patches from attending across images, masked pooling gives one vector per image, and factorized x/y position embeddings handle any grid. Packing is greedy (first sequence with room), and typically less than 2% of tokens are padding. It also enables per-image token dropping; the paper reports about five times more images processed at the same compute.
      One GPU is always the slowest in your VLM training. How do you find and fix it?advanced
      Log per-rank step time, token count, visual-token count and number of images. If the slow rank has more tokens, sort and balance tokens per GPU (LLaVA-style greedy: sort long to short, give each sample to the least-loaded GPU). If its token count is normal but it has more images, the vision encoder is the bottleneck: balance a cost \(c_i = n_i + n_i^2/n_\star + r V_i\) (formula 188.9) with \(r\) measured, or cap tiles per packed row as InternVL 2.5 does, or run the encoder on a balanced split of all images. If counts per GPU become unequal, keep the loss normalized by the global token count.
      How much attention work does a document mask save in a packed row?core
      The ratio of computed pairs is \(\sum_k n_k^2 / (\sum_k n_k)^2\) (formula 188.5), if the kernel skips fully masked blocks (variable-length FlashAttention, FlexAttention block masks). For samples of 300, 696, 2,751 and 2,960 tokens in one row of 6,707, that is 37.6%. A dense kernel with a boolean mask computes all pairs and gets no saving. With a causal mask both sides are halved, so the ratio is about the same.
      Should image tokens attend to each other bidirectionally?advanced
      It is a design choice. Inside one image there is no natural order, so causal attention forces the top-left patches to ignore the rest of the image in the language model. Bidirectional attention inside the image block (or a full prefix, as in PaliGemma) lets every visual token use the whole image and costs nothing at inference, because the image is known in full before generation. The cost is a more complex mask (it is no longer pure causal, so some fast causal kernels need a block mask), and the model must be trained with the same mask it is served with. Most LLaVA-style models keep plain causal attention and still work well.
      Why does sorting by length hurt training if you do it over the whole dataset?basic
      Batches become homogeneous: all long document samples come in one part of the epoch, all short VQA samples in another. The gradient of each step is then biased toward one task and one length, and the loss curve shows the order of the data instead of learning. Grouping inside random megabatches (LLaVA) keeps the batches mixed across steps while still removing most padding.
      A sample is longer than the maximum sequence length. What do you do?core
      Never cut inside an image: the placeholder count would not match the features. Options: reduce the visual tokens (lower the max pixels, fewer tiles, fewer frames, as InternVL 2.5 does by sharing a tile budget across images), split a multi-turn chat at a turn boundary into two samples with the image repeated, or drop the sample and log it. Check \(n_i\) in the data loader, before batching.

      Exercises

      1. A batch has four samples of 700, 700, 700 and 3,000 tokens. What is the padding efficiency (188.2)?
      2. Pack the same four samples into rows of 4,096 with first-fit decreasing. How many rows, and what is the efficiency?
      3. Three samples of 13, 4 and 13 tokens are packed into one row. Write cu_seqlens and the position id of the first token of the third sample.
      4. Two samples have answers of 4 and 400 tokens. What share of the loss weight does the short one get with token, square and sample averaging?
      5. Micro-batch 1 has 50 answer tokens with mean loss 3.0, micro-batch 2 has 150 with mean loss 1.0. What is the correct token-mean loss, and what does the mean of means give?
      6. In Figure 4, how many attention pairs does a causal mask allow over the 22 tokens, and how many does the document mask allow? Why is the second number so much smaller?
      Answerscheck
      1. \(5{,}100 / (4 \times 3{,}000) = 42.5\%\). 2. Sorted: 3,000 goes to row 1; 700 fits (3,700); the next 700 does not (4,400 \(\gt\) 4,096), so row 2 gets 700 + 700. Two rows: \(5{,}100 / 8{,}192 = 62.3\%\). 3. cu_seqlens \(= (0, 13, 17, 30)\); the first token of the third sample is at row position 17 and has position id \(17 - 17 = 0\). 4. Weights \(\propto x^{1-q}\): token mean \(4/404 = 1.0\%\); square mean \(2/(2 + 20) = 9.1\%\); sample mean 50%. 5. Correct: \((50 \cdot 3 + 150 \cdot 1)/200 = 1.5\). Mean of means: \((3 + 1)/2 = 2.0\). 6. Causal over 22 tokens: \(22 \cdot 23/2 = 253\). Document mask: \(9 \cdot 10/2 + 4 \cdot 5/2 + 9 \cdot 10/2 = 45 + 10 + 45 = 100\). Pairs grow with the square of the length, so cutting one row into three short blocks removes most of them (formula 188.5).

      Key takeaways

      • A multimodal sample has \(n_i = V_i + T_i\) positions (188.1); the visual part varies from 0 to thousands and decides the length. Count it in the data loader.
      • Padding wastes \(1 - \eta\) of the work (188.2), and more for attention; sorting inside megabatches and grouping by modality remove most of it.
      • Packing needs a document mask (188.3), restarted positions (188.4) and per-sample label shifts; with cu_seqlens kernels, attention work falls to \(\sum n_k^2 / (\sum n_k)^2\) (188.5). The vision encoder packs images the same way (NaViT).
      • The loss is only on answer tokens (188.6); image, prompt and pad positions get −100.
      • The averaging rule (188.7) decides how much long and short answers count; divide by the global count across micro-batches and GPUs (188.8).
      • Step time is the slowest GPU (188.9): sort and balance tokens per GPU, and balance the vision-encoder load too when it is heavy.

      Key formulas

      \[ n_i = V_i + T_i, \qquad \eta = \frac{\sum_i n_i}{B \max_i n_i}, \qquad p_t = t - c_{s(t)} \]
      \[ \mathcal{L} = \frac{\sum_s w_s \sum_{t \in s} \ell_t}{\sum_s w_s x_s}, \quad w_s = x_s^{-q}, \qquad \mathcal{L}_{\text{global}} = \frac{\sum_k S_k}{\sum_k N_k} \]
      \[ c_i = n_i + \frac{n_i^2}{n_\star} + r V_i, \qquad \text{step} = \max_g \sum_{i \in g} c_i \]

      Next, session 189 uses these batches to fine-tune a VLM: which parts to train, LoRA on the language model and the connector, and how the data mixture is set.

      References for this session

      1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
      2. Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
      3. Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
      4. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
      5. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
      6. Chen et al. (2024). Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (InternVL 2.5)
      7. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
      8. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
      9. Dong, Feng, Guessous, Liang, He (2024). Flex Attention: A Programming Model for Generating Optimized Attention Kernels
      10. Lee, Zucker, Kundu, Wynter, Ganti and Mishra (2024). Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2 (blog)
      11. The Hugging Face Transformers team (2024). Fixing Gradient Accumulation (blog)
      12. Liu et al. (2023). LLaVA code: constants.py, llava_trainer.py (modality and length grouped sampler) and scripts/v1_5/finetune.sh