Training video-language models
A Video-LLM is only as good as the video–text data it was trained on, and video training is expensive. This session follows the data from noisy speech transcripts to clean synthetic captions and time-stamped dense captions, then shows how to adapt a model to one domain (soccer) with LoRA, and how to make training cheaper with fewer frames, lower resolution, token merging and sequence packing.
What you will learn
- The three kinds of video–text data (ASR-aligned, captioned, synthetic) and what each one is good and bad at, with sizes from the real datasets
- Why speech transcripts are only weakly aligned with what is on screen, and how windows, filters and alignment models fix part of it
- How dense captions with timestamps become one training sequence with time tokens (Vid2Seq, TimeChat), and how many time bins you need
- A staged training recipe and how to mix data sources with temperature sampling
- Domain fine-tuning for sports with LoRA: which matrices, which rank, how many parameters and how much memory, for a Qwen2-7B-shaped LLM
- How frames, resolution, spatial merge and token merging change the token count and the training compute (6NT + 6LT²d)
- Sequence packing with a block-diagonal mask and restarted positions, and why it can lift token use from about 50% to over 95%
Why this matters
In session 205 we built the architecture of a Video-LLM: sample frames, encode them, merge tokens, tell the model the time. That architecture is almost the same in every system. What makes one Video-LLM much better than another is mostly how it is trained: which video–text data it saw, whether that data had timestamps, how the stages were ordered, and how much video the team could afford to push through the GPUs.
Video training has three problems that image training does not have so strongly:
- Good paired data is rare. The web has billions of images with alt-text, but few videos with text that describes what happens, and when. Most video text is speech (what people say), not a description (what people do).
- Time must be labeled. A question like “when does the goal happen?” needs training data with start and end times. Writing those by hand is slow.
- Video is expensive. One minute of video can be 15,000 visual tokens (see session 205). Training cost grows with tokens, and the attention part grows with the square of the tokens.
This session follows the four key points of the topic: the kinds of video–text data (ASR-aligned, captioned, synthetic), temporal annotations and dense captions, domain fine-tuning with LoRA (our example is soccer), and the efficiency tools that make it affordable (fewer frames, lower resolution, token merging, packing).
We use the book chapter on video-language models (Noyan et al., chapter 9), the data papers (HowTo100M, WebVid, InternVid, Panda-70M, ShareGPT4Video, LLaVA-Video-178K), the time-aware models Vid2Seq and TimeChat, the soccer datasets SoccerNet-v2, SoccerNet-Caption and MatchTime, and the LoRA, QLoRA and ToMe papers. After this session, session 213 shows how to check whether the training worked.
The training pipeline at a glance
Figure 1 puts the data on one map. The horizontal axis is how many clips a dataset has (log scale). The vertical axis is how well the text describes what is visible, at the right time. The two goals pull in opposite directions: the biggest datasets have the weakest text, and the cleanest data is the smallest. A training recipe uses each kind of data for the stage where its strength matters.
Kind 1: ASR-aligned data (what people say)
ASR (automatic speech recognition) turns the audio track into text with a start and end time for each sentence. Narrated videos, such as cooking or repair tutorials, come with this “free” text. HowTo100M (Miech et al., 2019) collected 136 million clips from 1.22 million narrated instructional videos, covering over 23,000 tasks, with no manual labels. [Certain] Vid2Seq later pretrained on YT-Temporal-1B, a larger set of narrated YouTube videos. [Certain]
The simplest way to make a training pair: take one ASR sentence, cut the video clip that plays while it is spoken, and pair the two. To catch actions that happen just before or after the words, widen the window by \(\delta\) seconds on each side:
- Read the symbols
- \(s_i, e_i\)
- start and end time (seconds) of ASR sentence \(i\)
- \(\delta\)
- the extra window on each side, in seconds
- \(T\)
- length of the whole video
- \(c_i\)
- the video clip that is paired with sentence \(i\)
The problem is that people do not speak exactly while they act. They say “now chop the onion” before they do it, explain after they did it, or say things that are not visible at all (“don’t forget to subscribe”). The TAN paper (Han et al., 2022) describes HowTo100M narrations as having “significant noise”, and “only weakly aligned when relevant”; the authors hand-labeled an 80-video, 10-hour subset (HTM-Align) to measure it. [Certain] To measure alignment we use temporal IoU (intersection over union of two time intervals):
- Read the symbols
- \(a = [a_0, a_1]\)
- the clip we cut from the ASR sentence
- \(b = [b_0, b_1]\)
- the interval where the action is really visible
- numerator
- length of the overlap (0 if they do not overlap)
- denominator
- length of the smallest interval that covers both (equal to the union when they overlap)
Figure 2 is a toy cooking video with 9 ASR sentences and 7 visible actions. Two sentences (the intro and “subscribe”) match nothing. Try to find a window \(\delta\) that pairs each sentence well with its action.
How to read Figure 2
Each row is one sentence. The green bar is when the action is on screen, the red bar is when the words are spoken, and the orange bar is the clip we would cut. With \(\delta = 0\) the mean tIoU is only 0.083: most clips miss their action. Widening the window helps (0.408 at 10 s) but each clip then also contains other steps. “After an alignment step” lets each clip move by up to 10 s to the best place, a stand-in for an alignment model such as TAN (here it uses the true times, so it is an upper bound). Now a small window is best: 0.722 at \(\delta = 2\), while \(\delta = 10\) falls to 0.508. Rows 1 and 6 never match anything; only filtering can remove them.
Three fixes are used in practice, often together:
- Widen and merge. Use \(\delta\) of a few seconds, or merge neighboring sentences into one longer clip. This raises the chance that the action is inside, at the cost of extra content.
- Filter. Score each pair with a video–text model (for example a CLIP-style similarity, see session 166) and drop low scores. This removes “subscribe” sentences.
- Re-align. Train a model to predict whether and where each sentence is visible. TAN does this for HowTo100M. MatchTime (Rao et al., 2024) found “prevalent video-text misalignment” in soccer commentary datasets and built a multi-modal temporal alignment pipeline to correct and filter them at scale, with 49 matches aligned by hand as a test set. [Certain]
Kind 2: captioned clips (what is visible)
The second kind pairs a clip with text that describes the picture. There are two ways to get it.
Text from the web. WebVid-2M (Bain et al., 2021) has over two million videos “with weak captions scraped from the internet”. [Certain] The captions come from a stock-footage site, so they describe the scene in one short sentence, but say little about motion or order. [Likely] Its paper also trained with a curriculum that starts by treating images as “frozen” snapshots of video and slowly adds temporal context. [Certain] This idea (an image is a one-frame video) is still used in every joint image–video recipe.
Text from captioning models. Modern datasets write the captions with models:
- InternVid (Wang et al., 2023): over 7 million videos lasting nearly 760K hours, cut into 234M clips, with 4.1B words of descriptions written by a multi-scale approach with LLMs. A contrastive video–text model, ViCLIP (ViT-L), was trained on it. [Certain] LLaVA-Video-178K later took some of its videos from InternVid-10M. [Certain]
- Panda-70M (Chen et al., 2024): 3.8M high-resolution videos from HD-VILA-100M, split into “semantically consistent” clips, giving 70M captioned clips. Several cross-modality teacher models each write a caption from the frames, the subtitles and the video description. A retrieval model, fine-tuned on a subset where people picked the best caption, then picks the best of the teachers’ captions for every clip. [Certain]
Model-written captions are much closer to the pixels than ASR, and they cost compute, not people. Their risks are hallucination (the captioner says something that is not there) and sameness (one captioner’s style everywhere). Panda-70M’s “many teachers plus a selector” design is a direct answer to both.
Kind 3: synthetic dense captions and QA
The third kind is small but rich: long, detailed captions and question–answer pairs written by a strong proprietary model. Two examples:
- ShareGPT4Video (Chen et al., 2024): 40K dense video captions written by GPT-4V. The authors found that giving GPT-4V many frames at once gave less detailed and sometimes temporally confused captions, so they use a differential strategy: describe the first frame, then describe what changed between each pair of key frames, then summarize. They then trained a captioner, ShareCaptioner-Video, and used it to caption 4.8M videos. [Certain]
- LLaVA-Video-178K (Zhang et al., 2024): 178,510 videos of up to 3 minutes, with 178K detailed captions, 960K open-ended questions and 196K multiple-choice questions, all written by GPT-4o. Captions are recurrent and three-level: every 10 s from 1-fps frames, a summary every 30 s, and one description of the whole video. [Certain] (Details in session 205.)
Both follow the same pattern: a big model sees the video piece by piece and writes captions that keep the order of events, because each step is conditioned on the previous text. A smaller “student” captioner is then trained on that output to make millions more. This is distillation of a captioner, and it is the main way video data scales today. [Likely]
- input: raw videos with titles, ASR and metadata
- filter: drop short, low-resolution, static or duplicate videos // fingerprinting, see Track 7
- cut each video into shots with a scene detector; merge very short shots
- ASR pairs: pair each sentence with clip \(c_i\) from formula (212.1)
- captions: run one or more captioners on each shot (frames + ASR + title as input)
- score every (clip, text) pair with a video–text model; keep the best caption, drop low scores
- dense: for a smaller subset, caption recurrently every \(w\) seconds, then summarize
- keep the start and end time of every caption // free temporal labels
- QA: from the dense captions, ask an LLM for open-ended and multiple-choice questions
- hold out human-labeled benchmarks; check no test video is in the training set
Temporal annotations and dense captions
A dense caption is a list of events, each with a start time, an end time and a sentence: “3.4–21.7 s: a player dribbles past two defenders; 21.7–26.2 s: he shoots and scores; …”. Classic human-labeled sets are ActivityNet Captions and YouCook2. [Certain] Dense captions teach a model the two skills that plain captions cannot: localization (when) and ordering (what comes after what). A model that saw only clip-level captions can say “someone scores” but not “at 22 s”.
How do we write a time in the training target? Vid2Seq (Yang et al., 2023) adds \(N\) special time tokens to the vocabulary. A time \(t\) in a video of length \(T\) becomes the token of its bin, and each event in the target is “start token, end token, sentence”, sorted by start time. Vid2Seq uses \(N = 100\) time tokens added to a 32,128-token SentencePiece vocabulary, with 100 frames at 1 fps as input. [Certain]
- Read the symbols
- \(t\)
- a time in seconds, for example the start of an event
- \(T\)
- length of the video in seconds
- \(N\)
- number of time tokens (bins); each bin covers \(T/N\) seconds
- \(b(t)\)
- the index of the time token, written as
<t17>for bin 17 - \(\hat t\)
- the time we read back from the token: the center of its bin
Time tokens are relative: bin 50 is the middle of any video. So the error grows with the video length. With \(N = 100\), a 2-minute clip has a worst-case error of 0.6 s, but a 1-hour video has 18 s. The other option is to write seconds as text (“21.7s”), as LLaVA-Video and many Qwen-based models do. [Certain] Text seconds are absolute and need no new vocabulary, but each number costs several tokens.
The trick that made Vid2Seq work at scale: ASR sentence boundaries are pseudo events. In pretraining, the transcribed sentences and their times become the target sequence, so the model learns to emit time tokens from millions of narrated videos with no human labels. [Certain] The paper’s ablation shows what the time tokens add: with untrimmed videos but no time tokens in pretraining, YouCook2 dense-captioning CIDEr was 35.0 and localization F1 23.3; adding time tokens raised them to 47.1 and 27.3. [Certain]
How to read Figure 3
Move \(N\) down to 10: each bin is 15 s wide, the first two events (3.4 to 21.7 s and 21.7 to 26.2 s) fall into bins 0–1 and 1–1, and the mean tIoU drops to 0.599. At \(N = 100\) (Vid2Seq’s choice) the largest error is 0.75 s and the mean tIoU is 0.976. Going to 300 gains little more (0.986), and every extra bin is one more vocabulary entry that must be learned. Switch to “Seconds as text” to see the token cost of absolute times: 55 tokens instead of 10 for these 5 events.
The clip is \(T = 150\) s and \(N = 100\), so each bin is 1.5 s. The event “he shoots and scores” runs from 21.7 to 26.2 s. Start: \(\lfloor 21.7 \times 100 / 150 \rfloor = \lfloor 14.47 \rfloor = 14\). End: \(\lfloor 26.2 \times 100/150 \rfloor = \lfloor 17.47 \rfloor = 17\). The target is <t14><t17> he shoots and scores. Reading back: \(\hat t = 14.5 \times 1.5 = 21.75\) s and \(17.5 \times 1.5 = 26.25\) s. Both errors are 0.05 s, well under the bound \(T/(2N) = 0.75\) s. As text, “21.7s-26.2s” costs about 11 tokens with a tokenizer that splits digits; as time tokens it costs 2.
Other models put the time on the input side. TimeChat (Ren et al., 2023) uses a timestamp-aware frame encoder that “binds visual content with the timestamp of each frame”, and a sliding video Q-Former that makes a token sequence whose length grows with the video. Its instruction set has 6 tasks and 125K instances. [Certain] It reported +27.5 R@1 (IoU = 0.5) on Charades-STA over earlier Video-LLMs. [Certain] Qwen2.5-VL puts absolute time into M-RoPE and, for grounding data, writes times in seconds and in an hour-minute-second-frame format. [Certain] Whatever the format, the rule is the same: the format used at training must be the format used at inference.
Putting it together: stages and data mixing
Most open Video-LLMs follow the same order of stages. Early stages use the largest, noisiest data and train only the small parts; later stages use cleaner data and train more of the model.
- start: an image VLM (vision encoder, projector, LLM) // skills from images carry over
- stage 1, alignment: freeze encoder and LLM; train the projector
- data: millions of short (clip, caption) or (clip, ASR) pairs; few frames per clip
- stage 2, captions and time: unfreeze the LLM (or add LoRA)
- data: dense, time-stamped captions; ASR boundaries as pseudo events; more frames
- stage 3, instructions: train projector + LLM, vision encoder with a small learning rate
- data: video QA and captions mixed with image instructions, sampled by formula (212.4)
- loss on answer tokens only, formula (212.5)
- stage 4 (optional), domain: LoRA on expert data + a share of general data // next section
- after every stage: run video and image benchmarks // catch forgetting early
Inside a stage, data comes from many sources of very different sizes. If we sample in proportion to size, a 20K-example domain set is almost never seen. A common fix is temperature sampling: raise each source size to a power \(\alpha \le 1\) before normalizing (the same rule used for languages in multilingual pretraining, see session 34):
- Read the symbols
- \(n_j\)
- number of examples in source \(j\)
- \(\alpha\)
- temperature exponent: 1 means “sample by size”, 0 means “all sources equal”
- \(p_j\)
- probability that the next training example comes from source \(j\)
Each example then gives the usual next-token loss on its answer tokens (see session 54). When samples are packed into one row (later in this session), the loss is averaged over all answer tokens in the batch:
- Read the symbols
- \(m\)
- index of a sample in the batch (one video or image with its conversation)
- \(\mathcal{A}_m\)
- positions of the answer tokens of sample \(m\); prompt and visual tokens get no loss
- \(\mathbf{V}_m, \mathbf{q}_m\)
- the visual tokens and the question of sample \(m\)
- \(y_{m,t}\)
- answer token \(t\) of sample \(m\); for a dense caption, time tokens are answer tokens too
Take 1,600,000 video samples and 1,100,000 image samples (the sizes LLaVA-Video reports for its video and image data [Certain]), plus a hypothetical 20,000-sample soccer set. With \(\alpha = 1\): \(p_{\text{soccer}} = 20{,}000 / 2{,}720{,}000 = 0.74\%\). With \(\alpha = 0.5\): \(\sqrt{1.6\text{M}} = 1{,}265\), \(\sqrt{1.1\text{M}} = 1{,}049\), \(\sqrt{20\text{K}} = 141\), so \(p_{\text{soccer}} = 141/2{,}455 = 5.76\%\). With \(\alpha = 0.3\) it is 12.4%. Lower \(\alpha\) means each soccer example is repeated more often; more than a few repeats per example risks memorizing it.
Domain fine-tuning with LoRA: a soccer example
A general Video-LLM knows what a ball and a goal are. It does not know what a “through ball”, an “offside trap” or a “VAR check” looks like, and it does not know which moments a soccer editor cares about. That knowledge lives in domain data. For soccer there is good public data:
- SoccerNet-v2 (Deliège et al., 2021): around 300K annotations over 500 untrimmed broadcast games, for action spotting, camera-shot segmentation and replay grounding. [Certain] Action spotting has 17 classes such as goal, card and corner. [Likely]
- SoccerNet-Caption (Mkhallati et al., 2023): almost 37K timestamped commentaries over 715.9 hours of broadcasts. The task is dense captioning where each comment is anchored to a single timestamp. [Certain]
- MatchTime (Rao et al., 2024): the same kind of commentary data, re-aligned and filtered, as described above. [Certain]
Full fine-tuning on 20K examples would be expensive and could make the model forget its general skills. LoRA (low-rank adaptation, Hu et al., 2021, see session 57) freezes every original weight \(W_0\) and learns a small low-rank change:
- Read the symbols
- \(W_0\)
- the frozen pretrained weight matrix, \(d_{\text{out}} \times d_{\text{in}}\)
- \(A, B\)
- the two small trainable matrices; their product has rank at most \(r\)
- \(r\)
- the rank, for example 8, 16 or 64
- \(\alpha\)
- a fixed scale; \(\alpha / r\) keeps the update size similar when you change \(r\)
- \(B_{\text{init}} = 0\)
- so at step 0 the model is exactly the pretrained model
Each adapted matrix adds \(r(d_{\text{in}} + d_{\text{out}})\) parameters. Over \(L\) layers and a set \(S\) of target matrices:
The LoRA paper reports that, compared with full fine-tuning of GPT-3 175B with Adam, LoRA cut trainable parameters by 10,000 times and GPU memory by 3 times, with no extra inference latency (the update can be merged into \(W_0\)). [Certain] QLoRA (Dettmers et al., 2023) stores the frozen weights in 4-bit NF4 and fine-tuned a 65B model on a single 48 GB GPU. [Certain] Figure 4 does the count for a Qwen2-7B-shaped LLM (28 layers, width 3,584, 28 query heads and 4 key-value heads of size 128, MLP width 18,944, from the released config [Certain]), which is the LLM size of many 7B Video-LLMs.
How to read Figure 4
The top bar is full fine-tuning: \(7.62\text{B} \times 16\) bytes = 121.8 GB before activations, more than one 80 GB GPU. The lower bar is LoRA: mostly the frozen copy of the model (15.2 GB in bf16), plus a thin slice for the trainable adapters. With \(r = 16\) on all linear layers, LoRA trains 40.4M parameters (0.53% of the model) and needs 15.9 GB. Doubling \(r\) doubles the adapter slice but barely moves the total. Switch to 4-bit to see where the rest of the memory goes. The per-matrix line shows why the MLP matrices dominate: 18,944 is much wider than 3,584.
Per layer, the shapes \((d_{\text{in}}, d_{\text{out}})\) are: q (3584, 3584), k (3584, 512), v (3584, 512), o (3584, 3584), gate and up (3584, 18944), down (18944, 3584). With \(r = 16\), formula (212.7) gives per layer: q and o \(16 \times 7{,}168 = 114{,}688\) each, k and v \(16 \times 4{,}096 = 65{,}536\) each, gate, up and down \(16 \times 22{,}528 = 360{,}448\) each. Sum: \(2 \times 114{,}688 + 2 \times 65{,}536 + 3 \times 360{,}448 = 1{,}441{,}792\). Times 28 layers: 40,370,176 parameters, 0.53% of 7,615,616,512. Only q and v: \(28 \times (114{,}688 + 65{,}536) = 5{,}046{,}272\) (0.066%).
- input: a general Video-LLM; domain clips with timestamped labels or commentary
- re-align and filter the domain text (MatchTime-style) // noisy times teach wrong times
- convert labels to the model’s own formats: chat template, time format, frame rate
- mix in general video and image data, for example by formula (212.4)
- freeze encoder, projector and LLM; add LoRA to the LLM linear layers (\(r\) = 8 to 64)
- optional: also train the projector, if domain visuals are unusual
- keep the sampler of the base model: same fps, same frame cap, same resolution
- train 1 to 3 epochs with a higher learning rate than full fine-tuning
- evaluate on held-out games (not held-out clips of the same games)
- also run general benchmarks to measure forgetting
- merge \(W_0 + \frac{\alpha}{r}BA\) for serving, or keep one adapter per sport
Two choices matter a lot for sports. First, resolution: a ball can be a few pixels wide in a broadcast frame, so a model that works on 224-pixel frames may not see it. Second, evaluation split: clips from the same game share teams, kits, stadium and light, so splitting by clip leaks information; split by game, or by season. [Likely] For more on LoRA variants see session 62 and, for VLMs, session 189.
Efficiency: fewer frames, lower resolution, token merging
For a video sample, the number of tokens \(T\) is set mostly by the visual part. From session 205, formula (205.3), with one more factor for token merging:
- Read the symbols
- \(K\)
- frames per sample
- \(\tau\)
- frames per temporal patch (1, or 2 in Qwen2-VL and Qwen2.5-VL)
- \(H, W, p, s\)
- frame size, patch size (14), and spatial merge (each \(s \times s\) patches become one token)
- \(\rho\)
- fraction of tokens kept after token merging or pruning (1 = keep all)
- \(T_{\text{text}}\)
- prompt and answer tokens; we use 300 in the figure
Training compute for one sample with \(T\) tokens, in the form of Kaplan et al. (2020): about 6 FLOPs per parameter per token for the matrix multiplies (2 forward, 4 backward), plus the attention term that grows with \(T^2\):
- Read the symbols
- \(N\)
- non-embedding parameters of the LLM (6.53B for the Qwen2-7B shape)
- \(L, d\)
- number of layers (28) and model width (3,584)
- \(6LT^2d\)
- attention scores and weighted sums, forward and backward (Kaplan’s per-token \(2 L T d\), times \(T\) tokens, times 3)
This counts only the LLM; the vision encoder adds its own cost per frame. See session 25 for the derivation. Figure 5 lets you turn the four knobs.
How to read Figure 5
At the baseline a sample is 8,492 tokens and 0.38 PFLOP, and attention is 11.6% of it. Halving the frames (32) or keeping half the tokens (\(\rho = 0.5\)) both give 4,396 tokens and 0.489 times the compute, a little better than half, because the \(T^2\) term shrinks by 4. Frames of 336 pixels give 0.55 times. Turn off both merges: 65,836 tokens, 13.8 times the compute, and attention becomes half of the cost. Large token counts hurt twice: more linear work, and much more attention work.
What each knob costs in quality:
- Fewer frames lose short events and timing. Use few frames in stage 1 (alignment needs “what”, not “when”), and more in stages 2 and 3.
- Lower resolution loses small objects and text. It is the wrong knob for sports balls and scoreboards; the right knob for scene-level captions.
- Spatial merge (\(2 \times 2\) pooling or an MLP merger) is the standard first step; almost every Video-LLM uses some form of it.
- Token merging (ToMe, Bolya et al., 2022) merges the most similar tokens inside the vision transformer, layer by layer. On video it gave 2.2 times the throughput of ViT-L with a 0.2–0.3% accuracy drop, without retraining, and up to 2 times faster MAE fine-tuning on video. [Certain] Video has many near-identical tokens across frames (a static background), so merging over time is especially effective; see session 206 and session 183.
- Train short, then long. Qwen2.5-VL trained its last pretraining phase with sequences packed to 32,768 tokens with long videos, after earlier phases with shorter ones. [Certain] Short sequences early make the expensive \(T^2\) term small while the model learns the basics. [Likely]
64 frames at \(448 \times 448\), \(\tau = 2\), \(p = 14\), \(s = 2\): \((64/2) \times (448/28)^2 = 32 \times 256 = 8{,}192\) visual tokens, plus 300 text tokens: \(T = 8{,}492\). Linear part: \(6 \times 6.526 \times 10^9 \times 8{,}492 = 3.33 \times 10^{14}\). Attention: \(6 \times 28 \times 8{,}492^2 \times 3{,}584 = 4.34 \times 10^{13}\). Total 0.38 PFLOP, 11.6% of it attention. With 128 frames, \(T = 16{,}684\) and the attention share rises to 20.4%.
Sequence packing
A training batch mixes text-only samples (a few hundred tokens), images (one to two thousand) and videos (several thousand to more than ten thousand). If we pad every sample in a micro-batch to the longest one, the GPU spends much of its time on padding tokens that carry no information. Packing puts several samples one after another into one row of fixed length \(L_{\max}\), so almost every position is a real token. We measure the share of real tokens:
- Read the symbols
- \(\ell_i\)
- length of sample \(i\) in tokens
- \(R\)
- number of rows computed (padded sequences, or packed rows)
- \(L_{\text{row}}\)
- length of each row: the longest sample in its micro-batch for padding, \(L_{\max}\) for packing
- \(\eta\)
- utilization: 1 means no padding at all
Packing has two rules that must not be broken (see session 18 and session 188):
- Block-diagonal attention. A token may only attend to earlier tokens of its own sample. In practice this is done without building a mask, by giving the kernel the start offset of each sample (the
cu_seqlensargument of FlashAttention’s variable-length function). [Certain] - Restarted positions. Position ids start again at 0 for each sample. With M-RoPE, each sample’s temporal, height and width ids are computed as if it were alone.
How to read Figure 6
At 50% video, random-order padding wastes about half the compute (utilization 50.5%), because almost every micro-batch has one long video that the short samples are padded up to. Grouping by length lifts it to 85.1%, but then each batch is all-video or all-text, which changes the gradient mix. Packing reaches 98.2% in 10 rows. With only text and images (0%), packing makes only 3 rows and the last is half empty (81.1%): packing works best when there are many samples per row.
- input: sample lengths \(\ell_1, \dots, \ell_n\) (each \(\le L_{\max}\)); row length \(L_{\max}\)
- sort samples by length, longest first
- for each sample \(i\):
- put it in the first row with at least \(\ell_i\) free tokens; if none, open a new row
- for each row: concatenate its samples; record start offsets
cu_seqlens - position ids restart at 0 at every offset // or M-RoPE ids per sample
- labels: answer tokens only; the first token of each sample has no previous-token target
- run attention with block-diagonal causal masking (varlen kernel)
- loss: average over all answer tokens in the batch, formula (212.5)
Code: time tokens, LoRA and packing in PyTorch
The code runs on a CPU in under a second and prints the same output every run. It checks three things from this session: the time-token target of formula (212.3), that a LoRA layer starts as the frozen layer and has \(r(d_{\text{in}} + d_{\text{out}})\) trainable parameters, and that packed attention with a block-diagonal mask gives exactly the same result as running each sample alone.
import torch, torch.nn.functional as F torch.manual_seed(0) # 1) Dense captions -> one target sequence with time tokens, formula (212.3) def time_token(t, dur, n_bins=100): return f"<t{min(int(t * n_bins / dur), n_bins - 1)}>" def dense_target(events, dur, n_bins=100): parts = [time_token(s, dur, n_bins) + time_token(e, dur, n_bins) + " " + txt for s, e, txt in sorted(events)] return " ".join(parts) events = [(21.7, 26.2, "he shoots and scores."), (3.4, 21.7, "a player dribbles past two defenders.")] print(dense_target(events, dur=150.0)) # 2) LoRA on a frozen linear layer, formulas (212.6) and (212.7) class LoRALinear(torch.nn.Module): def __init__(self, base, r=8, alpha=16): super().__init__() self.base = base.requires_grad_(False) # frozen W0 (and bias) self.A = torch.nn.Parameter(torch.randn(r, base.in_features) / r**0.5) self.B = torch.nn.Parameter(torch.zeros(base.out_features, r)) # B = 0: starts as W0 self.scale = alpha / r def forward(self, x): return self.base(x) + self.scale * (x @ self.A.T) @ self.B.T base = torch.nn.Linear(3584, 512) # a Qwen2-7B k_proj shape lora = LoRALinear(base, r=16) x = torch.randn(4, 3584) print(torch.allclose(lora(x), base(x)), # True: same output at step 0 sum(p.numel() for p in lora.parameters() if p.requires_grad)) # 16*(3584+512) = 65536 # 3) Packing: two samples in one row, block-diagonal causal mask, positions restart def packed_mask(lengths): n = sum(lengths); m = torch.zeros(n, n, dtype=torch.bool); o = 0 for L in lengths: m[o:o+L, o:o+L] = torch.tril(torch.ones(L, L, dtype=torch.bool)); o += L return m # True = may attend def attend(q, k, v, mask): s = (q @ k.T) / q.shape[-1]**0.5 return torch.softmax(s.masked_fill(~mask, float("-inf")), -1) @ v lens = [5, 3] q, k, v = (torch.randn(sum(lens), 8) for _ in range(3)) packed = attend(q, k, v, packed_mask(lens)) alone = torch.cat([attend(q[:5], k[:5], v[:5], packed_mask([5])), attend(q[5:], k[5:], v[5:], packed_mask([3]))]) pos = torch.cat([torch.arange(L) for L in lens]) # position ids restart per sample print(torch.allclose(packed, alone, atol=1e-6), pos.tolist(), f"{sum(lens) / (2 * max(lens)):.3f}") # padding would use 2 rows of 5
Output (the same on every run):
<t2><t14> a player dribbles past two defenders. <t14><t17> he shoots and scores. True 65536 True [0, 1, 2, 3, 4, 0, 1, 2] 0.800
Read it: the events are sorted by start time and each time becomes one token, as in the worked example. The LoRA layer gives exactly the base output at step 0 because \(B = 0\), and it trains \(16 \times (3{,}584 + 512) = 65{,}536\) parameters, while the frozen base has 1,835,520. Packing the 5-token and 3-token samples into one row gives the same attention output as running them alone, and uses 8 of 8 positions; padding would use 8 of 10 (utilization 0.8).
Where this is used
| Application | Training data that matters | Efficiency choice |
|---|---|---|
| Sports highlights and key-moment detection | Timestamped events (SoccerNet-v2), aligned commentary (MatchTime); domain LoRA (session 207) | Dense frames around events, enough resolution for the ball and scoreboard |
| Video retrieval and Video-RAG | Large captioned sets (InternVid, Panda-70M) for contrastive training of the retriever (session 208) | Few frames, small encoder; retriever runs on every clip |
| Dense captioning and chaptering of long videos | Dense, time-stamped captions; ASR boundaries as pseudo events (Vid2Seq) | Time tokens or text seconds; packing of long samples |
| Instructional video assistants | ASR-aligned narrated videos (HowTo100M) after filtering and re-alignment | ASR text as extra input tokens, cheaper than more frames |
| Action recognition, zero-shot | Captions that describe motion, not only scenes (session 203) | Keep temporal order; avoid heavy pooling over time |
| Content moderation, surveillance search | Domain data with rare events; strong mixing (low \(\alpha\)) so rare classes are seen | LoRA per policy or per camera domain; low resolution is often enough |
| Video question answering | Synthetic QA (LLaVA-Video-178K) plus image instructions (session 210) | Packing mixed image and video samples; short-then-long schedule |
Common mistakes
- Treating ASR as a caption. Speech is often before, after, or unrelated to the action (Figure 2). Use windows, filtering and re-alignment, and do not use raw ASR for the final stages.
- Changing the time format between training and use. A model trained on
<t17>tokens, or on “21.7s”, must be prompted and parsed in that format. The same holds for the frame rate and the frame cap. - Relative time tokens on long videos. With 100 bins, a 1-hour video has 36-second bins (18 s worst-case error). Use more bins, absolute seconds, or work on chunks.
- Splitting domain data by clip. Clips from one game, one camera or one recipe video are near-duplicates. Split by game, source video, or date; otherwise the test score is inflated.
- Fine-tuning only on domain data. A soccer-only LoRA can forget general skills and the chat format. Mix in general data and evaluate both.
- Packing without the block-diagonal mask or with running positions. Samples then attend to each other, and positions (or M-RoPE time ids) of the second sample are wrong. The loss still goes down, so the bug is easy to miss.
- Lowering resolution for a small-object task. It saves the most tokens and loses exactly what sports and OCR tasks need. Reduce frames or merge tokens first, and test on the hard cases.
- Letting test videos leak into training. Web-scale datasets often contain benchmark videos or re-uploads. Deduplicate with video fingerprints before training.
Interview questions
What are the main kinds of video–text training data, and what is each good for?basic
Why is ASR text a weak label, and how do you make it better?core
How does Vid2Seq represent time, and what is the error of that representation?core
Time tokens or seconds written as text: which would you choose?advanced
Describe a staged training recipe for a Video-LLM and say why the order matters.core
You need a soccer commentary model and have one 80 GB GPU. Plan it.advanced
How many parameters does LoRA add, and which matrices dominate?core
What are the four efficiency knobs for video training, and what does each cost?core
What is sequence packing and what can go wrong?core
cu_seqlens) so samples do not see each other, position ids (or M-RoPE ids) restarted per sample, and labels that do not cross sample borders. Bugs here still let the loss go down, so they are found only by tests like the one in the code section. Packing also changes loss weighting between per-token and per-sample averaging.How does Panda-70M make captions better than a single captioning model?advanced
Your Video-LLM fine-tuned on sports gets great validation scores but fails on new matches. Why?advanced
Exercises
- A narrated video has a sentence from 40 to 43 s; the action it describes is visible from 31 to 38 s. Compute the tIoU of the clip with \(\delta = 0\) and with \(\delta = 5\).
- Vid2Seq-style time tokens with \(N = 100\): a 20-minute video has an event from 312 s to 340 s. Which tokens are written, and what is the worst-case error?
- Using formula (212.7), how many LoRA parameters do q, k, v, o add at \(r = 64\) for the Qwen2-7B shape? What fraction of 7,615,616,512 is that?
- With formula (212.4) and sources of 1,600,000, 1,100,000 and 20,000 examples, find \(p\) for the smallest source at \(\alpha = 0\). How many times would each of its examples be seen in 1 million draws?
- Formula (212.8): 32 frames at \(336 \times 336\), \(\tau = 2\), \(s = 2\), \(\rho = 1\), 300 text tokens. How many tokens? If ToMe keeps \(\rho = 0.6\), how many visual tokens?
- Four samples of 9,000, 3,000, 2,000 and 2,000 tokens. Compute utilization with padding as one micro-batch of 4, and with packing into rows of 16,384.
Answerscheck
<t26><t28>. Worst-case error \(T/(2N) = 6\) s; read-back times are 318 s and 342 s. 3. Per layer: \(64 \times (7{,}168 + 4{,}096 + 4{,}096 + 7{,}168) = 64 \times 22{,}528 = 1{,}441{,}792\); times 28 = 40,370,176, which is 0.53%, the same as all linear layers at \(r = 16\). 4. At \(\alpha = 0\) every source has \(p = 1/3\): 333,333 draws for 20,000 examples, so each is seen about 16.7 times: a strong risk of memorizing. 5. \(16 \times (336/28)^2 = 16 \times 144 = 2{,}304\) visual tokens, \(T = 2{,}604\). With \(\rho = 0.6\): \(\lfloor 1{,}382.4 \rfloor = 1{,}382\) visual tokens. 6. Total 16,000 real tokens. Padding: \(4 \times 9{,}000 = 36{,}000\) positions, utilization 44.4%. Packing: all four fit in one row of 16,384, utilization \(16{,}000/16{,}384 = 97.7\%\).Key takeaways
- Video–text data comes in three kinds: ASR-aligned (huge, weakly aligned), captioned (web or model-written, closer to pixels), and synthetic dense captions and QA (small, rich, ordered). Recipes go from large and noisy to small and clean.
- ASR clips (212.1) often miss their action; measure with tIoU (212.2), and fix with windows, filtering and alignment models.
- Dense captions with timestamps teach “when”. Time tokens (212.3) cost 2 tokens per event but their error grows with video length; text seconds are absolute but cost more tokens. Train and infer in the same format.
- Mix sources with temperature sampling (212.4) and train with answer-only loss (212.5); evaluate images and video after each stage.
- LoRA (212.6, 212.7) makes domain fine-tuning cheap: 40.4M trainable parameters (0.53%) for all linear layers of a 7B LLM at \(r = 16\), about 15.9 GB instead of 121.8 GB before activations.
- Tokens (212.8) drive cost (212.9); frames, resolution, spatial merge and ToMe each cut tokens with a different quality cost. Packing (212.10) removes padding but needs a block-diagonal mask and restarted positions.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Zhang et al. (2024). Video Instruction Tuning With Synthetic Data
- Wang et al. (2023). InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
- Miech, Zhukov, Alayrac, Tapaswi, Laptev & Sivic (2019). HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Bain, Nagrani, Varol & Zisserman (2021). Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
- Chen et al. (2024). Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
- Chen et al. (2024). ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
- Han, Xie & Zisserman (2022). Temporal Alignment Networks for Long-term Video
- Yang et al. (2023). Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
- Ren et al. (2023). TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
- Deliège et al. (2021). SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos
- Mkhallati et al. (2023). SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries
- Rao et al. (2024). MatchTime: Towards Automatic Soccer Game Commentary Generation
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Bolya et al. (2022). Token Merging: Your ViT But Faster
- Kaplan et al. (2020). Scaling Laws for Neural Language Models
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Lin et al. (2023). Video-LLaVA: Learning United Visual Representation by Alignment Before Projection