LLM & VLM courseBlog
212Track 8: Video and multimodal systems · session 12 of 21

Training video-language models

A Video-LLM is only as good as the video–text data it was trained on, and video training is expensive. This session follows the data from noisy speech transcripts to clean synthetic captions and time-stamped dense captions, then shows how to adapt a model to one domain (soccer) with LoRA, and how to make training cheaper with fewer frames, lower resolution, token merging and sequence packing.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on Video-LLMs (205) and long video (206); LoRA basics; masks and packing

What you will learn

  • The three kinds of video–text data (ASR-aligned, captioned, synthetic) and what each one is good and bad at, with sizes from the real datasets
  • Why speech transcripts are only weakly aligned with what is on screen, and how windows, filters and alignment models fix part of it
  • How dense captions with timestamps become one training sequence with time tokens (Vid2Seq, TimeChat), and how many time bins you need
  • A staged training recipe and how to mix data sources with temperature sampling
  • Domain fine-tuning for sports with LoRA: which matrices, which rank, how many parameters and how much memory, for a Qwen2-7B-shaped LLM
  • How frames, resolution, spatial merge and token merging change the token count and the training compute (6NT + 6LT²d)
  • Sequence packing with a block-diagonal mask and restarted positions, and why it can lift token use from about 50% to over 95%

Why this matters

In session 205 we built the architecture of a Video-LLM: sample frames, encode them, merge tokens, tell the model the time. That architecture is almost the same in every system. What makes one Video-LLM much better than another is mostly how it is trained: which video–text data it saw, whether that data had timestamps, how the stages were ordered, and how much video the team could afford to push through the GPUs.

Video training has three problems that image training does not have so strongly:

  1. Good paired data is rare. The web has billions of images with alt-text, but few videos with text that describes what happens, and when. Most video text is speech (what people say), not a description (what people do).
  2. Time must be labeled. A question like “when does the goal happen?” needs training data with start and end times. Writing those by hand is slow.
  3. Video is expensive. One minute of video can be 15,000 visual tokens (see session 205). Training cost grows with tokens, and the attention part grows with the square of the tokens.

This session follows the four key points of the topic: the kinds of video–text data (ASR-aligned, captioned, synthetic), temporal annotations and dense captions, domain fine-tuning with LoRA (our example is soccer), and the efficiency tools that make it affordable (fewer frames, lower resolution, token merging, packing).

We use the book chapter on video-language models (Noyan et al., chapter 9), the data papers (HowTo100M, WebVid, InternVid, Panda-70M, ShareGPT4Video, LLaVA-Video-178K), the time-aware models Vid2Seq and TimeChat, the soccer datasets SoccerNet-v2, SoccerNet-Caption and MatchTime, and the LoRA, QLoRA and ToMe papers. After this session, session 213 shows how to check whether the training worked.

The training pipeline at a glance

Figure 1 puts the data on one map. The horizontal axis is how many clips a dataset has (log scale). The vertical axis is how well the text describes what is visible, at the right time. The two goals pull in opposite directions: the biggest datasets have the weakest text, and the cleanest data is the smallest. A training recipe uses each kind of data for the stage where its strength matters.

number of clips or videos (log scale) how well text matches the video (qualitative) HowTo100M: 136M clips, ASR InternVid: 234M clips, LLM captions Panda-70M: 70M clips, teacher captions WebVid: over 2M videos, web text ShareCaptioner-Video: 4.8M LLaVA-Video-178K ShareGPT4Video: 40K GPT-4V SoccerNet-Caption: ~37K comments 10K100K1M10M100M Stage 1: alignmenthuge, noisy pairs; train projector Stage 2: captions + timedense, timestamped captions Stage 3: instructionssynthetic QA + images, all params Stage 4: domain (LoRA)small, expert data, e.g. soccer Red: speech transcripts (ASR). Orange: captions from the web or from captioning models.Green: dense synthetic captions and QA. Grey: a domain dataset.
Figure 1. The video–text data landscape and the stages that use it. Sizes are from the dataset papers; the vertical position is a qualitative judgment, not a measured score. Big and noisy data comes first, small and clean data comes last.

Kind 1: ASR-aligned data (what people say)

ASR (automatic speech recognition) turns the audio track into text with a start and end time for each sentence. Narrated videos, such as cooking or repair tutorials, come with this “free” text. HowTo100M (Miech et al., 2019) collected 136 million clips from 1.22 million narrated instructional videos, covering over 23,000 tasks, with no manual labels. [Certain] Vid2Seq later pretrained on YT-Temporal-1B, a larger set of narrated YouTube videos. [Certain]

The simplest way to make a training pair: take one ASR sentence, cut the video clip that plays while it is spoken, and pair the two. To catch actions that happen just before or after the words, widen the window by \(\delta\) seconds on each side:

\[ c_i = \big[\, \max(0,\ s_i - \delta),\ \min(T,\ e_i + \delta) \,\big] \tag{212.1} \]
Read the symbols
\(s_i, e_i\)
start and end time (seconds) of ASR sentence \(i\)
\(\delta\)
the extra window on each side, in seconds
\(T\)
length of the whole video
\(c_i\)
the video clip that is paired with sentence \(i\)

The problem is that people do not speak exactly while they act. They say “now chop the onion” before they do it, explain after they did it, or say things that are not visible at all (“don’t forget to subscribe”). The TAN paper (Han et al., 2022) describes HowTo100M narrations as having “significant noise”, and “only weakly aligned when relevant”; the authors hand-labeled an 80-video, 10-hour subset (HTM-Align) to measure it. [Certain] To measure alignment we use temporal IoU (intersection over union of two time intervals):

\[ \mathrm{tIoU}(a, b) = \frac{\max\big(0,\ \min(a_1, b_1) - \max(a_0, b_0)\big)}{\max(a_1, b_1) - \min(a_0, b_0)} \tag{212.2} \]
Read the symbols
\(a = [a_0, a_1]\)
the clip we cut from the ASR sentence
\(b = [b_0, b_1]\)
the interval where the action is really visible
numerator
length of the overlap (0 if they do not overlap)
denominator
length of the smallest interval that covers both (equal to the union when they overlap)

Figure 2 is a toy cooking video with 9 ASR sentences and 7 visible actions. Two sentences (the intro and “subscribe”) match nothing. Try to find a window \(\delta\) that pairs each sentence well with its action.

action visibleASR sentenceclip paired with the sentence
Figure 2. A toy 120-second cooking video (made-up times, chosen to show common patterns: words before the action, during it, and after it). The readout gives the mean tIoU (212.2) between each clip and its true action, over the 7 sentences that have a visible action.
How to read Figure 2

Each row is one sentence. The green bar is when the action is on screen, the red bar is when the words are spoken, and the orange bar is the clip we would cut. With \(\delta = 0\) the mean tIoU is only 0.083: most clips miss their action. Widening the window helps (0.408 at 10 s) but each clip then also contains other steps. “After an alignment step” lets each clip move by up to 10 s to the best place, a stand-in for an alignment model such as TAN (here it uses the true times, so it is an upper bound). Now a small window is best: 0.722 at \(\delta = 2\), while \(\delta = 10\) falls to 0.508. Rows 1 and 6 never match anything; only filtering can remove them.

Three fixes are used in practice, often together:

  • Widen and merge. Use \(\delta\) of a few seconds, or merge neighboring sentences into one longer clip. This raises the chance that the action is inside, at the cost of extra content.
  • Filter. Score each pair with a video–text model (for example a CLIP-style similarity, see session 166) and drop low scores. This removes “subscribe” sentences.
  • Re-align. Train a model to predict whether and where each sentence is visible. TAN does this for HowTo100M. MatchTime (Rao et al., 2024) found “prevalent video-text misalignment” in soccer commentary datasets and built a multi-modal temporal alignment pipeline to correct and filter them at scale, with 49 matches aligned by hand as a test set. [Certain]

Kind 2: captioned clips (what is visible)

The second kind pairs a clip with text that describes the picture. There are two ways to get it.

Text from the web. WebVid-2M (Bain et al., 2021) has over two million videos “with weak captions scraped from the internet”. [Certain] The captions come from a stock-footage site, so they describe the scene in one short sentence, but say little about motion or order. [Likely] Its paper also trained with a curriculum that starts by treating images as “frozen” snapshots of video and slowly adds temporal context. [Certain] This idea (an image is a one-frame video) is still used in every joint image–video recipe.

Text from captioning models. Modern datasets write the captions with models:

  • InternVid (Wang et al., 2023): over 7 million videos lasting nearly 760K hours, cut into 234M clips, with 4.1B words of descriptions written by a multi-scale approach with LLMs. A contrastive video–text model, ViCLIP (ViT-L), was trained on it. [Certain] LLaVA-Video-178K later took some of its videos from InternVid-10M. [Certain]
  • Panda-70M (Chen et al., 2024): 3.8M high-resolution videos from HD-VILA-100M, split into “semantically consistent” clips, giving 70M captioned clips. Several cross-modality teacher models each write a caption from the frames, the subtitles and the video description. A retrieval model, fine-tuned on a subset where people picked the best caption, then picks the best of the teachers’ captions for every clip. [Certain]

Model-written captions are much closer to the pixels than ASR, and they cost compute, not people. Their risks are hallucination (the captioner says something that is not there) and sameness (one captioner’s style everywhere). Panda-70M’s “many teachers plus a selector” design is a direct answer to both.

Kind 3: synthetic dense captions and QA

The third kind is small but rich: long, detailed captions and question–answer pairs written by a strong proprietary model. Two examples:

  • ShareGPT4Video (Chen et al., 2024): 40K dense video captions written by GPT-4V. The authors found that giving GPT-4V many frames at once gave less detailed and sometimes temporally confused captions, so they use a differential strategy: describe the first frame, then describe what changed between each pair of key frames, then summarize. They then trained a captioner, ShareCaptioner-Video, and used it to caption 4.8M videos. [Certain]
  • LLaVA-Video-178K (Zhang et al., 2024): 178,510 videos of up to 3 minutes, with 178K detailed captions, 960K open-ended questions and 196K multiple-choice questions, all written by GPT-4o. Captions are recurrent and three-level: every 10 s from 1-fps frames, a summary every 30 s, and one description of the whole video. [Certain] (Details in session 205.)

Both follow the same pattern: a big model sees the video piece by piece and writes captions that keep the order of events, because each step is conditioned on the previous text. A smaller “student” captioner is then trained on that output to make millions more. This is distillation of a captioner, and it is the main way video data scales today. [Likely]

Algorithm 1: building a video–text corpus (typical pipeline)
  1. input: raw videos with titles, ASR and metadata
  2. filter: drop short, low-resolution, static or duplicate videos // fingerprinting, see Track 7
  3. cut each video into shots with a scene detector; merge very short shots
  4. ASR pairs: pair each sentence with clip \(c_i\) from formula (212.1)
  5. captions: run one or more captioners on each shot (frames + ASR + title as input)
  6. score every (clip, text) pair with a video–text model; keep the best caption, drop low scores
  7. dense: for a smaller subset, caption recurrently every \(w\) seconds, then summarize
  8. keep the start and end time of every caption // free temporal labels
  9. QA: from the dense captions, ask an LLM for open-ended and multiple-choice questions
  10. hold out human-labeled benchmarks; check no test video is in the training set

Temporal annotations and dense captions

A dense caption is a list of events, each with a start time, an end time and a sentence: “3.4–21.7 s: a player dribbles past two defenders; 21.7–26.2 s: he shoots and scores; …”. Classic human-labeled sets are ActivityNet Captions and YouCook2. [Certain] Dense captions teach a model the two skills that plain captions cannot: localization (when) and ordering (what comes after what). A model that saw only clip-level captions can say “someone scores” but not “at 22 s”.

How do we write a time in the training target? Vid2Seq (Yang et al., 2023) adds \(N\) special time tokens to the vocabulary. A time \(t\) in a video of length \(T\) becomes the token of its bin, and each event in the target is “start token, end token, sentence”, sorted by start time. Vid2Seq uses \(N = 100\) time tokens added to a 32,128-token SentencePiece vocabulary, with 100 frames at 1 fps as input. [Certain]

\[ b(t) = \min\!\left(\left\lfloor \frac{t\,N}{T} \right\rfloor,\ N - 1\right), \qquad \hat t = \left(b + \tfrac12\right)\frac{T}{N}, \qquad |\hat t - t| \le \frac{T}{2N} \tag{212.3} \]
Read the symbols
\(t\)
a time in seconds, for example the start of an event
\(T\)
length of the video in seconds
\(N\)
number of time tokens (bins); each bin covers \(T/N\) seconds
\(b(t)\)
the index of the time token, written as <t17> for bin 17
\(\hat t\)
the time we read back from the token: the center of its bin

Time tokens are relative: bin 50 is the middle of any video. So the error grows with the video length. With \(N = 100\), a 2-minute clip has a worst-case error of 0.6 s, but a 1-hour video has 18 s. The other option is to write seconds as text (“21.7s”), as LLaVA-Video and many Qwen-based models do. [Certain] Text seconds are absolute and need no new vocabulary, but each number costs several tokens.

The trick that made Vid2Seq work at scale: ASR sentence boundaries are pseudo events. In pretraining, the transcribed sentences and their times become the target sequence, so the model learns to emit time tokens from millions of narrated videos with no human labels. [Certain] The paper’s ablation shows what the time tokens add: with untrimmed videos but no time tokens in pretraining, YouCook2 dense-captioning CIDEr was 35.0 and localization F1 23.3; adding time tokens raised them to 47.1 and 27.3. [Certain]

true eventread back from tokens
Figure 3. A made-up dense caption of a 150-second soccer clip with 5 events. Each event is turned into tokens with formula (212.3) and read back. The readout gives the largest time error, the mean tIoU (212.2) between true and read-back events, and how many tokens the times cost.
How to read Figure 3

Move \(N\) down to 10: each bin is 15 s wide, the first two events (3.4 to 21.7 s and 21.7 to 26.2 s) fall into bins 0–1 and 1–1, and the mean tIoU drops to 0.599. At \(N = 100\) (Vid2Seq’s choice) the largest error is 0.75 s and the mean tIoU is 0.976. Going to 300 gains little more (0.986), and every extra bin is one more vocabulary entry that must be learned. Switch to “Seconds as text” to see the token cost of absolute times: 55 tokens instead of 10 for these 5 events.

Worked example: time tokens for one event

The clip is \(T = 150\) s and \(N = 100\), so each bin is 1.5 s. The event “he shoots and scores” runs from 21.7 to 26.2 s. Start: \(\lfloor 21.7 \times 100 / 150 \rfloor = \lfloor 14.47 \rfloor = 14\). End: \(\lfloor 26.2 \times 100/150 \rfloor = \lfloor 17.47 \rfloor = 17\). The target is <t14><t17> he shoots and scores. Reading back: \(\hat t = 14.5 \times 1.5 = 21.75\) s and \(17.5 \times 1.5 = 26.25\) s. Both errors are 0.05 s, well under the bound \(T/(2N) = 0.75\) s. As text, “21.7s-26.2s” costs about 11 tokens with a tokenizer that splits digits; as time tokens it costs 2.

Other models put the time on the input side. TimeChat (Ren et al., 2023) uses a timestamp-aware frame encoder that “binds visual content with the timestamp of each frame”, and a sliding video Q-Former that makes a token sequence whose length grows with the video. Its instruction set has 6 tasks and 125K instances. [Certain] It reported +27.5 R@1 (IoU = 0.5) on Charades-STA over earlier Video-LLMs. [Certain] Qwen2.5-VL puts absolute time into M-RoPE and, for grounding data, writes times in seconds and in an hour-minute-second-frame format. [Certain] Whatever the format, the rule is the same: the format used at training must be the format used at inference.

Putting it together: stages and data mixing

Most open Video-LLMs follow the same order of stages. Early stages use the largest, noisiest data and train only the small parts; later stages use cleaner data and train more of the model.

Algorithm 2: a staged training recipe for a Video-LLM
  1. start: an image VLM (vision encoder, projector, LLM) // skills from images carry over
  2. stage 1, alignment: freeze encoder and LLM; train the projector
  3. data: millions of short (clip, caption) or (clip, ASR) pairs; few frames per clip
  4. stage 2, captions and time: unfreeze the LLM (or add LoRA)
  5. data: dense, time-stamped captions; ASR boundaries as pseudo events; more frames
  6. stage 3, instructions: train projector + LLM, vision encoder with a small learning rate
  7. data: video QA and captions mixed with image instructions, sampled by formula (212.4)
  8. loss on answer tokens only, formula (212.5)
  9. stage 4 (optional), domain: LoRA on expert data + a share of general data // next section
  10. after every stage: run video and image benchmarks // catch forgetting early

Inside a stage, data comes from many sources of very different sizes. If we sample in proportion to size, a 20K-example domain set is almost never seen. A common fix is temperature sampling: raise each source size to a power \(\alpha \le 1\) before normalizing (the same rule used for languages in multilingual pretraining, see session 34):

\[ p_j = \frac{n_j^{\,\alpha}}{\sum_{k} n_k^{\,\alpha}} \tag{212.4} \]
Read the symbols
\(n_j\)
number of examples in source \(j\)
\(\alpha\)
temperature exponent: 1 means “sample by size”, 0 means “all sources equal”
\(p_j\)
probability that the next training example comes from source \(j\)

Each example then gives the usual next-token loss on its answer tokens (see session 54). When samples are packed into one row (later in this session), the loss is averaged over all answer tokens in the batch:

\[ \mathcal{L}(\theta) = -\frac{1}{\sum_{m} |\mathcal{A}_m|} \sum_{m} \sum_{t \in \mathcal{A}_m} \log p_\theta\!\left(y_{m,t} \mid y_{m,\lt t},\ \mathbf{V}_m,\ \mathbf{q}_m\right) \tag{212.5} \]
Read the symbols
\(m\)
index of a sample in the batch (one video or image with its conversation)
\(\mathcal{A}_m\)
positions of the answer tokens of sample \(m\); prompt and visual tokens get no loss
\(\mathbf{V}_m, \mathbf{q}_m\)
the visual tokens and the question of sample \(m\)
\(y_{m,t}\)
answer token \(t\) of sample \(m\); for a dense caption, time tokens are answer tokens too
Worked example: mixing a soccer set into a general mix

Take 1,600,000 video samples and 1,100,000 image samples (the sizes LLaVA-Video reports for its video and image data [Certain]), plus a hypothetical 20,000-sample soccer set. With \(\alpha = 1\): \(p_{\text{soccer}} = 20{,}000 / 2{,}720{,}000 = 0.74\%\). With \(\alpha = 0.5\): \(\sqrt{1.6\text{M}} = 1{,}265\), \(\sqrt{1.1\text{M}} = 1{,}049\), \(\sqrt{20\text{K}} = 141\), so \(p_{\text{soccer}} = 141/2{,}455 = 5.76\%\). With \(\alpha = 0.3\) it is 12.4%. Lower \(\alpha\) means each soccer example is repeated more often; more than a few repeats per example risks memorizing it.

Domain fine-tuning with LoRA: a soccer example

A general Video-LLM knows what a ball and a goal are. It does not know what a “through ball”, an “offside trap” or a “VAR check” looks like, and it does not know which moments a soccer editor cares about. That knowledge lives in domain data. For soccer there is good public data:

  • SoccerNet-v2 (Deliège et al., 2021): around 300K annotations over 500 untrimmed broadcast games, for action spotting, camera-shot segmentation and replay grounding. [Certain] Action spotting has 17 classes such as goal, card and corner. [Likely]
  • SoccerNet-Caption (Mkhallati et al., 2023): almost 37K timestamped commentaries over 715.9 hours of broadcasts. The task is dense captioning where each comment is anchored to a single timestamp. [Certain]
  • MatchTime (Rao et al., 2024): the same kind of commentary data, re-aligned and filtered, as described above. [Certain]

Full fine-tuning on 20K examples would be expensive and could make the model forget its general skills. LoRA (low-rank adaptation, Hu et al., 2021, see session 57) freezes every original weight \(W_0\) and learns a small low-rank change:

\[ W = W_0 + \frac{\alpha}{r}\, B A, \qquad A \in \R^{r \times d_{\text{in}}},\ B \in \R^{d_{\text{out}} \times r},\ B_{\text{init}} = 0 \tag{212.6} \]
Read the symbols
\(W_0\)
the frozen pretrained weight matrix, \(d_{\text{out}} \times d_{\text{in}}\)
\(A, B\)
the two small trainable matrices; their product has rank at most \(r\)
\(r\)
the rank, for example 8, 16 or 64
\(\alpha\)
a fixed scale; \(\alpha / r\) keeps the update size similar when you change \(r\)
\(B_{\text{init}} = 0\)
so at step 0 the model is exactly the pretrained model

Each adapted matrix adds \(r(d_{\text{in}} + d_{\text{out}})\) parameters. Over \(L\) layers and a set \(S\) of target matrices:

\[ P_{\text{LoRA}} = L \sum_{W \in S} r\,\big(d_{\text{in}}(W) + d_{\text{out}}(W)\big) \tag{212.7} \]

The LoRA paper reports that, compared with full fine-tuning of GPT-3 175B with Adam, LoRA cut trainable parameters by 10,000 times and GPU memory by 3 times, with no extra inference latency (the update can be merged into \(W_0\)). [Certain] QLoRA (Dettmers et al., 2023) stores the frozen weights in 4-bit NF4 and fine-tuned a 65B model on a single 48 GB GPU. [Certain] Figure 4 does the count for a Qwen2-7B-shaped LLM (28 layers, width 3,584, 28 query heads and 4 key-value heads of size 128, MLP width 18,944, from the released config [Certain]), which is the LLM size of many 7B Video-LLMs.

Figure 4. LoRA on a Qwen2-7B-shaped LLM (7.62B parameters, counted from the config). Memory counts weights, gradients and AdamW states only: 16 bytes per trainable parameter (bf16 weight and gradient, fp32 master copy, two fp32 Adam moments) and 2 bytes (bf16) or about 0.5 bytes (4-bit) per frozen parameter. Activations are not included; for video they are large (see the efficiency section).
How to read Figure 4

The top bar is full fine-tuning: \(7.62\text{B} \times 16\) bytes = 121.8 GB before activations, more than one 80 GB GPU. The lower bar is LoRA: mostly the frozen copy of the model (15.2 GB in bf16), plus a thin slice for the trainable adapters. With \(r = 16\) on all linear layers, LoRA trains 40.4M parameters (0.53% of the model) and needs 15.9 GB. Doubling \(r\) doubles the adapter slice but barely moves the total. Switch to 4-bit to see where the rest of the memory goes. The per-matrix line shows why the MLP matrices dominate: 18,944 is much wider than 3,584.

Worked example: LoRA parameter count for Qwen2-7B

Per layer, the shapes \((d_{\text{in}}, d_{\text{out}})\) are: q (3584, 3584), k (3584, 512), v (3584, 512), o (3584, 3584), gate and up (3584, 18944), down (18944, 3584). With \(r = 16\), formula (212.7) gives per layer: q and o \(16 \times 7{,}168 = 114{,}688\) each, k and v \(16 \times 4{,}096 = 65{,}536\) each, gate, up and down \(16 \times 22{,}528 = 360{,}448\) each. Sum: \(2 \times 114{,}688 + 2 \times 65{,}536 + 3 \times 360{,}448 = 1{,}441{,}792\). Times 28 layers: 40,370,176 parameters, 0.53% of 7,615,616,512. Only q and v: \(28 \times (114{,}688 + 65{,}536) = 5{,}046{,}272\) (0.066%).

Algorithm 3: domain fine-tuning a Video-LLM with LoRA (soccer)
  1. input: a general Video-LLM; domain clips with timestamped labels or commentary
  2. re-align and filter the domain text (MatchTime-style) // noisy times teach wrong times
  3. convert labels to the model’s own formats: chat template, time format, frame rate
  4. mix in general video and image data, for example by formula (212.4)
  5. freeze encoder, projector and LLM; add LoRA to the LLM linear layers (\(r\) = 8 to 64)
  6. optional: also train the projector, if domain visuals are unusual
  7. keep the sampler of the base model: same fps, same frame cap, same resolution
  8. train 1 to 3 epochs with a higher learning rate than full fine-tuning
  9. evaluate on held-out games (not held-out clips of the same games)
  10. also run general benchmarks to measure forgetting
  11. merge \(W_0 + \frac{\alpha}{r}BA\) for serving, or keep one adapter per sport

Two choices matter a lot for sports. First, resolution: a ball can be a few pixels wide in a broadcast frame, so a model that works on 224-pixel frames may not see it. Second, evaluation split: clips from the same game share teams, kits, stadium and light, so splitting by clip leaks information; split by game, or by season. [Likely] For more on LoRA variants see session 62 and, for VLMs, session 189.

Efficiency: fewer frames, lower resolution, token merging

For a video sample, the number of tokens \(T\) is set mostly by the visual part. From session 205, formula (205.3), with one more factor for token merging:

\[ T = \underbrace{\frac{K}{\tau} \cdot \frac{H}{p\,s} \cdot \frac{W}{p\,s} \cdot \rho}_{\text{visual tokens}} \;+\; T_{\text{text}} \tag{212.8} \]
Read the symbols
\(K\)
frames per sample
\(\tau\)
frames per temporal patch (1, or 2 in Qwen2-VL and Qwen2.5-VL)
\(H, W, p, s\)
frame size, patch size (14), and spatial merge (each \(s \times s\) patches become one token)
\(\rho\)
fraction of tokens kept after token merging or pruning (1 = keep all)
\(T_{\text{text}}\)
prompt and answer tokens; we use 300 in the figure

Training compute for one sample with \(T\) tokens, in the form of Kaplan et al. (2020): about 6 FLOPs per parameter per token for the matrix multiplies (2 forward, 4 backward), plus the attention term that grows with \(T^2\):

\[ C_{\text{train}} \approx 6\,N\,T \;+\; 6\,L\,T^{2}\,d \tag{212.9} \]
Read the symbols
\(N\)
non-embedding parameters of the LLM (6.53B for the Qwen2-7B shape)
\(L, d\)
number of layers (28) and model width (3,584)
\(6LT^2d\)
attention scores and weighted sums, forward and backward (Kaplan’s per-token \(2 L T d\), times \(T\) tokens, times 3)

This counts only the LLM; the vision encoder adds its own cost per frame. See session 25 for the derivation. Figure 5 lets you turn the four knobs.

\(6NT\) (matrix multiplies)\(6LT^2d\) (attention)
Figure 5. Tokens and LLM training compute per sample, by formulas (212.8) and (212.9), for a Qwen2-7B-shaped LLM with patch size 14. The baseline (64 frames, 448 pixels, \(2 \times 2\) merge, 2 frames per patch, all tokens kept) is the dashed line.
How to read Figure 5

At the baseline a sample is 8,492 tokens and 0.38 PFLOP, and attention is 11.6% of it. Halving the frames (32) or keeping half the tokens (\(\rho = 0.5\)) both give 4,396 tokens and 0.489 times the compute, a little better than half, because the \(T^2\) term shrinks by 4. Frames of 336 pixels give 0.55 times. Turn off both merges: 65,836 tokens, 13.8 times the compute, and attention becomes half of the cost. Large token counts hurt twice: more linear work, and much more attention work.

What each knob costs in quality:

  • Fewer frames lose short events and timing. Use few frames in stage 1 (alignment needs “what”, not “when”), and more in stages 2 and 3.
  • Lower resolution loses small objects and text. It is the wrong knob for sports balls and scoreboards; the right knob for scene-level captions.
  • Spatial merge (\(2 \times 2\) pooling or an MLP merger) is the standard first step; almost every Video-LLM uses some form of it.
  • Token merging (ToMe, Bolya et al., 2022) merges the most similar tokens inside the vision transformer, layer by layer. On video it gave 2.2 times the throughput of ViT-L with a 0.2–0.3% accuracy drop, without retraining, and up to 2 times faster MAE fine-tuning on video. [Certain] Video has many near-identical tokens across frames (a static background), so merging over time is especially effective; see session 206 and session 183.
  • Train short, then long. Qwen2.5-VL trained its last pretraining phase with sequences packed to 32,768 tokens with long videos, after earlier phases with shorter ones. [Certain] Short sequences early make the expensive \(T^2\) term small while the model learns the basics. [Likely]
Worked example: tokens and compute of one training sample

64 frames at \(448 \times 448\), \(\tau = 2\), \(p = 14\), \(s = 2\): \((64/2) \times (448/28)^2 = 32 \times 256 = 8{,}192\) visual tokens, plus 300 text tokens: \(T = 8{,}492\). Linear part: \(6 \times 6.526 \times 10^9 \times 8{,}492 = 3.33 \times 10^{14}\). Attention: \(6 \times 28 \times 8{,}492^2 \times 3{,}584 = 4.34 \times 10^{13}\). Total 0.38 PFLOP, 11.6% of it attention. With 128 frames, \(T = 16{,}684\) and the attention share rises to 20.4%.

Sequence packing

A training batch mixes text-only samples (a few hundred tokens), images (one to two thousand) and videos (several thousand to more than ten thousand). If we pad every sample in a micro-batch to the longest one, the GPU spends much of its time on padding tokens that carry no information. Packing puts several samples one after another into one row of fixed length \(L_{\max}\), so almost every position is a real token. We measure the share of real tokens:

\[ \eta = \frac{\sum_{i} \ell_i}{R \cdot L_{\text{row}}} \tag{212.10} \]
Read the symbols
\(\ell_i\)
length of sample \(i\) in tokens
\(R\)
number of rows computed (padded sequences, or packed rows)
\(L_{\text{row}}\)
length of each row: the longest sample in its micro-batch for padding, \(L_{\max}\) for packing
\(\eta\)
utilization: 1 means no padding at all

Packing has two rules that must not be broken (see session 18 and session 188):

  • Block-diagonal attention. A token may only attend to earlier tokens of its own sample. In practice this is done without building a mask, by giving the kernel the start offset of each sample (the cu_seqlens argument of FlashAttention’s variable-length function). [Certain]
  • Restarted positions. Position ids start again at 0 for each sample. With M-RoPE, each sample’s temporal, height and width ids are computed as if it were alone.
textimagevideopadding (wasted)
Figure 6. 32 samples with seeded random lengths: text 150 to 1,000 tokens, images 600 to 2,500, videos 2,000 to 16,000. Padding uses micro-batches of 4, each padded to its longest sample. Packing fills rows of 16,384 tokens with first-fit decreasing (Algorithm 4). The readout gives utilization by formula (212.10).
How to read Figure 6

At 50% video, random-order padding wastes about half the compute (utilization 50.5%), because almost every micro-batch has one long video that the short samples are padded up to. Grouping by length lifts it to 85.1%, but then each batch is all-video or all-text, which changes the gradient mix. Packing reaches 98.2% in 10 rows. With only text and images (0%), packing makes only 3 rows and the last is half empty (81.1%): packing works best when there are many samples per row.

Algorithm 4: first-fit-decreasing packing with a per-sample mask
  1. input: sample lengths \(\ell_1, \dots, \ell_n\) (each \(\le L_{\max}\)); row length \(L_{\max}\)
  2. sort samples by length, longest first
  3. for each sample \(i\):
  4. put it in the first row with at least \(\ell_i\) free tokens; if none, open a new row
  5. for each row: concatenate its samples; record start offsets cu_seqlens
  6. position ids restart at 0 at every offset // or M-RoPE ids per sample
  7. labels: answer tokens only; the first token of each sample has no previous-token target
  8. run attention with block-diagonal causal masking (varlen kernel)
  9. loss: average over all answer tokens in the batch, formula (212.5)

Code: time tokens, LoRA and packing in PyTorch

The code runs on a CPU in under a second and prints the same output every run. It checks three things from this session: the time-token target of formula (212.3), that a LoRA layer starts as the frozen layer and has \(r(d_{\text{in}} + d_{\text{out}})\) trainable parameters, and that packed attention with a block-diagonal mask gives exactly the same result as running each sample alone.

import torch, torch.nn.functional as F
torch.manual_seed(0)

# 1) Dense captions -> one target sequence with time tokens, formula (212.3)
def time_token(t, dur, n_bins=100):
    return f"<t{min(int(t * n_bins / dur), n_bins - 1)}>"
def dense_target(events, dur, n_bins=100):
    parts = [time_token(s, dur, n_bins) + time_token(e, dur, n_bins) + " " + txt
             for s, e, txt in sorted(events)]
    return " ".join(parts)
events = [(21.7, 26.2, "he shoots and scores."), (3.4, 21.7, "a player dribbles past two defenders.")]
print(dense_target(events, dur=150.0))

# 2) LoRA on a frozen linear layer, formulas (212.6) and (212.7)
class LoRALinear(torch.nn.Module):
    def __init__(self, base, r=8, alpha=16):
        super().__init__()
        self.base = base.requires_grad_(False)                    # frozen W0 (and bias)
        self.A = torch.nn.Parameter(torch.randn(r, base.in_features) / r**0.5)
        self.B = torch.nn.Parameter(torch.zeros(base.out_features, r))  # B = 0: starts as W0
        self.scale = alpha / r
    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T) @ self.B.T
base = torch.nn.Linear(3584, 512)                                 # a Qwen2-7B k_proj shape
lora = LoRALinear(base, r=16)
x = torch.randn(4, 3584)
print(torch.allclose(lora(x), base(x)),                           # True: same output at step 0
      sum(p.numel() for p in lora.parameters() if p.requires_grad))  # 16*(3584+512) = 65536

# 3) Packing: two samples in one row, block-diagonal causal mask, positions restart
def packed_mask(lengths):
    n = sum(lengths); m = torch.zeros(n, n, dtype=torch.bool); o = 0
    for L in lengths:
        m[o:o+L, o:o+L] = torch.tril(torch.ones(L, L, dtype=torch.bool)); o += L
    return m                                                      # True = may attend
def attend(q, k, v, mask):
    s = (q @ k.T) / q.shape[-1]**0.5
    return torch.softmax(s.masked_fill(~mask, float("-inf")), -1) @ v
lens = [5, 3]
q, k, v = (torch.randn(sum(lens), 8) for _ in range(3))
packed = attend(q, k, v, packed_mask(lens))
alone = torch.cat([attend(q[:5], k[:5], v[:5], packed_mask([5])),
                   attend(q[5:], k[5:], v[5:], packed_mask([3]))])
pos = torch.cat([torch.arange(L) for L in lens])                  # position ids restart per sample
print(torch.allclose(packed, alone, atol=1e-6), pos.tolist(),
      f"{sum(lens) / (2 * max(lens)):.3f}")                      # padding would use 2 rows of 5

Output (the same on every run):

<t2><t14> a player dribbles past two defenders. <t14><t17> he shoots and scores.
True 65536
True [0, 1, 2, 3, 4, 0, 1, 2] 0.800

Read it: the events are sorted by start time and each time becomes one token, as in the worked example. The LoRA layer gives exactly the base output at step 0 because \(B = 0\), and it trains \(16 \times (3{,}584 + 512) = 65{,}536\) parameters, while the frozen base has 1,835,520. Packing the 5-token and 3-token samples into one row gives the same attention output as running them alone, and uses 8 of 8 positions; padding would use 8 of 10 (utilization 0.8).

Where this is used

ApplicationTraining data that mattersEfficiency choice
Sports highlights and key-moment detectionTimestamped events (SoccerNet-v2), aligned commentary (MatchTime); domain LoRA (session 207)Dense frames around events, enough resolution for the ball and scoreboard
Video retrieval and Video-RAGLarge captioned sets (InternVid, Panda-70M) for contrastive training of the retriever (session 208)Few frames, small encoder; retriever runs on every clip
Dense captioning and chaptering of long videosDense, time-stamped captions; ASR boundaries as pseudo events (Vid2Seq)Time tokens or text seconds; packing of long samples
Instructional video assistantsASR-aligned narrated videos (HowTo100M) after filtering and re-alignmentASR text as extra input tokens, cheaper than more frames
Action recognition, zero-shotCaptions that describe motion, not only scenes (session 203)Keep temporal order; avoid heavy pooling over time
Content moderation, surveillance searchDomain data with rare events; strong mixing (low \(\alpha\)) so rare classes are seenLoRA per policy or per camera domain; low resolution is often enough
Video question answeringSynthetic QA (LLaVA-Video-178K) plus image instructions (session 210)Packing mixed image and video samples; short-then-long schedule

Common mistakes

  • Treating ASR as a caption. Speech is often before, after, or unrelated to the action (Figure 2). Use windows, filtering and re-alignment, and do not use raw ASR for the final stages.
  • Changing the time format between training and use. A model trained on <t17> tokens, or on “21.7s”, must be prompted and parsed in that format. The same holds for the frame rate and the frame cap.
  • Relative time tokens on long videos. With 100 bins, a 1-hour video has 36-second bins (18 s worst-case error). Use more bins, absolute seconds, or work on chunks.
  • Splitting domain data by clip. Clips from one game, one camera or one recipe video are near-duplicates. Split by game, source video, or date; otherwise the test score is inflated.
  • Fine-tuning only on domain data. A soccer-only LoRA can forget general skills and the chat format. Mix in general data and evaluate both.
  • Packing without the block-diagonal mask or with running positions. Samples then attend to each other, and positions (or M-RoPE time ids) of the second sample are wrong. The loss still goes down, so the bug is easy to miss.
  • Lowering resolution for a small-object task. It saves the most tokens and loses exactly what sports and OCR tasks need. Reduce frames or merge tokens first, and test on the hard cases.
  • Letting test videos leak into training. Web-scale datasets often contain benchmark videos or re-uploads. Deduplicate with video fingerprints before training.

Interview questions

What are the main kinds of video–text training data, and what is each good for?basic
(1) ASR-aligned: speech transcripts with times from narrated videos (HowTo100M: 136M clips). Huge and free, but weakly aligned with what is visible; good for early alignment and as pseudo time labels. (2) Captioned clips: web text (WebVid) or model-written captions (InternVid, 234M clips; Panda-70M, 70M clips). Closer to the pixels, cost compute; risk of hallucination and sameness. (3) Synthetic dense captions and QA from a strong model (ShareGPT4Video, LLaVA-Video-178K). Small but detailed, ordered and in instruction format; used in the last stages. Domain datasets (for example SoccerNet) come at the very end.
Why is ASR text a weak label, and how do you make it better?core
People talk before or after they act, explain things that are not visible, or talk about something else (“subscribe”). So the clip cut while a sentence is spoken often does not contain the action. Fixes: widen the window by a few seconds or merge sentences; filter pairs with a video–text similarity model; or train an alignment model that predicts whether and where a sentence is visible (TAN for HowTo100M, MatchTime for soccer commentary). Even then, ASR is better as pretraining signal than as final supervision.
How does Vid2Seq represent time, and what is the error of that representation?core
It adds \(N = 100\) time tokens to the vocabulary. A time \(t\) in a video of length \(T\) becomes bin \(\lfloor tN/T \rfloor\); each event in the target is start token, end token, sentence, sorted by start. Reading back at the bin center gives an error of at most \(T/(2N)\): 0.75 s for a 150 s clip, 18 s for one hour. Time tokens are cheap (2 per event) but relative, so precision falls with length. Vid2Seq pretrains on narrated videos using ASR sentence boundaries as pseudo events.
Time tokens or seconds written as text: which would you choose?advanced
Time tokens: 2 tokens per event, a fixed vocabulary, easy to parse, but relative (precision depends on length) and new embeddings must be learned. Text seconds: absolute, no new vocabulary, works with any LLM, and the model can copy times from a timestamp list in the prompt; but each number costs several tokens (about 5 to 6 for “21.7s” with digit-split tokenizers) and the model can output malformed numbers. For long or variable-length videos and chat use I would choose text seconds or absolute time ids (as Qwen2.5-VL does); for short clips with many events, time tokens are compact.
Describe a staged training recipe for a Video-LLM and say why the order matters.core
Start from an image VLM. Stage 1: train only the projector on large, noisy clip–text pairs with few frames, so video features land in the LLM’s input space without damaging the LLM. Stage 2: unfreeze the LLM on dense, time-stamped captions (and ASR pseudo events), with more frames, to learn order and time. Stage 3: instruction-tune on synthetic video QA mixed with image instructions, vision encoder at a small learning rate. Optional stage 4: domain LoRA. The order goes from large and noisy to small and clean, and from few trained parameters to many, so noisy data cannot harm the expensive skills and the cheap stages do most of the volume.
You need a soccer commentary model and have one 80 GB GPU. Plan it.advanced
Base: a 7B Video-LLM. Full fine-tuning needs about 122 GB for weights, gradients and AdamW states alone, so use LoRA: frozen bf16 base about 15.2 GB, adapters on all linear layers at \(r = 16\) are 40.4M parameters (0.65 GB with optimizer states). That leaves room for activations of 8K to 16K-token samples with activation checkpointing; QLoRA (4-bit base, about 3.8 GB) if more room is needed. Data: SoccerNet-Caption or MatchTime commentary, re-aligned; timestamps in the base model’s time format; mix in general data. Keep the base sampler; consider higher resolution for the ball. Evaluate on held-out games and on general benchmarks.
How many parameters does LoRA add, and which matrices dominate?core
Each adapted matrix adds \(r(d_{\text{in}} + d_{\text{out}})\). For a Qwen2-7B shape at \(r = 16\): q and o add 114,688 each per layer, k and v 65,536 (they are narrow because of GQA: 4 key-value heads of size 128, \(4 \times 128 = 512\)), and each MLP matrix 360,448, because the MLP width 18,944 is large. Over 28 layers: 5.0M for q, v only; 10.1M for q, k, v, o; 40.4M for all linear layers (0.53% of 7.62B). The MLP matrices are 75% of the all-linear count.
What are the four efficiency knobs for video training, and what does each cost?core
Fewer frames: linear savings in tokens; loses short events and timing. Lower resolution: quadratic savings per frame; loses small objects and text. Spatial merging (\(2 \times 2\) pooling or MLP merger, plus temporal patches of 2 frames): a fixed 4 to 8 times reduction with little loss, used almost everywhere. Token merging (ToMe) or pruning: merges similar tokens, very effective on video’s redundant backgrounds (2.2 times ViT-L throughput on video with 0.2–0.3% drop). Plus packing, which removes padding but does not change any sample. Because compute is \(6NT + 6LT^2d\), cutting \(T\) by half saves a bit more than half.
What is sequence packing and what can go wrong?core
Packing concatenates several samples into one fixed-length row so there is almost no padding (in Figure 6, utilization rises from about 50% to 98% for a mixed image–video batch). It requires a block-diagonal causal mask (or the varlen attention kernel with cu_seqlens) so samples do not see each other, position ids (or M-RoPE ids) restarted per sample, and labels that do not cross sample borders. Bugs here still let the loss go down, so they are found only by tests like the one in the code section. Packing also changes loss weighting between per-token and per-sample averaging.
How does Panda-70M make captions better than a single captioning model?advanced
It splits 3.8M long videos into semantically consistent clips, then runs several cross-modality teacher models that see different inputs (frames, subtitles, the video description). Each gives a candidate caption. People pick the best caption on a small subset, a retrieval model is fine-tuned on those choices, and it then picks the best candidate for all 70M clips. Many teachers reduce the style bias and blind spots of one captioner; the learned selector reduces hallucinated captions.
Your Video-LLM fine-tuned on sports gets great validation scores but fails on new matches. Why?advanced
Most likely leakage: validation clips came from the same games as training clips (same teams, kits, stadium, commentator), so the model memorized context rather than learning events. Also check: misaligned commentary times (the model learns the commentator’s delay), different broadcast style or resolution in new matches, a sampler or time format different from training, and class imbalance (goals are rare). Fix the split (by game or season), re-align labels, and evaluate on a truly held-out competition.

Exercises

  1. A narrated video has a sentence from 40 to 43 s; the action it describes is visible from 31 to 38 s. Compute the tIoU of the clip with \(\delta = 0\) and with \(\delta = 5\).
  2. Vid2Seq-style time tokens with \(N = 100\): a 20-minute video has an event from 312 s to 340 s. Which tokens are written, and what is the worst-case error?
  3. Using formula (212.7), how many LoRA parameters do q, k, v, o add at \(r = 64\) for the Qwen2-7B shape? What fraction of 7,615,616,512 is that?
  4. With formula (212.4) and sources of 1,600,000, 1,100,000 and 20,000 examples, find \(p\) for the smallest source at \(\alpha = 0\). How many times would each of its examples be seen in 1 million draws?
  5. Formula (212.8): 32 frames at \(336 \times 336\), \(\tau = 2\), \(s = 2\), \(\rho = 1\), 300 text tokens. How many tokens? If ToMe keeps \(\rho = 0.6\), how many visual tokens?
  6. Four samples of 9,000, 3,000, 2,000 and 2,000 tokens. Compute utilization with padding as one micro-batch of 4, and with packing into rows of 16,384.
Answerscheck
1. \(\delta = 0\): clip [40, 43], no overlap with [31, 38], tIoU 0. \(\delta = 5\): clip [35, 48]; overlap \([35, 38]\) = 3 s; union \([31, 48]\) = 17 s; \(\mathrm{tIoU} = 3/17 = 0.176\) (row 4 of Figure 2). 2. \(T = 1{,}200\) s, bins of 12 s. Start \(\lfloor 312 \times 100/1200 \rfloor = \lfloor 26 \rfloor = 26\); end \(\lfloor 28.33 \rfloor = 28\): <t26><t28>. Worst-case error \(T/(2N) = 6\) s; read-back times are 318 s and 342 s. 3. Per layer: \(64 \times (7{,}168 + 4{,}096 + 4{,}096 + 7{,}168) = 64 \times 22{,}528 = 1{,}441{,}792\); times 28 = 40,370,176, which is 0.53%, the same as all linear layers at \(r = 16\). 4. At \(\alpha = 0\) every source has \(p = 1/3\): 333,333 draws for 20,000 examples, so each is seen about 16.7 times: a strong risk of memorizing. 5. \(16 \times (336/28)^2 = 16 \times 144 = 2{,}304\) visual tokens, \(T = 2{,}604\). With \(\rho = 0.6\): \(\lfloor 1{,}382.4 \rfloor = 1{,}382\) visual tokens. 6. Total 16,000 real tokens. Padding: \(4 \times 9{,}000 = 36{,}000\) positions, utilization 44.4%. Packing: all four fit in one row of 16,384, utilization \(16{,}000/16{,}384 = 97.7\%\).

Key takeaways

  • Video–text data comes in three kinds: ASR-aligned (huge, weakly aligned), captioned (web or model-written, closer to pixels), and synthetic dense captions and QA (small, rich, ordered). Recipes go from large and noisy to small and clean.
  • ASR clips (212.1) often miss their action; measure with tIoU (212.2), and fix with windows, filtering and alignment models.
  • Dense captions with timestamps teach “when”. Time tokens (212.3) cost 2 tokens per event but their error grows with video length; text seconds are absolute but cost more tokens. Train and infer in the same format.
  • Mix sources with temperature sampling (212.4) and train with answer-only loss (212.5); evaluate images and video after each stage.
  • LoRA (212.6, 212.7) makes domain fine-tuning cheap: 40.4M trainable parameters (0.53%) for all linear layers of a 7B LLM at \(r = 16\), about 15.9 GB instead of 121.8 GB before activations.
  • Tokens (212.8) drive cost (212.9); frames, resolution, spatial merge and ToMe each cut tokens with a different quality cost. Packing (212.10) removes padding but needs a block-diagonal mask and restarted positions.

Key formulas

\[ c_i = [\max(0, s_i - \delta),\ \min(T, e_i + \delta)], \qquad b(t) = \min\!\left(\left\lfloor \tfrac{tN}{T} \right\rfloor, N-1\right), \qquad |\hat t - t| \le \tfrac{T}{2N} \]
\[ p_j = \frac{n_j^{\alpha}}{\sum_k n_k^{\alpha}}, \qquad W = W_0 + \tfrac{\alpha}{r} BA, \qquad P_{\text{LoRA}} = L \sum_{W \in S} r\,(d_{\text{in}} + d_{\text{out}}) \]
\[ T = \tfrac{K}{\tau}\cdot\tfrac{H}{ps}\cdot\tfrac{W}{ps}\cdot\rho + T_{\text{text}}, \qquad C_{\text{train}} \approx 6NT + 6LT^2 d, \qquad \eta = \frac{\sum_i \ell_i}{R\,L_{\text{row}}} \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  2. Zhang et al. (2024). Video Instruction Tuning With Synthetic Data
  3. Wang et al. (2023). InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
  4. Miech, Zhukov, Alayrac, Tapaswi, Laptev & Sivic (2019). HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
  5. Bain, Nagrani, Varol & Zisserman (2021). Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
  6. Chen et al. (2024). Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
  7. Chen et al. (2024). ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
  8. Han, Xie & Zisserman (2022). Temporal Alignment Networks for Long-term Video
  9. Yang et al. (2023). Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
  10. Ren et al. (2023). TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
  11. Deliège et al. (2021). SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos
  12. Mkhallati et al. (2023). SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries
  13. Rao et al. (2024). MatchTime: Towards Automatic Soccer Game Commentary Generation
  14. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  15. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
  16. Bolya et al. (2022). Token Merging: Your ViT But Faster
  17. Kaplan et al. (2020). Scaling Laws for Neural Language Models
  18. Bai et al. (2025). Qwen2.5-VL Technical Report
  19. Lin et al. (2023). Video-LLaVA: Learning United Visual Representation by Alignment Before Projection