LLM & VLM courseBlog
205Track 8: Video and multimodal systems · session 5 of 21

Video-LLMs

A Video-LLM is an image VLM that has learned to read a stack of frames and to know when each frame happened. This session builds one step by step: sample frames, encode each frame, merge tokens over space and time, and tell the language model the time of every token. We then follow the training data and three real systems: Video-LLaVA, LLaVA-Video and Qwen2.5-VL.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on the anatomy of a VLM, RoPE (2D and 3D), video as data and video encoders

What you will learn

  • The standard Video-LLM recipe: frame sampling, a per-frame image encoder, temporal merging, a projector and an LLM
  • How to count the visual tokens of a video for each design, and why the token budget forces a trade between frames and resolution
  • What spatial and temporal pooling keep and lose, including why mean pooling can erase the direction of motion
  • Three ways to tell the model about time: frame index, timestamps written as text, and absolute time in M-RoPE
  • How video instruction data is made (Video-ChatGPT, LLaVA-Video-178K) and why images and videos are trained together
  • The design choices of Video-LLaVA, LLaVA-Video and Qwen2.5-VL, with the numbers from their papers and released configs

Why this matters

A Video-LLM is a language model that can answer questions about a video: “What does the man do after he opens the fridge?”, “At what time does the goal happen?”, “Summarize this lecture.” Almost every Video-LLM today is built from an image VLM. The big question is how to turn minutes of video into a few thousand tokens without losing the things people ask about: what happened, in which order, and when.

In session 201 we learned that video is huge and that tokens are the scarce resource. In session 202 we saw networks that model motion directly. This session puts the pieces together into a working system. The recipe has four parts, and each part is a design choice with a cost:

  1. Frame sampling: which frames to keep. Too few, and short events fall between frames.
  2. A per-frame encoder: usually the same image encoder (CLIP, SigLIP or a ViT) that the image VLM already has.
  3. Temporal merging: pooling or merging tokens over space and time, so the video fits in the context window.
  4. The LLM, which reads the visual tokens together with the question and writes the answer. It must also be told when each token happened.

We will follow three real systems as case studies. Video-LLaVA (Lin et al., 2023) shows the simplest recipe: 8 frames, a frozen encoder, joint image and video training. LLaVA-Video (Zhang et al., 2024) shows how good synthetic video data changes the results, and how to put timestamps into the prompt as text. Qwen2.5-VL (Bai et al., 2025) shows dynamic frame rates and absolute time inside the position encoding. The book chapter on video-language models (Noyan et al., chapter 9) and the survey by Tang et al. give the wider picture. After this session, session 206 handles hours of footage, and session 207 builds on the time encodings you will learn here.

The recipe: four parts

Figure 1 shows the whole path. A video goes in on the left; the LLM sees an ordinary sequence of tokens on the right, where some tokens come from text and some from frames.

video: N frames 1. sample K frames uniform K, or f fpswith a frame cap 2. image encoder same weights for every frame K·(H/p)·(W/p) tokens 3. merge tokens space: 2×2 pool time: 2 frames → 1 4 to 8 times fewer projector 2-layer MLP to LLM width 4. LLM reads video + text writes the answer What the LLM sees (each box is a group of tokens; t = time of the frames) system, user framest = 0 s framest = 1 s framest = 2 s … framest = 59 s “When does he fall?” answer: “at 42 s” Time can enter in three ways: the order of the tokens (frame index), text such as “42.00s” between frames, or a temporal position id that grows with seconds, not with frames (absolute time in M-RoPE). Training only teaches the parts that are not frozen; the image encoder is often frozen.
Figure 1. The standard Video-LLM recipe. Steps 1 to 3 decide how many visual tokens the video costs. Step 4 decides whether the model can tell when something happened. Shaded boxes are visual tokens; the green box is the generated answer.

Step 1: choose the frames

The first choice is how many frames \(K\) to keep from a video with \(N\) frames, and which ones. There are two common rules.

Uniform K. Always keep the same number of frames, spread evenly over the video. Cut the video into \(K\) equal segments and take the middle frame of each:

\[ i_k = \left\lfloor \left(k + \tfrac12\right) \frac{N}{K} \right\rfloor, \qquad k = 0, 1, \dots, K-1 \tag{205.1} \]
Read the symbols
\(N\)
number of frames in the video (duration times the video’s frame rate)
\(K\)
number of frames we keep, for example 8, 32 or 64
\(i_k\)
the index of the \(k\)-th kept frame; \(\lfloor \cdot \rfloor\) means round down

Uniform K gives a fixed token cost, which makes batching easy. But the time between kept frames grows with the video: 8 frames from a 10-second clip are 1.25 s apart, 8 frames from a 10-minute video are 75 s apart. Video-LLaVA uses 8 frames. LLaVA-Video uses up to 64. [Certain]

Fixed fps with a cap. Keep \(f\) frames per second (Qwen2-VL and Qwen2.5-VL use 2 fps by default), but never more than \(F_{\max}\) frames. For a video of \(D\) seconds the real sampling rate is then

\[ f_{\text{eff}} = \min\!\left(f,\ \frac{F_{\max}}{D}\right) \tag{205.2} \]

Short videos get dense frames; long videos get as many frames as the cap allows. With \(f = 2\) and \(F_{\max} = 768\) (the cap Qwen2.5-VL used in its evaluations [Certain]), a 10-minute video gets \(f_{\text{eff}} = 768/600 = 1.28\) fps. This rule keeps short events visible in short clips. But now the time between frames depends on the video, and the model must be told what that time is. We return to this in Step 4.

Step 2: encode each frame

Most Video-LLMs do not use a special video network. They run the image encoder of an image VLM on every kept frame, with the same weights. This has three big advantages:

  • The image encoder is already aligned with language (CLIP or SigLIP pretraining), so its features are easy for the LLM to read.
  • The whole image VLM, with its projector and its image skills, is the starting point. Video training only has to add the time dimension.
  • Image and video tokens look the same to the LLM, so one model can be trained on both.

The price is that one frame alone has no motion. Motion must be found by the LLM, by comparing tokens of different frames through attention. Two small changes help:

  • A video encoder aligned with the image encoder. Video-LLaVA uses LanguageBind: a video encoder and an image encoder that were both trained to match the same text space before the projector is trained. The paper calls this “alignment before projection”. One shared projector then works for both. [Certain]
  • A temporal patch. Qwen2-VL and Qwen2.5-VL cut the video into blocks of 2 frames × 14 × 14 pixels, embedded by a 3D convolution. Each token then sees two consecutive frames, so it can carry a little motion, and the token count is halved. An image is treated as two identical frames, so images and videos share one patch embedder. [Certain]

For a frame of \(H \times W\) pixels and patch size \(p\), the encoder makes \((H/p)(W/p)\) tokens. A 384×384 frame in SigLIP-so400m with \(p = 14\) gives \(27 \times 27 = 729\) tokens; a 224×224 frame with \(p = 14\) gives \(16 \times 16 = 256\). Without merging, 64 frames of SigLIP tokens would be \(64 \times 729 = 46{,}656\) tokens. That is why Step 3 exists.

Step 3: merge tokens over space and time

The LLM is the expensive part, and its cost grows with the number of tokens (prefill compute grows a little faster than linearly, because of attention). So nearly every Video-LLM merges tokens before the LLM. A general count of visual tokens is

\[ T_{\text{video}} \;=\; \frac{K}{\tau} \cdot \frac{H}{p\,s} \cdot \frac{W}{p\,s} \tag{205.3} \]
Read the symbols
\(K\)
number of sampled frames
\(\tau\)
temporal merge: how many frames share one token (1 for most models, 2 for Qwen2-VL and Qwen2.5-VL)
\(H, W\)
frame height and width in pixels after resizing
\(p\)
patch size of the encoder (14 for CLIP ViT-L/14, SigLIP and the Qwen ViT)
\(s\)
spatial merge: each \(s \times s\) block of patches becomes one token (2 in LLaVA-Video and Qwen2.5-VL)

For Qwen2.5-VL, \(p\,s = 28\) and \(\tau = 2\), so each pair of 448×448 frames costs \((448/28)^2 = 256\) tokens. At 2 fps that is 256 tokens per second of video. [Certain]

Figure 2 runs this count in your browser, one temporal patch at a time, for a toy clip filmed at 30 fps: a ball crosses the frame from left to right. Each step applies one line of the pseudo-code, and the box next to it shows the formula with the real numbers of that step.

kept frame, patch gridcurrent pair, ball patchestubelet (2 frames deep)merged token, doneover the token cap
  1. \(K \leftarrow \min(\max(D f,\ 4),\ F_{\max})\), rounded down to even; \(f_s \leftarrow K/D\) (205.2)
  2. for \(k = 0, \dots, K-1\): \(i_k \leftarrow \lfloor (k + \tfrac12)\, N/K \rfloor\) (205.1)
  3. for \(j = 0, \dots, K/\tau - 1\): temporal patch \(j\) = frames \(2j,\ 2j+1\)
  4.   cut both frames into \(p \times p\) patches: \(2 \cdot (H/p) \cdot (W/p)\)
  5.   3D patch embed, \(\tau = 2\) frames deep: \((H/p) \cdot (W/p)\) tubelets
  6.   merge each \(2 \times 2\) group (MLP): \((H/ps) \cdot (W/ps)\) tokens
  7.   append to the context, \(\mathrm{id}_t = \mathrm{id}_0 + \lfloor j\, r\tau / f_s \rfloor\) (205.7)
  8. text after the video starts at the largest id \(+\,1\); total is (205.3)
Figure 2. The token pipeline of a Qwen2.5-VL style Video-LLM (\(p = 14\), \(\tau = 2\), \(s = 2\), \(r = 2\) ids per second, \(F_{\max} = 768\)), computed live. Top: the \(N\) frames of the video and the \(K\) kept ones. Middle: the current temporal patch: two frames with their 14-pixel patch grid, the tubelets (darker where the ball is in both frames, lighter where it is in only one, which is the motion), and the merged tokens. Bottom: the LLM context, with 15 text tokens before the video and the visual tokens of each finished patch.
How to read Figure 2

Press Play. The first steps take patch 0 through every line of the algorithm: sample, pick the frames, cut patches, merge over time (half the count), merge over space (a quarter again). After that each step appends one or more patches, and the context bar grows by the same amount each time. With 10 s at 2 fps and 448 px, 20 frames become 10 patches of 256 tokens: 2,560 tokens. Now move the sliders; the readout updates at once. Doubling the fps doubles the tokens; doubling the frame side multiplies them by 4. Set 10 minutes: the frame cap (768) is reached, the real rate drops to 1.28 fps, and the 98,304 tokens pass the 24,576 cap (red), so the frames would have to shrink to 224 px. Watch the temporal ids in the box: they always equal 2 × the patch start time in seconds.

Four ways to merge

  • Spatial pooling per frame. Average (or interpolate) each \(2 \times 2\) block of tokens. LLaVA-Video reduces SigLIP’s \(27 \times 27\) grid to \(14 \times 14 = 196\) tokens per frame with bilinear interpolation, and adds one “row end” token per row, so 210 tokens per frame in the released config. [Certain] Qwen2.5-VL concatenates each \(2 \times 2\) group of features and sends it through a 2-layer MLP, so nothing is thrown away before the MLP. [Certain]
  • Temporal merging. Merge neighboring frames into one token (Qwen’s temporal patch \(\tau = 2\)).
  • Global average over space and over time. Video-ChatGPT (Maaz et al., 2023) averages the frame features in two ways. Averaging over all patches of a frame gives one token per frame (\(K\) “temporal” tokens). Averaging each patch position over all frames gives one token per position (\(N_p\) “spatial” tokens):
\[ \mathbf{t}_i = \frac{1}{N_p} \sum_{j=1}^{N_p} \mathbf{x}_{i,j}, \qquad \mathbf{z}_j = \frac{1}{K} \sum_{i=1}^{K} \mathbf{x}_{i,j}, \qquad T_{\text{video}} = K + N_p \tag{205.4} \]
Read the symbols
\(\mathbf{x}_{i,j}\)
the feature of patch \(j\) in frame \(i\), from the image encoder
\(N_p\)
patches per frame, for example 256
\(\mathbf{t}_i\)
temporal token of frame \(i\): what the frame contains, with no “where”
\(\mathbf{z}_j\)
spatial token of position \(j\): what was at that place on average, with no “when”

This is extremely cheap: with 100 frames and 256 patches, 356 tokens instead of 25,600. [Likely] (The paper gives the formula; 100 frames is the setting in its code.) We will see in Figure 4 what it loses.

  • SlowFast. Borrowed from the SlowFast network for action recognition. Every \(s\)-th frame is a “slow” frame with light pooling (detail); the other frames are “fast” frames with heavy pooling (motion). LLaVA-Video’s 72B model uses it with \(s = 3\). [Certain] With \(M\) tokens per frame and pooling size \(p\) for slow frames, \(2p\) for fast ones:
\[ T_{\text{SlowFast}} = \left\lceil \tfrac{K}{s} \right\rceil \frac{M}{p^2} + \left(K - \left\lceil \tfrac{K}{s} \right\rceil\right) \frac{M}{(2p)^2} \tag{205.5} \]

With \(K = 64\), \(s = 3\) and SigLIP tokens rounded up as in the released code (196 per slow frame, \(7 \times 7 = 49\) per fast frame): \(22 \times 196 + 42 \times 49 = 6{,}370\) tokens, about half of \(64 \times 196 = 12{,}544\). (The paper’s own count uses slightly different rounding.)

Other systems use a learned compressor instead of pooling: Video-LLaMA puts a Q-Former over the frame features (see session 178). The idea is the same: a fixed, small number of tokens per frame or per clip.

Figure 3 lets you compare the four case-study designs on the same video.

kept framevisual tokenscontext sizes
Figure 3. The visual-token cost of one video under four designs. Top: where the kept frames fall on the timeline. Bottom: total visual tokens on a log scale, with 8K, 32K and 128K context sizes and Qwen2.5-VL’s evaluation cap of 24,576 video tokens.
How to read Figure 3

Start with Video-LLaVA and slide the video length from 1 minute to 1 hour: the token count stays at 2,048, but the gap between frames grows from 7.5 s to 450 s. Switch to Qwen2.5-VL: at 2 fps the token count grows with the video until the 768-frame cap. When the count passes the 24,576-token cap, the figure lowers the frame resolution (a simplified version of what the Qwen preprocessing does with its pixel budget) and the readout shows the new frame size. Video-ChatGPT is almost free, but read Figure 4 before you choose it.

Worked example: a 10-minute video in two designs

Video-LLaVA. 8 frames, \(16 \times 16 = 256\) tokens each: \(8 \times 256 = 2{,}048\) tokens. The gap between frames is \(600/8 = 75\) s. Any event shorter than about a minute may be invisible.

Qwen2.5-VL. At 2 fps the video has 1,200 candidate frames; the cap keeps 768, so \(f_{\text{eff}} = 1.28\) fps (formula 205.2) and the gap is 0.78 s. The 768 frames form 384 temporal patches. At 448×448 that would be \(384 \times 256 = 98{,}304\) tokens. The 24,576-token cap leaves \(24{,}576 / 384 = 64\) tokens per patch, an \(8 \times 8\) grid, which means frames of \(8 \times 28 = 224\) pixels per side.

So the two designs make opposite trades: Video-LLaVA keeps resolution and loses time; Qwen2.5-VL keeps time and loses resolution. Neither is free. For one hour, 768 frames at 64 tokens per pair is again exactly 24,576 tokens, with one frame every 4.69 s.

What pooling keeps and what it loses

Merging saves tokens, but each kind of merge throws away a different kind of information. Averaging over space keeps “what” but loses “where”. Averaging over time keeps “where” but loses “when”, and therefore the order of events. Figure 4 makes this concrete with a toy video: a ball moves across a 4×4 grid, and the question is “Does the ball move left or right?”.

Figure 4. A toy video of 8 frames, 16 patches each. A patch’s feature is 1 where the ball is and 0 elsewhere (so features carry no position of their own). The lower row shows the tokens the LLM would get after the chosen pooling. The readout compares the tokens of the two directions.
How to read Figure 4

Flip the direction and watch the lower row. With all patch tokens or a 2×2 pool, the two directions give different tokens (distance above 0), so the LLM can tell them apart. With a mean over space, every frame becomes the same token (the ball is always somewhere). With a mean over time, both directions leave the same trail. Video-ChatGPT’s tokens are the union of those two, so they are identical for both directions. In real models the features are richer and carry some position, so the loss is softer, but the lesson holds: heavy pooling makes order, direction and counting questions hard. [Likely]

Step 4: telling the model about time

After merging, the LLM has a long row of visual tokens. It still needs to know when each token happened. Without that, it can say “the man falls” but not “the man falls at 42 s”, and it cannot compare videos sampled at different rates. There are three ways to give time to the model.

Way 1: frame index (order only)

The simplest way: the tokens of frame 0 come first, then frame 1, and so on, and the normal position encoding of the LLM marks their order. In Qwen2-VL’s M-RoPE (multimodal RoPE, see session 14) every token has three position ids: temporal, height and width. For text the three ids are equal, so it is ordinary RoPE. For an image the temporal id is constant. For a video “the temporal ID increments for each frame”. [Certain] So the \(j\)-th temporal patch gets

\[ \mathrm{id}_t(j) = \mathrm{id}_0 + j \tag{205.6} \]

The problem: id 15 means second 15 if the video was sampled at 2 fps (one temporal patch of 2 frames per second), but second 60 at 0.5 fps. The model only knows the order, not the time. The Qwen2.5-VL report says exactly this: Qwen2-VL’s temporal ids were tied to the number of frames, which “ignored content speed and absolute timing”. [Certain]

Way 2: timestamps written as text

Write the time into the prompt. LLaVA-Video’s training code, when its time option is on (it is on in the released 7B config), puts this sentence before the question: “The video lasts for 60.00 seconds, and 32 frames are uniformly sampled from it. These frames are located at 0.94s,2.81s,4.69s,...” [Certain] Other models put a timestamp token next to each frame (TimeChat, for example, binds timestamps to frames). [Likely]

The model must then link “the 12th timestamp in the sentence” to “the 12th group of frame tokens”, which it learns from data. Text timestamps are cheap. With the Qwen2.5 tokenizer, every digit is a separate token, so “0.94s,” is 6 tokens and “12.50s,” is 7. For 64 frames of a 3-minute video the list costs 472 tokens, about 3.5% of the 13,440 visual tokens of LLaVA-Video. They also make the model’s answers natural: it can copy the time it reads.

Way 3: absolute time in the position encoding

Qwen2.5-VL changes formula (205.6) so that the temporal id grows with seconds, not with frames. The released config has tokens_per_second = 2 and temporal_patch_size = 2, and the transformers code spaces the temporal ids of a video by \(r \tau / f_s\): [Certain]

\[ \mathrm{id}_t(j) = \mathrm{id}_0 + \left\lfloor j \cdot \frac{r\,\tau}{f_s} \right\rfloor \;\approx\; \mathrm{id}_0 + r \cdot t_j \tag{205.7} \]
Read the symbols
\(j\)
index of the temporal patch (each patch holds \(\tau\) frames)
\(\tau\)
frames per temporal patch, 2
\(f_s\)
the real sampled frame rate of this video, for example 1.28 fps
\(r\)
temporal ids per second of video, 2 in the released Qwen2.5-VL config
\(t_j = j \tau / f_s\)
start time in seconds of patch \(j\)
\(\mathrm{id}_0\)
the position where the video starts in the sequence (after the text before it)

So one temporal id is always half a second, whatever the sampling rate. Why does this help attention? RoPE rotates each query and key pair by an angle proportional to its position, and the attention score depends only on the difference of positions:

\[ \big\langle R(m\theta)\,\mathbf{q},\ R(n\theta)\,\mathbf{k} \big\rangle = \big\langle \mathbf{q},\ R\big((n - m)\theta\big)\,\mathbf{k} \big\rangle \tag{205.8} \]

With formula (205.7), \(n - m \approx r\,(t_n - t_m)\): the angle measures the time between two tokens in seconds. A pattern such as “2 seconds apart” then looks the same to attention in a video sampled at 2 fps and in one sampled at 0.5 fps. In Qwen2.5-VL-7B, the 128-dimensional attention heads have 64 rotating pairs, split by mrope_section = [16, 24, 24]: 16 pairs rotate with the temporal id, 24 with the height id and 24 with the width id. [Certain] After the video, the text continues from the largest id used plus 1.

Figure 5 shows the three ways side by side for a 60-second video with an event at 40 s.

video A, 2 fpsvideo B, chosen fpsevent
  1. \(f_s \leftarrow K/D\); \(P \leftarrow K/\tau\) temporal patches
  2. for \(j = 0, \dots, P-1\) (A and B side by side in time):
  3.   \(t_j \leftarrow j\tau / f_s\) start time of the patch, seconds
  4.   frame index: \(\mathrm{id}_t \leftarrow \mathrm{id}_0 + j\) (205.6)
  5.   text: write the times of frames \(2j,\ 2j+1\) into the prompt; \(\mathrm{id}_t \leftarrow \mathrm{id}_0 + j\)
  6.   absolute time: \(\mathrm{id}_t \leftarrow \mathrm{id}_0 + \lfloor j\, r\tau / f_s \rfloor\) (205.7)
  7. event at \(e\) s \(\to\) patch \(\lfloor e f_s / \tau \rfloor\); compare its id in A and B
Figure 5. The same 60-second video sampled at 2 fps (A) and at a lower rate (B). Each dot is one temporal patch of 2 frames, drawn at its temporal id (or, for text timestamps, at its token order). The question is whether the id of the event is the same in A and B. Play walks through the patches of both videos in time order and computes each id with the chosen rule; the box shows the formula with the live values (\(\mathrm{id}_0\) is left as a symbol).
How to read Figure 5

Press Play: the dots appear patch by patch, both videos moving forward at the same clock time, and the box applies line 4, 5 or 6 of the pseudo-code to the newest patch. In frame-index mode the ids of B fall behind those of A as time runs; in absolute-time mode the two rows stay aligned. With frame index, the event gets id 40 in A but a smaller id in B: the same moment has two different positions, and the same id means two different times. With absolute time, both A and B put the event at about id 80 (2 ids per second), so time is consistent. With text timestamps the ids are just token order again, but the prompt contains the time of every frame; the readout counts how many text tokens that costs.

Worked example: temporal ids for a 10-minute video

From the first worked example, \(f_s = 1.28\) fps. Formula (205.7) gives a spacing of \(r\tau/f_s = 2 \times 2 / 1.28 = 3.125\) ids per temporal patch, so the first patches get offsets \(0, 3, 6, 9, 12, 15\) after rounding down. The patch that starts at 300 s is patch \(j = 300 \times 1.28 / 2 = 192\), with offset \(\lfloor 192 \times 3.125 \rfloor = 600 = 2 \times 300\). With frame index (205.6) the same patch would get offset 192, but at 2 fps the moment “300 s” would get offset 300. Only absolute time gives the same id for the same second.

Video instruction data and joint image-video training

Architecture is only half of a Video-LLM. The other half is data: videos paired with questions and answers that need the video to be answered. Such data is rare and expensive to label by hand, so most of it is made semi-automatically.

  • Video-ChatGPT built 100,000 video-instruction pairs, about 30% with human help (enriching ActivityNet captions) and 70% with a semi-automatic pipeline (image captioners, taggers and GPT-3.5). [Certain]
  • LLaVA-Video-178K has 178,510 videos of up to 3 minutes from ten sources (for example HD-VILA-100M, InternVid-10M, Charades, ActivityNet, Kinetics-700, Something-Something v2 and Ego4D), with 178K detailed captions, 960K open-ended questions and 196K multiple-choice questions, all written by GPT-4o. [Certain]

The LLaVA-Video captions are made by a recurrent, three-level process, because GPT-4o cannot take all frames at once. Frames are sampled at 1 fps. Every 10 seconds, GPT-4o writes a level-1 description from the current frames and the recent descriptions. Every 30 seconds it writes a level-2 summary from the last three level-1 descriptions. At the end it writes one level-3 description of the whole video. Questions are then generated from these captions. [Certain]

Why train on images and videos together?

Video data is smaller and noisier than image data. If a model is fine-tuned only on video, it can forget image skills such as reading text and fine detail, and it sees fewer kinds of visual concepts. Training on both keeps those skills, and the skills transfer from images to frames. Video-LLaVA tested this directly: compared with a version trained on video alone, joint training raised MSVD-QA accuracy from 64.8 to 70.7, MSRVTT-QA from 58.3 to 59.2, TGIF-QA from 67.8 to 70.0 and ActivityNet-QA from 40.7 to 45.3. [Certain] LLaVA-Video trained on about 1.6 million video samples together with 1.1 million image samples from LLaVA-OneVision. [Certain]

The training loss is the usual next-token loss of session 54, computed only on the answer tokens:

\[ \mathcal{L}(\theta) = -\sum_{t \in \mathcal{A}} \log p_\theta\!\left(y_t \mid y_{\lt t},\ \mathbf{V},\ \mathbf{q}\right) \tag{205.9} \]
Read the symbols
\(\mathbf{V}\)
the visual tokens of the video (or of an image, treated as a one-frame video)
\(\mathbf{q}\)
the question tokens, plus the time sentence if timestamps are given as text
\(\mathcal{A}\)
the positions of the answer tokens; prompt and visual tokens get no loss
\(y_t\)
the answer token at position \(t\)

Mixing images and videos in one batch is easy when every sample is “some visual tokens, then text”: an image is one frame (or two identical frames in Qwen), a video is many. Batches mix sequences of very different lengths, so packing and masking matter (session 188).

Algorithm 1: turning an image VLM into a Video-LLM (typical recipe)
  1. input: an image VLM (encoder, projector, LLM); image-text data; video caption data; video instruction data
  2. choose the sampler (uniform \(K\), or \(f\) fps with cap \(F_{\max}\)) and the merge (formula 205.3) so that a typical video fits the context
  3. choose the time encoding: frame index, text timestamps, or absolute time ids (formula 205.7)
  4. stage A (optional): train the projector only, on video captions, with encoder and LLM frozen // aligns video tokens to the LLM
  5. stage B: instruction-tune the projector and the LLM on a mix of image and video instructions, loss (205.9) on answers only
  6. in every batch, sample images and videos from the mix; treat an image as a one-frame (or two-identical-frame) video
  7. optionally unfreeze the encoder with a small learning rate // LLaVA-Video’s config uses 2e-6 for the vision tower
  8. evaluate on video benchmarks and image benchmarks, to catch forgetting (session 213)

Case studies: Video-LLaVA, LLaVA-Video, Qwen2.5-VL

Video-LLaVA (2023): the simple baseline

Video-LLaVA takes LLaVA-1.5 and adds video with the smallest possible change. The encoders are LanguageBind’s image encoder (initialized from OpenCLIP-L/14) and LanguageBind’s video encoder, both at 224×224 and both frozen. A video is 8 uniformly sampled frames. One shared projector (two linear layers with GELU) maps both images and videos into Vicuna-7B v1.5. Stage 1 trains the projector on 558K image-text pairs and 702K video-text pairs; stage 2 trains the projector and LLM on 665K image instructions and 100K video instructions (from Video-ChatGPT). [Certain] With 256 patch tokens per frame, a video costs 2,048 tokens. [Likely]

LLaVA-Video (2024): better data, more frames, timestamps as text

LLaVA-Video starts from the LLaVA-OneVision single-image checkpoint: SigLIP-so400m at 384×384 and a Qwen2 LLM. Video frames are pooled from \(27 \times 27\) to \(14 \times 14\) tokens, and up to 64 frames are used in the final setting (ablations went to 110). The main contribution is the LLaVA-Video-178K dataset and its recurrent captioning. Time enters as text: the prompt lists the duration and the time of each sampled frame. The 72B model uses the SlowFast representation (\(s = 3\)) to fit more frames. [Certain]

Qwen2.5-VL (2025): dynamic fps and absolute time

Qwen2.5-VL uses one native-resolution ViT for images and videos, with 14×14 patches, temporal patches of 2 frames, and a 2×2 MLP merger. To keep cost linear in image size, most ViT layers use window attention with windows of at most 112×112 pixels (8×8 patches); only 4 of the 32 layers (indices 7, 15, 23, 31) use full attention. [Certain] During training the sampling fps is varied, so the model sees many frame rates, and temporal ids follow absolute time (formula 205.7). Training data for grounding writes times both in seconds and in an hour-minute-second-frame format, and the last pretraining phase packs sequences of 32,768 tokens with long videos. In evaluation, videos were capped at 768 frames and 24,576 video tokens. [Certain]

Video-LLaVALLaVA-Video (7B)Qwen2.5-VL
Frame encoderLanguageBind image and video encoders (ViT-L/14, 224 px), frozenSigLIP-so400m, 384 pxown ViT, native resolution, window attention
Sampling8 frames, uniformup to 64 frames, uniform2 fps by default, cap 768 frames in evaluation
Mergingnone (256 tokens per frame)2×2 bilinear: 196 per frame (+14 row tokens)2 frames × 2×2 patches → 1 token (MLP)
Timeframe order onlytimestamps as text in the promptabsolute time in M-RoPE (2 ids per second)
LLMVicuna-7B v1.5Qwen2Qwen2.5
Video data702K video-text + 100K instructionsLLaVA-Video-178K plus public QA sets, about 1.6M sampleslarge mixed corpus, long-video captions, grounding with timestamps
Images too?yes (558K + 665K)yes (1.1M from LLaVA-OneVision)yes

Code: build the inputs of a Video-LLM in PyTorch

Algorithm 2: from a video file to the LLM input (Qwen2.5-VL style)
  1. input: video with \(N\) frames at rate \(f_v\); target fps \(f\); cap \(F_{\max}\); text before and after the video
  2. \(D \leftarrow N / f_v\); \(K \leftarrow \min(\max(D f, 4), F_{\max}, N)\), rounded down to an even number
  3. pick frame indices with formula (205.1); \(f_s \leftarrow K / D\) // real sampled fps
  4. decode and resize the frames to \(H \times W\), multiples of 28
  5. cut into patches of \(2 \times 14 \times 14\) pixels; embed; run the ViT
  6. merge each \(2 \times 2\) group with the MLP: \((K/2) \cdot (H/28) \cdot (W/28)\) tokens (formula 205.3)
  7. position ids: text before the video gets \(0, 1, \dots, \mathrm{id}_0 - 1\) on all three axes
  8. video tokens: temporal id by formula (205.7), height and width ids from the grid, all offset by \(\mathrm{id}_0\)
  9. text after the video continues at (largest id used) + 1
  10. run the LLM with M-RoPE on the joined sequence

The code below runs on a CPU in under a second. It uses a tiny random “ViT” (a single linear layer), so the numbers are about shapes, token counts and time ids, not about quality. It checks two facts from this session: absolute-time ids match seconds for two sampling rates, and Video-ChatGPT pooling cannot see the order of frames.

import torch
torch.manual_seed(0)

# 1) Frame sampling: a target fps with a frame cap, formula (205.1)
def sample_frames(n_frames, video_fps, target_fps=2.0, max_frames=768, tp=2):
    dur = n_frames / video_fps                        # seconds
    k = min(max(dur * target_fps, 4), max_frames, n_frames)
    k = int(k // tp * tp)                             # even: frames pair into temporal patches
    idx = ((torch.arange(k) + 0.5) * n_frames / k).long()   # centre of each of k equal segments
    return idx, k / dur                               # frame indices, real sampled fps

# 2) Toy visual side: 3D patch embed (2 frames x 14 x 14), then 2x2 merge + MLP
class ToyVisual(torch.nn.Module):
    def __init__(self, d_vit=64, d_llm=96, p=14, tp=2, m=2):
        super().__init__()
        self.p, self.tp, self.m = p, tp, m
        self.embed = torch.nn.Linear(3 * tp * p * p, d_vit)     # same as Conv3d, stride (tp, p, p)
        self.merge = torch.nn.Sequential(torch.nn.Linear(d_vit * m * m, d_llm),
                                         torch.nn.GELU(), torch.nn.Linear(d_llm, d_llm))
    def forward(self, video):                                    # video: (K, 3, H, W)
        K, C, H, W = video.shape
        p, tp, m = self.p, self.tp, self.m
        t, h, w = K // tp, H // p, W // p
        x = video.reshape(t, tp, C, h, p, w, p).permute(0, 3, 5, 1, 2, 4, 6)
        x = self.embed(x.reshape(t, h, w, tp * C * p * p))       # (t, h, w, d_vit); ViT blocks go here
        x = x.reshape(t, h // m, m, w // m, m, -1).permute(0, 1, 3, 2, 4, 5)
        return self.merge(x.reshape(t, h // m, w // m, -1))      # (t, h/2, w/2, d_llm)

# 3) M-RoPE ids with absolute time, formula (205.7)
def mrope_ids(start, t, h, w, fps, tp=2, tokens_per_second=2):
    step = tokens_per_second * tp / fps                          # temporal ids per temporal patch
    T = (torch.arange(t) * step).long()
    tt, hh, ww = torch.meshgrid(T, torch.arange(h), torch.arange(w), indexing="ij")
    return torch.stack([tt, hh, ww]).reshape(3, -1) + start      # (3, t*h*w)

vis = ToyVisual()
for target in (2.0, 0.5):                                        # same 30 s clip, two sampling rates
    idx, fps = sample_frames(n_frames=900, video_fps=30, target_fps=target)
    video = torch.rand(len(idx), 3, 112, 112)                    # stand-in for decoded frames
    tok = vis(video)
    t, h, w, _ = tok.shape
    ids = mrope_ids(start=10, t=t, h=h, w=w, fps=fps)            # 10 text tokens come first
    last_t = (ids[0].max() - 10).item()
    true_t = idx[-2].item() / 30                                 # first frame of the last patch
    print(f"fps {fps:.3f}: frames {len(idx)}, tokens {t*h*w}, last temporal id {last_t}"
          f" -> {last_t / 2:.1f} s (true {true_t:.1f} s)")
    nxt = ids.max() + 1                                          # text after the video continues here

# 4) Video-ChatGPT style pooling: (T + N) tokens, and it ignores frame order
f = torch.randn(8, 256, 32)                                      # T=8 frames, N=256 patches, D=32
def vcg_pool(f):
    return torch.cat([f.mean(1), f.mean(0)])                     # (T + N, D), formula (205.4)
a, b = vcg_pool(f), vcg_pool(f.flip(0))
print(a.shape[0],                                                # 264 = 8 + 256 tokens
      torch.allclose(a[8:], b[8:], atol=1e-6),                   # True: spatial part forgets order
      torch.allclose(a[:8], b[:8].flip(0), atol=1e-6))           # True: temporal part is just reversed

Output (the same on every run):

fps 2.000: frames 60, tokens 480, last temporal id 58 -> 29.0 s (true 29.2 s)
fps 0.467: frames 14, tokens 112, last temporal id 51 -> 25.5 s (true 26.8 s)
264 True True

Read it: at 2 fps the 30-second clip gives 60 frames, 30 temporal patches of \(4 \times 4 = 16\) tokens each, so 480 tokens. At 0.5 fps the clip gives 15 frames, rounded down to 14 (an even number), so the real rate is 0.467 fps. In both cases the temporal id divided by 2 is a time in seconds. The id marks the start of a patch’s time slot, so it can be up to one patch length (\(\tau / f_s\), here 4.3 s) earlier than the frames inside it. The last line shows that the 256 spatial tokens of Video-ChatGPT are identical when the video is played backwards, and the 8 temporal tokens are just reversed.

Where this is used

ApplicationWhat the Video-LLM doesDesign choices that matter
Video question answeringAnswers free-form questions about a clipFrames vs resolution; joint image-video training for detail (session 210)
Key-moment detection and temporal groundingReturns start and end times of an event (“the goal is at 42 to 47 s”)Dense sampling and an explicit time encoding; training data with timestamps (session 207)
Dense captioning and summariesWrites a time-stamped description of a whole videoRecurrent or hierarchical captions, as in LLaVA-Video-178K; long context
Action recognition, zero-shotNames the action from a list in the promptTemporal order must survive pooling (Figure 4); compare with trained classifiers (session 203)
Video retrieval and re-rankingChecks whether a retrieved clip really matches a text queryCheap token budget per candidate; a contrastive model retrieves first (session 208)
Video-RAG over large archivesAnswers questions using clips found by searchShort clips per answer; timestamps so answers can cite the moment
Content moderation and sports analyticsFlags or describes events in streamsFixed fps sampling so short events are not missed; latency budget (session 211)

Common mistakes

  • Too few frames for “when” questions. 8 uniform frames from a 10-minute video are 75 s apart. No model can localize an event it never saw.
  • Sampling at a different rate than in training. A model trained with 8 frames and run with 64 may get worse, not better, especially with frame-index time. Check the training setting, or use a model trained with many rates (dynamic fps).
  • Frame-index time with variable fps. If the temporal id counts frames, then id 15 means different seconds in different videos. Either fix the fps, give timestamps as text, or use absolute-time ids.
  • Wrong fps passed to the processor. With absolute time, the temporal spacing uses the sampled fps (formula 205.7). If your own sampler passes the video’s native 30 fps instead of the sampled rate, every id is wrong by a factor of 15 at 2 fps, and the model reports wrong times.
  • Pooling away order and counting. Mean pooling over time (Figure 4) cannot tell “opens the door” from “closes the door”. Keep per-frame tokens, even if fewer of them.
  • Training only on video. Image skills such as OCR and fine detail fade. Keep images in the mix and evaluate both.
  • Ignoring the token count of text timestamps or “row end” tokens. They are small (a few percent), but they count toward the context and the cap.
  • Forgetting decode cost. The LLM may be fast, but decoding 768 frames from a long video can take longer than the model (session 201).

Interview questions

Describe the architecture of a typical Video-LLM.basic
Start from an image VLM. Sample \(K\) frames (uniform \(K\) or a fixed fps with a cap). Run the image encoder on each frame with shared weights. Merge tokens over space (2×2 pooling or an MLP merger) and sometimes over time (temporal patches of 2 frames). Project to the LLM width with an MLP, put the visual tokens before the question, and train with next-token loss on answers. Add a time encoding: frame order, timestamps as text, or absolute-time position ids.
How many visual tokens does one minute of video cost in Qwen2.5-VL at 2 fps and 448×448?core
120 frames form 60 temporal patches. Each patch is \((448/28)^2 = 256\) tokens after the 14-pixel patches and the 2×2 merge. Total \(60 \times 256 = 15{,}360\) tokens, or 256 tokens per second. A 10-minute video at the same setting would be 153,600 tokens, which is why there is a frame cap and a token cap.
Uniform K frames or fixed fps: when would you choose each?core
Uniform \(K\) gives a constant token cost and easy batching; it is fine for short clips and “what happens” questions. Its gap between frames grows with the length, so it misses short events in long videos. Fixed fps keeps the gap constant (good for timing and short events) but the cost grows with length, so you need a cap, and then the real rate changes per video. That is why fixed-fps models need a time encoding that knows the real rate.
Why can mean pooling over time hurt a Video-LLM?core
The average over frames is the same for any order of the frames, so it cannot express direction (“left to right” vs “right to left”), order (“before” vs “after”) or how many times something happened. Video-ChatGPT’s spatial tokens are exactly such averages; its temporal tokens keep order but average away all spatial layout. Keeping per-frame tokens (with spatial pooling only) keeps order at a higher token cost.
What changed between Qwen2-VL and Qwen2.5-VL in how time is encoded?advanced
In Qwen2-VL’s M-RoPE the temporal id grew by one per temporal patch, so it counted frames. In Qwen2.5-VL the temporal id is spaced by \(r\tau/f_s\) (with \(r = 2\) ids per second, \(\tau = 2\) frames per patch, \(f_s\) the sampled fps), so it measures absolute time: one id is half a second. Because RoPE scores depend on position differences, attention now sees time differences in seconds, consistent across videos with different sampling rates. Training with varied fps teaches the model to use this.
What are the pros and cons of putting timestamps into the prompt as text?core
Pros: it works with any LLM and any position encoding, the model can copy times into its answer, and it is cheap (about 6 to 8 tokens per frame with a tokenizer that splits digits). Cons: the model must learn to link the \(k\)-th timestamp in the sentence to the \(k\)-th group of frame tokens, which can fail for many frames; the times live far from the frame tokens in the sequence; and it adds tokens that grow with the number of frames.
Why train Video-LLMs on images and videos together?core
Video instruction data is smaller and noisier than image data, so video-only fine-tuning can make the model forget image skills (OCR, fine detail, many concepts). A shared token format lets one batch mix both, and skills learned on images transfer to frames. Video-LLaVA measured it: joint training raised accuracy on all four video QA sets it tested, for example ActivityNet-QA from 40.7 to 45.3 and MSVD-QA from 64.8 to 70.7.
How was LLaVA-Video-178K built, and why not just use human labels?core
178,510 videos of up to 3 minutes were selected from ten sources with filters on scenes, length and resolution. GPT-4o wrote captions recurrently at three levels (every 10 s, every 30 s, and once for the whole video) from 1-fps frames, then generated 960K open-ended and 196K multiple-choice questions. Human labels for detailed, dense, multi-level captions and many questions per video would be far too expensive at this scale; synthetic data trades some noise for coverage. Evaluate on human-labeled benchmarks to check it.
Your Video-LLM describes events well but gives wrong times. What do you check?advanced
(1) Sampling: is the gap between frames smaller than the events? (2) Time encoding: with frame-index time, are you using the same fps as in training? With absolute time, is the processor given the sampled fps (not the native fps)? With text timestamps, are they in the prompt and in the same format as in training? (3) Data: did the model see grounding data with timestamps? (4) Evaluate on a temporal-grounding benchmark, and test the same video at two fps values: answers in seconds should not change.
You must answer questions on a 1-hour video with a 24,576-token video budget in Qwen2.5-VL. What are your options?advanced
At the minimum of 64 tokens per 2-frame patch (224×224 frames), 24,576 tokens allow 384 patches = 768 frames, one every 4.69 s. Options: accept coarse sampling; lower resolution further for more frames; select frames by content (scene changes, query relevance) instead of uniformly; split the video into chunks, caption or index each, and retrieve the relevant chunks for the question (Video-RAG); or use a model with token compression for long video. The choice depends on whether the question needs detail, timing, or global summary.
What does “alignment before projection” mean in Video-LLaVA?basic
The image encoder and the video encoder (from LanguageBind) were both pretrained to map their inputs into the same language-aligned space. So their outputs are already compatible before any projector is trained, and one shared projector can serve both modalities. This makes joint image-video training simpler and, according to the paper, helps both.

Exercises

  1. Video-LLaVA samples 8 frames from a 2-minute clip. How far apart are the frames, and how many visual tokens does the clip cost?
  2. Qwen2.5-VL samples a 20-second clip at 1 fps with frames of 336×336. How many visual tokens? How many temporal ids apart are two neighboring temporal patches, and what is the offset of the last one?
  3. Use formula (205.5) with \(K = 32\), \(s = 4\), 196 tokens per slow frame and 49 per fast frame. Compare with all 32 frames slow.
  4. In Figure 4, explain why the 2×2 pool still tells the two directions apart, while the mean over space does not.
  5. A prompt lists the times of 32 uniformly sampled frames of a 60-second video in the format “0.94s,”. Using the rule “digits of the integer part + 5 tokens” (digits, the point, two decimals, “s” and the comma are separate tokens), how many tokens does the list cost?
  6. A user runs Qwen2.5-VL with their own sampler at 2 fps but passes the native 30 fps to the processor. By what factor are the temporal ids wrong, and will the model’s answers in seconds be too early or too late?
Answerscheck
1. \(120/8 = 15\) s apart; \(8 \times 256 = 2{,}048\) tokens. 2. 20 frames → 10 temporal patches of \((336/28)^2 = 144\) tokens: 1,440 tokens. Spacing \(r\tau/f_s = 2 \times 2/1 = 4\) ids; the last patch (\(j = 9\)) has offset 36, which is \(2 \times 18\) s, its start time. 3. \(\lceil 32/4 \rceil = 8\) slow frames: \(8 \times 196 + 24 \times 49 = 1{,}568 + 1{,}176 = 2{,}744\) tokens, versus \(32 \times 196 = 6{,}272\): 2.3 times fewer. 4. The 2×2 pool keeps a coarse 2×2 layout in each frame, so it still knows whether the ball is in the left or right half, and frame order is kept; in the toy, the distance between the two directions is 1.0. The mean over space has one number per frame, and every frame has exactly one ball in 16 cells, so every token is 1/16 for both directions. 5. Frame times are 0.94, 2.81, …; 5 of them are below 10 s (6 tokens each) and 27 are between 10 and 60 s (7 tokens each): \(30 + 189 = 219\) tokens. 6. The spacing uses \(r\tau/f_s\): with 30 instead of 2, ids are 15 times too close together. The model reads the video as 15 times shorter, so times it reports in seconds are too early (too small) by a factor of about 15.

Key takeaways

  • A Video-LLM is an image VLM plus four choices: frame sampling (205.1, 205.2), the per-frame encoder, token merging over space and time (205.3 to 205.5), and a time encoding.
  • The token budget forces a trade: few frames at high resolution (Video-LLaVA, 2,048 tokens for 8 frames) or many frames at low resolution (Qwen2.5-VL, up to 768 frames in 24,576 tokens).
  • Mean pooling over time or space can erase order and direction; keep per-frame tokens when the task is about events.
  • Time can be given as frame order (205.6), as text timestamps (cheap, about 6 to 8 tokens per frame), or as absolute time in M-RoPE (205.7), where one temporal id is half a second and RoPE (205.8) sees time differences in seconds.
  • Data is half the system: synthetic, multi-level video captions and QA (LLaVA-Video-178K), and joint image-video training (205.9), which improved every video QA set in Video-LLaVA’s ablation.

Key formulas

\[ i_k = \left\lfloor \left(k + \tfrac12\right)\tfrac{N}{K} \right\rfloor, \qquad f_{\text{eff}} = \min\!\left(f, \tfrac{F_{\max}}{D}\right), \qquad T_{\text{video}} = \frac{K}{\tau}\cdot\frac{H}{ps}\cdot\frac{W}{ps} \]
\[ \mathrm{id}_t(j) = \mathrm{id}_0 + \left\lfloor j \frac{r\tau}{f_s} \right\rfloor \approx \mathrm{id}_0 + r\,t_j, \qquad \mathcal{L} = -\sum_{t \in \mathcal{A}} \log p_\theta(y_t \mid y_{\lt t}, \mathbf{V}, \mathbf{q}) \]

References for this session

  1. Lin et al. (2023). Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
  2. Zhang et al. (2024). Video Instruction Tuning With Synthetic Data
  3. Bai et al. (2025). Qwen2.5-VL Technical Report
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  5. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  6. Lin et al. (2023). Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
  7. Zhang et al. (2024). Video Instruction Tuning With Synthetic Data
  8. Bai et al. (2025). Qwen2.5-VL Technical Report
  9. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  10. Tang et al. (2023). Video Understanding with Large Language Models: A Survey
  11. Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
  12. Feichtenhofer et al. (2018). SlowFast Networks for Video Recognition
  13. Ren et al. (2023). TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
  14. Maaz, Rasheed, Khan & Khan (2023). Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
  15. Zhu et al. (2023). LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
  16. Zhang, Li & Bing (2023). Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
  17. Li et al. (2024). LLaVA-OneVision: Easy Visual Task Transfer
  18. Hugging Face and Qwen team (2025). Qwen2.5-VL in transformers: config.json (tokens_per_second, temporal_patch_size) and get_rope_index
  19. LLaVA-VL team (2024). LLaVA-NeXT code base: llava_arch.py (video pooling) and train.py (time instruction)