Video-LLMs
A Video-LLM is an image VLM that has learned to read a stack of frames and to know when each frame happened. This session builds one step by step: sample frames, encode each frame, merge tokens over space and time, and tell the language model the time of every token. We then follow the training data and three real systems: Video-LLaVA, LLaVA-Video and Qwen2.5-VL.
What you will learn
- The standard Video-LLM recipe: frame sampling, a per-frame image encoder, temporal merging, a projector and an LLM
- How to count the visual tokens of a video for each design, and why the token budget forces a trade between frames and resolution
- What spatial and temporal pooling keep and lose, including why mean pooling can erase the direction of motion
- Three ways to tell the model about time: frame index, timestamps written as text, and absolute time in M-RoPE
- How video instruction data is made (Video-ChatGPT, LLaVA-Video-178K) and why images and videos are trained together
- The design choices of Video-LLaVA, LLaVA-Video and Qwen2.5-VL, with the numbers from their papers and released configs
Why this matters
A Video-LLM is a language model that can answer questions about a video: “What does the man do after he opens the fridge?”, “At what time does the goal happen?”, “Summarize this lecture.” Almost every Video-LLM today is built from an image VLM. The big question is how to turn minutes of video into a few thousand tokens without losing the things people ask about: what happened, in which order, and when.
In session 201 we learned that video is huge and that tokens are the scarce resource. In session 202 we saw networks that model motion directly. This session puts the pieces together into a working system. The recipe has four parts, and each part is a design choice with a cost:
- Frame sampling: which frames to keep. Too few, and short events fall between frames.
- A per-frame encoder: usually the same image encoder (CLIP, SigLIP or a ViT) that the image VLM already has.
- Temporal merging: pooling or merging tokens over space and time, so the video fits in the context window.
- The LLM, which reads the visual tokens together with the question and writes the answer. It must also be told when each token happened.
We will follow three real systems as case studies. Video-LLaVA (Lin et al., 2023) shows the simplest recipe: 8 frames, a frozen encoder, joint image and video training. LLaVA-Video (Zhang et al., 2024) shows how good synthetic video data changes the results, and how to put timestamps into the prompt as text. Qwen2.5-VL (Bai et al., 2025) shows dynamic frame rates and absolute time inside the position encoding. The book chapter on video-language models (Noyan et al., chapter 9) and the survey by Tang et al. give the wider picture. After this session, session 206 handles hours of footage, and session 207 builds on the time encodings you will learn here.
The recipe: four parts
Figure 1 shows the whole path. A video goes in on the left; the LLM sees an ordinary sequence of tokens on the right, where some tokens come from text and some from frames.
Step 1: choose the frames
The first choice is how many frames \(K\) to keep from a video with \(N\) frames, and which ones. There are two common rules.
Uniform K. Always keep the same number of frames, spread evenly over the video. Cut the video into \(K\) equal segments and take the middle frame of each:
- Read the symbols
- \(N\)
- number of frames in the video (duration times the video’s frame rate)
- \(K\)
- number of frames we keep, for example 8, 32 or 64
- \(i_k\)
- the index of the \(k\)-th kept frame; \(\lfloor \cdot \rfloor\) means round down
Uniform K gives a fixed token cost, which makes batching easy. But the time between kept frames grows with the video: 8 frames from a 10-second clip are 1.25 s apart, 8 frames from a 10-minute video are 75 s apart. Video-LLaVA uses 8 frames. LLaVA-Video uses up to 64. [Certain]
Fixed fps with a cap. Keep \(f\) frames per second (Qwen2-VL and Qwen2.5-VL use 2 fps by default), but never more than \(F_{\max}\) frames. For a video of \(D\) seconds the real sampling rate is then
Short videos get dense frames; long videos get as many frames as the cap allows. With \(f = 2\) and \(F_{\max} = 768\) (the cap Qwen2.5-VL used in its evaluations [Certain]), a 10-minute video gets \(f_{\text{eff}} = 768/600 = 1.28\) fps. This rule keeps short events visible in short clips. But now the time between frames depends on the video, and the model must be told what that time is. We return to this in Step 4.
Step 2: encode each frame
Most Video-LLMs do not use a special video network. They run the image encoder of an image VLM on every kept frame, with the same weights. This has three big advantages:
- The image encoder is already aligned with language (CLIP or SigLIP pretraining), so its features are easy for the LLM to read.
- The whole image VLM, with its projector and its image skills, is the starting point. Video training only has to add the time dimension.
- Image and video tokens look the same to the LLM, so one model can be trained on both.
The price is that one frame alone has no motion. Motion must be found by the LLM, by comparing tokens of different frames through attention. Two small changes help:
- A video encoder aligned with the image encoder. Video-LLaVA uses LanguageBind: a video encoder and an image encoder that were both trained to match the same text space before the projector is trained. The paper calls this “alignment before projection”. One shared projector then works for both. [Certain]
- A temporal patch. Qwen2-VL and Qwen2.5-VL cut the video into blocks of 2 frames × 14 × 14 pixels, embedded by a 3D convolution. Each token then sees two consecutive frames, so it can carry a little motion, and the token count is halved. An image is treated as two identical frames, so images and videos share one patch embedder. [Certain]
For a frame of \(H \times W\) pixels and patch size \(p\), the encoder makes \((H/p)(W/p)\) tokens. A 384×384 frame in SigLIP-so400m with \(p = 14\) gives \(27 \times 27 = 729\) tokens; a 224×224 frame with \(p = 14\) gives \(16 \times 16 = 256\). Without merging, 64 frames of SigLIP tokens would be \(64 \times 729 = 46{,}656\) tokens. That is why Step 3 exists.
Step 3: merge tokens over space and time
The LLM is the expensive part, and its cost grows with the number of tokens (prefill compute grows a little faster than linearly, because of attention). So nearly every Video-LLM merges tokens before the LLM. A general count of visual tokens is
- Read the symbols
- \(K\)
- number of sampled frames
- \(\tau\)
- temporal merge: how many frames share one token (1 for most models, 2 for Qwen2-VL and Qwen2.5-VL)
- \(H, W\)
- frame height and width in pixels after resizing
- \(p\)
- patch size of the encoder (14 for CLIP ViT-L/14, SigLIP and the Qwen ViT)
- \(s\)
- spatial merge: each \(s \times s\) block of patches becomes one token (2 in LLaVA-Video and Qwen2.5-VL)
For Qwen2.5-VL, \(p\,s = 28\) and \(\tau = 2\), so each pair of 448×448 frames costs \((448/28)^2 = 256\) tokens. At 2 fps that is 256 tokens per second of video. [Certain]
Figure 2 runs this count in your browser, one temporal patch at a time, for a toy clip filmed at 30 fps: a ball crosses the frame from left to right. Each step applies one line of the pseudo-code, and the box next to it shows the formula with the real numbers of that step.
- \(K \leftarrow \min(\max(D f,\ 4),\ F_{\max})\), rounded down to even; \(f_s \leftarrow K/D\) (205.2)
- for \(k = 0, \dots, K-1\): \(i_k \leftarrow \lfloor (k + \tfrac12)\, N/K \rfloor\) (205.1)
- for \(j = 0, \dots, K/\tau - 1\): temporal patch \(j\) = frames \(2j,\ 2j+1\)
- cut both frames into \(p \times p\) patches: \(2 \cdot (H/p) \cdot (W/p)\)
- 3D patch embed, \(\tau = 2\) frames deep: \((H/p) \cdot (W/p)\) tubelets
- merge each \(2 \times 2\) group (MLP): \((H/ps) \cdot (W/ps)\) tokens
- append to the context, \(\mathrm{id}_t = \mathrm{id}_0 + \lfloor j\, r\tau / f_s \rfloor\) (205.7)
- text after the video starts at the largest id \(+\,1\); total is (205.3)
How to read Figure 2
Press Play. The first steps take patch 0 through every line of the algorithm: sample, pick the frames, cut patches, merge over time (half the count), merge over space (a quarter again). After that each step appends one or more patches, and the context bar grows by the same amount each time. With 10 s at 2 fps and 448 px, 20 frames become 10 patches of 256 tokens: 2,560 tokens. Now move the sliders; the readout updates at once. Doubling the fps doubles the tokens; doubling the frame side multiplies them by 4. Set 10 minutes: the frame cap (768) is reached, the real rate drops to 1.28 fps, and the 98,304 tokens pass the 24,576 cap (red), so the frames would have to shrink to 224 px. Watch the temporal ids in the box: they always equal 2 × the patch start time in seconds.
Four ways to merge
- Spatial pooling per frame. Average (or interpolate) each \(2 \times 2\) block of tokens. LLaVA-Video reduces SigLIP’s \(27 \times 27\) grid to \(14 \times 14 = 196\) tokens per frame with bilinear interpolation, and adds one “row end” token per row, so 210 tokens per frame in the released config. [Certain] Qwen2.5-VL concatenates each \(2 \times 2\) group of features and sends it through a 2-layer MLP, so nothing is thrown away before the MLP. [Certain]
- Temporal merging. Merge neighboring frames into one token (Qwen’s temporal patch \(\tau = 2\)).
- Global average over space and over time. Video-ChatGPT (Maaz et al., 2023) averages the frame features in two ways. Averaging over all patches of a frame gives one token per frame (\(K\) “temporal” tokens). Averaging each patch position over all frames gives one token per position (\(N_p\) “spatial” tokens):
- Read the symbols
- \(\mathbf{x}_{i,j}\)
- the feature of patch \(j\) in frame \(i\), from the image encoder
- \(N_p\)
- patches per frame, for example 256
- \(\mathbf{t}_i\)
- temporal token of frame \(i\): what the frame contains, with no “where”
- \(\mathbf{z}_j\)
- spatial token of position \(j\): what was at that place on average, with no “when”
This is extremely cheap: with 100 frames and 256 patches, 356 tokens instead of 25,600. [Likely] (The paper gives the formula; 100 frames is the setting in its code.) We will see in Figure 4 what it loses.
- SlowFast. Borrowed from the SlowFast network for action recognition. Every \(s\)-th frame is a “slow” frame with light pooling (detail); the other frames are “fast” frames with heavy pooling (motion). LLaVA-Video’s 72B model uses it with \(s = 3\). [Certain] With \(M\) tokens per frame and pooling size \(p\) for slow frames, \(2p\) for fast ones:
With \(K = 64\), \(s = 3\) and SigLIP tokens rounded up as in the released code (196 per slow frame, \(7 \times 7 = 49\) per fast frame): \(22 \times 196 + 42 \times 49 = 6{,}370\) tokens, about half of \(64 \times 196 = 12{,}544\). (The paper’s own count uses slightly different rounding.)
Other systems use a learned compressor instead of pooling: Video-LLaMA puts a Q-Former over the frame features (see session 178). The idea is the same: a fixed, small number of tokens per frame or per clip.
Figure 3 lets you compare the four case-study designs on the same video.
How to read Figure 3
Start with Video-LLaVA and slide the video length from 1 minute to 1 hour: the token count stays at 2,048, but the gap between frames grows from 7.5 s to 450 s. Switch to Qwen2.5-VL: at 2 fps the token count grows with the video until the 768-frame cap. When the count passes the 24,576-token cap, the figure lowers the frame resolution (a simplified version of what the Qwen preprocessing does with its pixel budget) and the readout shows the new frame size. Video-ChatGPT is almost free, but read Figure 4 before you choose it.
Video-LLaVA. 8 frames, \(16 \times 16 = 256\) tokens each: \(8 \times 256 = 2{,}048\) tokens. The gap between frames is \(600/8 = 75\) s. Any event shorter than about a minute may be invisible.
Qwen2.5-VL. At 2 fps the video has 1,200 candidate frames; the cap keeps 768, so \(f_{\text{eff}} = 1.28\) fps (formula 205.2) and the gap is 0.78 s. The 768 frames form 384 temporal patches. At 448×448 that would be \(384 \times 256 = 98{,}304\) tokens. The 24,576-token cap leaves \(24{,}576 / 384 = 64\) tokens per patch, an \(8 \times 8\) grid, which means frames of \(8 \times 28 = 224\) pixels per side.
So the two designs make opposite trades: Video-LLaVA keeps resolution and loses time; Qwen2.5-VL keeps time and loses resolution. Neither is free. For one hour, 768 frames at 64 tokens per pair is again exactly 24,576 tokens, with one frame every 4.69 s.
What pooling keeps and what it loses
Merging saves tokens, but each kind of merge throws away a different kind of information. Averaging over space keeps “what” but loses “where”. Averaging over time keeps “where” but loses “when”, and therefore the order of events. Figure 4 makes this concrete with a toy video: a ball moves across a 4×4 grid, and the question is “Does the ball move left or right?”.
How to read Figure 4
Flip the direction and watch the lower row. With all patch tokens or a 2×2 pool, the two directions give different tokens (distance above 0), so the LLM can tell them apart. With a mean over space, every frame becomes the same token (the ball is always somewhere). With a mean over time, both directions leave the same trail. Video-ChatGPT’s tokens are the union of those two, so they are identical for both directions. In real models the features are richer and carry some position, so the loss is softer, but the lesson holds: heavy pooling makes order, direction and counting questions hard. [Likely]
Step 4: telling the model about time
After merging, the LLM has a long row of visual tokens. It still needs to know when each token happened. Without that, it can say “the man falls” but not “the man falls at 42 s”, and it cannot compare videos sampled at different rates. There are three ways to give time to the model.
Way 1: frame index (order only)
The simplest way: the tokens of frame 0 come first, then frame 1, and so on, and the normal position encoding of the LLM marks their order. In Qwen2-VL’s M-RoPE (multimodal RoPE, see session 14) every token has three position ids: temporal, height and width. For text the three ids are equal, so it is ordinary RoPE. For an image the temporal id is constant. For a video “the temporal ID increments for each frame”. [Certain] So the \(j\)-th temporal patch gets
The problem: id 15 means second 15 if the video was sampled at 2 fps (one temporal patch of 2 frames per second), but second 60 at 0.5 fps. The model only knows the order, not the time. The Qwen2.5-VL report says exactly this: Qwen2-VL’s temporal ids were tied to the number of frames, which “ignored content speed and absolute timing”. [Certain]
Way 2: timestamps written as text
Write the time into the prompt. LLaVA-Video’s training code, when its time option is on (it is on in the released 7B config), puts this sentence before the question: “The video lasts for 60.00 seconds, and 32 frames are uniformly sampled from it. These frames are located at 0.94s,2.81s,4.69s,...” [Certain] Other models put a timestamp token next to each frame (TimeChat, for example, binds timestamps to frames). [Likely]
The model must then link “the 12th timestamp in the sentence” to “the 12th group of frame tokens”, which it learns from data. Text timestamps are cheap. With the Qwen2.5 tokenizer, every digit is a separate token, so “0.94s,” is 6 tokens and “12.50s,” is 7. For 64 frames of a 3-minute video the list costs 472 tokens, about 3.5% of the 13,440 visual tokens of LLaVA-Video. They also make the model’s answers natural: it can copy the time it reads.
Way 3: absolute time in the position encoding
Qwen2.5-VL changes formula (205.6) so that the temporal id grows with seconds, not with frames. The released config has tokens_per_second = 2 and temporal_patch_size = 2, and the transformers code spaces the temporal ids of a video by \(r \tau / f_s\): [Certain]
- Read the symbols
- \(j\)
- index of the temporal patch (each patch holds \(\tau\) frames)
- \(\tau\)
- frames per temporal patch, 2
- \(f_s\)
- the real sampled frame rate of this video, for example 1.28 fps
- \(r\)
- temporal ids per second of video, 2 in the released Qwen2.5-VL config
- \(t_j = j \tau / f_s\)
- start time in seconds of patch \(j\)
- \(\mathrm{id}_0\)
- the position where the video starts in the sequence (after the text before it)
So one temporal id is always half a second, whatever the sampling rate. Why does this help attention? RoPE rotates each query and key pair by an angle proportional to its position, and the attention score depends only on the difference of positions:
With formula (205.7), \(n - m \approx r\,(t_n - t_m)\): the angle measures the time between two tokens in seconds. A pattern such as “2 seconds apart” then looks the same to attention in a video sampled at 2 fps and in one sampled at 0.5 fps. In Qwen2.5-VL-7B, the 128-dimensional attention heads have 64 rotating pairs, split by mrope_section = [16, 24, 24]: 16 pairs rotate with the temporal id, 24 with the height id and 24 with the width id. [Certain] After the video, the text continues from the largest id used plus 1.
Figure 5 shows the three ways side by side for a 60-second video with an event at 40 s.
- \(f_s \leftarrow K/D\); \(P \leftarrow K/\tau\) temporal patches
- for \(j = 0, \dots, P-1\) (A and B side by side in time):
- \(t_j \leftarrow j\tau / f_s\) start time of the patch, seconds
- frame index: \(\mathrm{id}_t \leftarrow \mathrm{id}_0 + j\) (205.6)
- text: write the times of frames \(2j,\ 2j+1\) into the prompt; \(\mathrm{id}_t \leftarrow \mathrm{id}_0 + j\)
- absolute time: \(\mathrm{id}_t \leftarrow \mathrm{id}_0 + \lfloor j\, r\tau / f_s \rfloor\) (205.7)
- event at \(e\) s \(\to\) patch \(\lfloor e f_s / \tau \rfloor\); compare its id in A and B
How to read Figure 5
Press Play: the dots appear patch by patch, both videos moving forward at the same clock time, and the box applies line 4, 5 or 6 of the pseudo-code to the newest patch. In frame-index mode the ids of B fall behind those of A as time runs; in absolute-time mode the two rows stay aligned. With frame index, the event gets id 40 in A but a smaller id in B: the same moment has two different positions, and the same id means two different times. With absolute time, both A and B put the event at about id 80 (2 ids per second), so time is consistent. With text timestamps the ids are just token order again, but the prompt contains the time of every frame; the readout counts how many text tokens that costs.
From the first worked example, \(f_s = 1.28\) fps. Formula (205.7) gives a spacing of \(r\tau/f_s = 2 \times 2 / 1.28 = 3.125\) ids per temporal patch, so the first patches get offsets \(0, 3, 6, 9, 12, 15\) after rounding down. The patch that starts at 300 s is patch \(j = 300 \times 1.28 / 2 = 192\), with offset \(\lfloor 192 \times 3.125 \rfloor = 600 = 2 \times 300\). With frame index (205.6) the same patch would get offset 192, but at 2 fps the moment “300 s” would get offset 300. Only absolute time gives the same id for the same second.
Video instruction data and joint image-video training
Architecture is only half of a Video-LLM. The other half is data: videos paired with questions and answers that need the video to be answered. Such data is rare and expensive to label by hand, so most of it is made semi-automatically.
- Video-ChatGPT built 100,000 video-instruction pairs, about 30% with human help (enriching ActivityNet captions) and 70% with a semi-automatic pipeline (image captioners, taggers and GPT-3.5). [Certain]
- LLaVA-Video-178K has 178,510 videos of up to 3 minutes from ten sources (for example HD-VILA-100M, InternVid-10M, Charades, ActivityNet, Kinetics-700, Something-Something v2 and Ego4D), with 178K detailed captions, 960K open-ended questions and 196K multiple-choice questions, all written by GPT-4o. [Certain]
The LLaVA-Video captions are made by a recurrent, three-level process, because GPT-4o cannot take all frames at once. Frames are sampled at 1 fps. Every 10 seconds, GPT-4o writes a level-1 description from the current frames and the recent descriptions. Every 30 seconds it writes a level-2 summary from the last three level-1 descriptions. At the end it writes one level-3 description of the whole video. Questions are then generated from these captions. [Certain]
Why train on images and videos together?
Video data is smaller and noisier than image data. If a model is fine-tuned only on video, it can forget image skills such as reading text and fine detail, and it sees fewer kinds of visual concepts. Training on both keeps those skills, and the skills transfer from images to frames. Video-LLaVA tested this directly: compared with a version trained on video alone, joint training raised MSVD-QA accuracy from 64.8 to 70.7, MSRVTT-QA from 58.3 to 59.2, TGIF-QA from 67.8 to 70.0 and ActivityNet-QA from 40.7 to 45.3. [Certain] LLaVA-Video trained on about 1.6 million video samples together with 1.1 million image samples from LLaVA-OneVision. [Certain]
The training loss is the usual next-token loss of session 54, computed only on the answer tokens:
- Read the symbols
- \(\mathbf{V}\)
- the visual tokens of the video (or of an image, treated as a one-frame video)
- \(\mathbf{q}\)
- the question tokens, plus the time sentence if timestamps are given as text
- \(\mathcal{A}\)
- the positions of the answer tokens; prompt and visual tokens get no loss
- \(y_t\)
- the answer token at position \(t\)
Mixing images and videos in one batch is easy when every sample is “some visual tokens, then text”: an image is one frame (or two identical frames in Qwen), a video is many. Batches mix sequences of very different lengths, so packing and masking matter (session 188).
- input: an image VLM (encoder, projector, LLM); image-text data; video caption data; video instruction data
- choose the sampler (uniform \(K\), or \(f\) fps with cap \(F_{\max}\)) and the merge (formula 205.3) so that a typical video fits the context
- choose the time encoding: frame index, text timestamps, or absolute time ids (formula 205.7)
- stage A (optional): train the projector only, on video captions, with encoder and LLM frozen // aligns video tokens to the LLM
- stage B: instruction-tune the projector and the LLM on a mix of image and video instructions, loss (205.9) on answers only
- in every batch, sample images and videos from the mix; treat an image as a one-frame (or two-identical-frame) video
- optionally unfreeze the encoder with a small learning rate // LLaVA-Video’s config uses 2e-6 for the vision tower
- evaluate on video benchmarks and image benchmarks, to catch forgetting (session 213)
Case studies: Video-LLaVA, LLaVA-Video, Qwen2.5-VL
Video-LLaVA (2023): the simple baseline
Video-LLaVA takes LLaVA-1.5 and adds video with the smallest possible change. The encoders are LanguageBind’s image encoder (initialized from OpenCLIP-L/14) and LanguageBind’s video encoder, both at 224×224 and both frozen. A video is 8 uniformly sampled frames. One shared projector (two linear layers with GELU) maps both images and videos into Vicuna-7B v1.5. Stage 1 trains the projector on 558K image-text pairs and 702K video-text pairs; stage 2 trains the projector and LLM on 665K image instructions and 100K video instructions (from Video-ChatGPT). [Certain] With 256 patch tokens per frame, a video costs 2,048 tokens. [Likely]
LLaVA-Video (2024): better data, more frames, timestamps as text
LLaVA-Video starts from the LLaVA-OneVision single-image checkpoint: SigLIP-so400m at 384×384 and a Qwen2 LLM. Video frames are pooled from \(27 \times 27\) to \(14 \times 14\) tokens, and up to 64 frames are used in the final setting (ablations went to 110). The main contribution is the LLaVA-Video-178K dataset and its recurrent captioning. Time enters as text: the prompt lists the duration and the time of each sampled frame. The 72B model uses the SlowFast representation (\(s = 3\)) to fit more frames. [Certain]
Qwen2.5-VL (2025): dynamic fps and absolute time
Qwen2.5-VL uses one native-resolution ViT for images and videos, with 14×14 patches, temporal patches of 2 frames, and a 2×2 MLP merger. To keep cost linear in image size, most ViT layers use window attention with windows of at most 112×112 pixels (8×8 patches); only 4 of the 32 layers (indices 7, 15, 23, 31) use full attention. [Certain] During training the sampling fps is varied, so the model sees many frame rates, and temporal ids follow absolute time (formula 205.7). Training data for grounding writes times both in seconds and in an hour-minute-second-frame format, and the last pretraining phase packs sequences of 32,768 tokens with long videos. In evaluation, videos were capped at 768 frames and 24,576 video tokens. [Certain]
| Video-LLaVA | LLaVA-Video (7B) | Qwen2.5-VL | |
|---|---|---|---|
| Frame encoder | LanguageBind image and video encoders (ViT-L/14, 224 px), frozen | SigLIP-so400m, 384 px | own ViT, native resolution, window attention |
| Sampling | 8 frames, uniform | up to 64 frames, uniform | 2 fps by default, cap 768 frames in evaluation |
| Merging | none (256 tokens per frame) | 2×2 bilinear: 196 per frame (+14 row tokens) | 2 frames × 2×2 patches → 1 token (MLP) |
| Time | frame order only | timestamps as text in the prompt | absolute time in M-RoPE (2 ids per second) |
| LLM | Vicuna-7B v1.5 | Qwen2 | Qwen2.5 |
| Video data | 702K video-text + 100K instructions | LLaVA-Video-178K plus public QA sets, about 1.6M samples | large mixed corpus, long-video captions, grounding with timestamps |
| Images too? | yes (558K + 665K) | yes (1.1M from LLaVA-OneVision) | yes |
Code: build the inputs of a Video-LLM in PyTorch
- input: video with \(N\) frames at rate \(f_v\); target fps \(f\); cap \(F_{\max}\); text before and after the video
- \(D \leftarrow N / f_v\); \(K \leftarrow \min(\max(D f, 4), F_{\max}, N)\), rounded down to an even number
- pick frame indices with formula (205.1); \(f_s \leftarrow K / D\) // real sampled fps
- decode and resize the frames to \(H \times W\), multiples of 28
- cut into patches of \(2 \times 14 \times 14\) pixels; embed; run the ViT
- merge each \(2 \times 2\) group with the MLP: \((K/2) \cdot (H/28) \cdot (W/28)\) tokens (formula 205.3)
- position ids: text before the video gets \(0, 1, \dots, \mathrm{id}_0 - 1\) on all three axes
- video tokens: temporal id by formula (205.7), height and width ids from the grid, all offset by \(\mathrm{id}_0\)
- text after the video continues at (largest id used) + 1
- run the LLM with M-RoPE on the joined sequence
The code below runs on a CPU in under a second. It uses a tiny random “ViT” (a single linear layer), so the numbers are about shapes, token counts and time ids, not about quality. It checks two facts from this session: absolute-time ids match seconds for two sampling rates, and Video-ChatGPT pooling cannot see the order of frames.
import torch torch.manual_seed(0) # 1) Frame sampling: a target fps with a frame cap, formula (205.1) def sample_frames(n_frames, video_fps, target_fps=2.0, max_frames=768, tp=2): dur = n_frames / video_fps # seconds k = min(max(dur * target_fps, 4), max_frames, n_frames) k = int(k // tp * tp) # even: frames pair into temporal patches idx = ((torch.arange(k) + 0.5) * n_frames / k).long() # centre of each of k equal segments return idx, k / dur # frame indices, real sampled fps # 2) Toy visual side: 3D patch embed (2 frames x 14 x 14), then 2x2 merge + MLP class ToyVisual(torch.nn.Module): def __init__(self, d_vit=64, d_llm=96, p=14, tp=2, m=2): super().__init__() self.p, self.tp, self.m = p, tp, m self.embed = torch.nn.Linear(3 * tp * p * p, d_vit) # same as Conv3d, stride (tp, p, p) self.merge = torch.nn.Sequential(torch.nn.Linear(d_vit * m * m, d_llm), torch.nn.GELU(), torch.nn.Linear(d_llm, d_llm)) def forward(self, video): # video: (K, 3, H, W) K, C, H, W = video.shape p, tp, m = self.p, self.tp, self.m t, h, w = K // tp, H // p, W // p x = video.reshape(t, tp, C, h, p, w, p).permute(0, 3, 5, 1, 2, 4, 6) x = self.embed(x.reshape(t, h, w, tp * C * p * p)) # (t, h, w, d_vit); ViT blocks go here x = x.reshape(t, h // m, m, w // m, m, -1).permute(0, 1, 3, 2, 4, 5) return self.merge(x.reshape(t, h // m, w // m, -1)) # (t, h/2, w/2, d_llm) # 3) M-RoPE ids with absolute time, formula (205.7) def mrope_ids(start, t, h, w, fps, tp=2, tokens_per_second=2): step = tokens_per_second * tp / fps # temporal ids per temporal patch T = (torch.arange(t) * step).long() tt, hh, ww = torch.meshgrid(T, torch.arange(h), torch.arange(w), indexing="ij") return torch.stack([tt, hh, ww]).reshape(3, -1) + start # (3, t*h*w) vis = ToyVisual() for target in (2.0, 0.5): # same 30 s clip, two sampling rates idx, fps = sample_frames(n_frames=900, video_fps=30, target_fps=target) video = torch.rand(len(idx), 3, 112, 112) # stand-in for decoded frames tok = vis(video) t, h, w, _ = tok.shape ids = mrope_ids(start=10, t=t, h=h, w=w, fps=fps) # 10 text tokens come first last_t = (ids[0].max() - 10).item() true_t = idx[-2].item() / 30 # first frame of the last patch print(f"fps {fps:.3f}: frames {len(idx)}, tokens {t*h*w}, last temporal id {last_t}" f" -> {last_t / 2:.1f} s (true {true_t:.1f} s)") nxt = ids.max() + 1 # text after the video continues here # 4) Video-ChatGPT style pooling: (T + N) tokens, and it ignores frame order f = torch.randn(8, 256, 32) # T=8 frames, N=256 patches, D=32 def vcg_pool(f): return torch.cat([f.mean(1), f.mean(0)]) # (T + N, D), formula (205.4) a, b = vcg_pool(f), vcg_pool(f.flip(0)) print(a.shape[0], # 264 = 8 + 256 tokens torch.allclose(a[8:], b[8:], atol=1e-6), # True: spatial part forgets order torch.allclose(a[:8], b[:8].flip(0), atol=1e-6)) # True: temporal part is just reversed
Output (the same on every run):
fps 2.000: frames 60, tokens 480, last temporal id 58 -> 29.0 s (true 29.2 s) fps 0.467: frames 14, tokens 112, last temporal id 51 -> 25.5 s (true 26.8 s) 264 True True
Read it: at 2 fps the 30-second clip gives 60 frames, 30 temporal patches of \(4 \times 4 = 16\) tokens each, so 480 tokens. At 0.5 fps the clip gives 15 frames, rounded down to 14 (an even number), so the real rate is 0.467 fps. In both cases the temporal id divided by 2 is a time in seconds. The id marks the start of a patch’s time slot, so it can be up to one patch length (\(\tau / f_s\), here 4.3 s) earlier than the frames inside it. The last line shows that the 256 spatial tokens of Video-ChatGPT are identical when the video is played backwards, and the 8 temporal tokens are just reversed.
Where this is used
| Application | What the Video-LLM does | Design choices that matter |
|---|---|---|
| Video question answering | Answers free-form questions about a clip | Frames vs resolution; joint image-video training for detail (session 210) |
| Key-moment detection and temporal grounding | Returns start and end times of an event (“the goal is at 42 to 47 s”) | Dense sampling and an explicit time encoding; training data with timestamps (session 207) |
| Dense captioning and summaries | Writes a time-stamped description of a whole video | Recurrent or hierarchical captions, as in LLaVA-Video-178K; long context |
| Action recognition, zero-shot | Names the action from a list in the prompt | Temporal order must survive pooling (Figure 4); compare with trained classifiers (session 203) |
| Video retrieval and re-ranking | Checks whether a retrieved clip really matches a text query | Cheap token budget per candidate; a contrastive model retrieves first (session 208) |
| Video-RAG over large archives | Answers questions using clips found by search | Short clips per answer; timestamps so answers can cite the moment |
| Content moderation and sports analytics | Flags or describes events in streams | Fixed fps sampling so short events are not missed; latency budget (session 211) |
Common mistakes
- Too few frames for “when” questions. 8 uniform frames from a 10-minute video are 75 s apart. No model can localize an event it never saw.
- Sampling at a different rate than in training. A model trained with 8 frames and run with 64 may get worse, not better, especially with frame-index time. Check the training setting, or use a model trained with many rates (dynamic fps).
- Frame-index time with variable fps. If the temporal id counts frames, then id 15 means different seconds in different videos. Either fix the fps, give timestamps as text, or use absolute-time ids.
- Wrong fps passed to the processor. With absolute time, the temporal spacing uses the sampled fps (formula 205.7). If your own sampler passes the video’s native 30 fps instead of the sampled rate, every id is wrong by a factor of 15 at 2 fps, and the model reports wrong times.
- Pooling away order and counting. Mean pooling over time (Figure 4) cannot tell “opens the door” from “closes the door”. Keep per-frame tokens, even if fewer of them.
- Training only on video. Image skills such as OCR and fine detail fade. Keep images in the mix and evaluate both.
- Ignoring the token count of text timestamps or “row end” tokens. They are small (a few percent), but they count toward the context and the cap.
- Forgetting decode cost. The LLM may be fast, but decoding 768 frames from a long video can take longer than the model (session 201).
Interview questions
Describe the architecture of a typical Video-LLM.basic
How many visual tokens does one minute of video cost in Qwen2.5-VL at 2 fps and 448×448?core
Uniform K frames or fixed fps: when would you choose each?core
Why can mean pooling over time hurt a Video-LLM?core
What changed between Qwen2-VL and Qwen2.5-VL in how time is encoded?advanced
What are the pros and cons of putting timestamps into the prompt as text?core
Why train Video-LLMs on images and videos together?core
How was LLaVA-Video-178K built, and why not just use human labels?core
Your Video-LLM describes events well but gives wrong times. What do you check?advanced
You must answer questions on a 1-hour video with a 24,576-token video budget in Qwen2.5-VL. What are your options?advanced
What does “alignment before projection” mean in Video-LLaVA?basic
Exercises
- Video-LLaVA samples 8 frames from a 2-minute clip. How far apart are the frames, and how many visual tokens does the clip cost?
- Qwen2.5-VL samples a 20-second clip at 1 fps with frames of 336×336. How many visual tokens? How many temporal ids apart are two neighboring temporal patches, and what is the offset of the last one?
- Use formula (205.5) with \(K = 32\), \(s = 4\), 196 tokens per slow frame and 49 per fast frame. Compare with all 32 frames slow.
- In Figure 4, explain why the 2×2 pool still tells the two directions apart, while the mean over space does not.
- A prompt lists the times of 32 uniformly sampled frames of a 60-second video in the format “0.94s,”. Using the rule “digits of the integer part + 5 tokens” (digits, the point, two decimals, “s” and the comma are separate tokens), how many tokens does the list cost?
- A user runs Qwen2.5-VL with their own sampler at 2 fps but passes the native 30 fps to the processor. By what factor are the temporal ids wrong, and will the model’s answers in seconds be too early or too late?
Answerscheck
Key takeaways
- A Video-LLM is an image VLM plus four choices: frame sampling (205.1, 205.2), the per-frame encoder, token merging over space and time (205.3 to 205.5), and a time encoding.
- The token budget forces a trade: few frames at high resolution (Video-LLaVA, 2,048 tokens for 8 frames) or many frames at low resolution (Qwen2.5-VL, up to 768 frames in 24,576 tokens).
- Mean pooling over time or space can erase order and direction; keep per-frame tokens when the task is about events.
- Time can be given as frame order (205.6), as text timestamps (cheap, about 6 to 8 tokens per frame), or as absolute time in M-RoPE (205.7), where one temporal id is half a second and RoPE (205.8) sees time differences in seconds.
- Data is half the system: synthetic, multi-level video captions and QA (LLaVA-Video-178K), and joint image-video training (205.9), which improved every video QA set in Video-LLaVA’s ablation.
Key formulas
References for this session
- Lin et al. (2023). Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
- Zhang et al. (2024). Video Instruction Tuning With Synthetic Data
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Lin et al. (2023). Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
- Zhang et al. (2024). Video Instruction Tuning With Synthetic Data
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Tang et al. (2023). Video Understanding with Large Language Models: A Survey
- Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
- Feichtenhofer et al. (2018). SlowFast Networks for Video Recognition
- Ren et al. (2023). TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
- Maaz, Rasheed, Khan & Khan (2023). Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
- Zhu et al. (2023). LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
- Zhang, Li & Bing (2023). Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
- Li et al. (2024). LLaVA-OneVision: Easy Visual Task Transfer
- Hugging Face and Qwen team (2025). Qwen2.5-VL in transformers: config.json (tokens_per_second, temporal_patch_size) and get_rope_index
- LLaVA-VL team (2024). LLaVA-NeXT code base: llava_arch.py (video pooling) and train.py (time instruction)