LLM & VLM courseBlog
201Track 8: Video and multimodal systems · session 1 of 21

Video as data

Before a model can watch a video, someone must open the file, decode compressed frames, pick a few of them and turn them into tokens. This session explains what a video file really contains (frames, codecs, I, P and B frames), why decoding is often the slowest part of a video pipeline, how to choose which frames to keep, and how many tokens one second of video costs a vision-language model.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on the Vision Transformer, the anatomy of a VLM and the VLM inference path

What you will learn

  • What a video is as data: frames, frame rate, resolution, YUV 4:2:0 color, and why raw video is about 100 times larger than the file
  • How codecs compress video with I, P and B frames, what a GOP is, and why decode order differs from display order
  • Why random access into a video is expensive, how many frames a decoder must decode to return one frame, and when one sequential pass is cheaper than seeking
  • Where decoding runs (CPU, NVDEC on the GPU) and why decoding can cost more than the vision encoder
  • Frame sampling strategies: uniform, fixed fps, keyframes, scene-based and adaptive, and what each one misses
  • Tokens per second of video: how patch size, token merging, temporal patches, fps and resolution set the token budget, with Qwen2-VL and Gemini numbers

Why this matters

A vision-language model (VLM) never sees a video file. It sees a few frames, cut into patches and turned into tokens. Between the file and the tokens there is a chain of steps: open the container, decode compressed frames, choose which frames to keep, resize them, and count how many tokens they will cost. Each step can be slow, wrong or expensive. This session explains that chain from the bottom up.

Three problems appear in every real video system, from a Video-LLM that answers questions to a search engine that indexes millions of clips:

  1. The data is huge. One second of 1080p video at 30 frames per second is 187 MB as RGB pixels. The file holds the same second in less than 1 MB, because it is compressed.
  2. Compression makes access expensive. Most frames in the file are stored as “changes from another frame”. To get one frame back, the decoder must first rebuild the frames it depends on. Picking 32 frames from a 10-minute video can mean decoding thousands of frames.
  3. Tokens are the scarce resource. A model that spends 256 tokens on each second of video fills a 16K-token budget in about one minute. So we must choose which frames, at which resolution, the model will see.

This is the first session of the Track 8 track. In session 196 we followed one image through a VLM; here the input is a stream of images with time. The next session, session 202, takes the sampled frames and asks how a network should model motion. Later sessions build on the budget problem: session 206 for hours of footage, session 207 for finding moments, and session 211 for running all of this at scale. The book chapter on video-language models (Noyan et al., chapter 9) and the survey by Tang et al. give the wider picture.

What a video is: frames, frame rate, resolution and color

A video is a sequence of still images called frames, shown quickly one after another. Three numbers describe it:

  • Resolution \(W \times H\): the width and height of each frame in pixels. 1080p means 1920×1080; 4K (UHD) means 3840×2160.
  • Frame rate \(f\): frames per second (fps). Film uses 24, TV and phones 25, 30 or 60.
  • Duration \(D\): seconds. So the video has \(N = D \cdot f\) frames.

Pixels are usually not stored as RGB. Video uses YUV: one brightness channel (Y, called luma) and two color-difference channels (U and V, called chroma). Our eyes see detail in brightness much better than in color, so codecs store the color channels at lower resolution. In the common 4:2:0 format, U and V have half the width and half the height of Y. Each pixel then costs \(1 + \tfrac14 + \tfrac14 = 1.5\) bytes at 8 bits per sample, instead of 3 bytes for RGB. A vision encoder wants RGB, so the pipeline converts YUV to RGB at some point.

The raw data rate of a video, before compression, is:

\[ R_{\text{raw}} \;=\; W \cdot H \cdot b_{\text{px}} \cdot f \quad \text{bytes per second} \tag{201.1} \]
Read the symbols
\(W,\ H\)
frame width and height in pixels
\(b_{\text{px}}\)
bytes per pixel: 1.5 for 8-bit YUV 4:2:0, 3 for 8-bit RGB
\(f\)
frame rate in frames per second

A file stores the video at an encoded bit rate \(R_{\text{enc}}\) (for example 5 Mbit/s). The compression ratio compares the two:

\[ c \;=\; \frac{R_{\text{raw}}}{R_{\text{enc}}} \tag{201.2} \]

For 1080p at 30 fps in YUV 4:2:0, \(R_{\text{raw}} = 1920 \times 1080 \times 1.5 \times 30 = 93.3\) MB/s, which is 746 Mbit/s. A 5 Mbit/s stream is then compressed about 149 times. The video lives in a container file (MP4, MKV, WebM, MOV). The container holds one or more streams (video, audio, subtitles) as small chunks called packets, plus timestamps and metadata such as rotation. The codec is the method that compresses each video stream: H.264 (also called AVC), H.265 (HEVC), VP9 or AV1. “MP4” says nothing about the codec; an MP4 file can hold H.264 or AV1.

MP4 fileH.264, 5 Mbit/s0.63 MB demuxpackets I P B0.63 MB decodeCPU or NVDEC93.3 MB YUV to RGB30 frames186.6 MB sample, resize2 frames, 448×4481.20 MB ViT + merge256 tokens1.84 MB One second of 1080p video at 30 fps. Numbers are the data size of that second after each step. Where the work is every frame in the decode chain is paid memory traffic only kept frames are paid Three budgets 1. decode: frames decoded per second (CPU cores or NVDEC engines) 2. bandwidth: 186.6 MB/s of RGB per 1080p30 stream if frames are decoded on the CPU and copied to the GPU 3. tokens: 256 tokens per second here; a 16,384-token video budget lasts 64 s
Figure 1. From file to tokens. The data grows about 300 times during decoding (0.63 MB to 186.6 MB), then shrinks again when we keep only a few small frames. The decoder pays for every frame it must decode; the vision encoder pays only for frames we keep. Token numbers use the Qwen2-VL scheme explained in the last part of this session; 1.84 MB is the size of the 256 token vectors entering the LLM (width 3,584, 2 bytes each in BF16).
Worked example: how big is a video?

Raw. One 1080p RGB frame is \(1920 \times 1080 \times 3 = 6{,}220{,}800\) bytes (6.2 MB). One hour at 30 fps has 108,000 frames, so 672 GB of RGB pixels. The same hour at 5 Mbit/s is a 2.25 GB file.

A real encode. We encoded 30 seconds of a synthetic 1080p, 30 fps test pattern (ffmpeg testsrc2) with the x264 encoder at its default settings. The file is 23,945,749 bytes, so 6.39 Mbit/s. The raw YUV 4:2:0 data is \(900 \times 3{,}110{,}400\) bytes = 2.80 GB, so the compression ratio (formula 201.2) is 117. Of the 900 frames, ffprobe reports 4 I frames, 419 P frames and 477 B frames. The I frames are at frames 0, 250, 500 and 750: one every 250 frames, which is the x264 default maximum keyframe interval. [Certain: measured with ffmpeg 9.0 and x264 defaults --keyint 250 --bframes 3 --scenecut 40]

Codecs and GOPs: I, P and B frames

Neighboring frames look almost the same. A codec uses this. It stores only a few frames completely and describes the others as changes. There are three frame types:

  • I frame (intra frame): compressed alone, like a JPEG. It can be decoded without any other frame. It is the largest type.
  • P frame (predicted frame): stores motion vectors and a small correction relative to an earlier frame. “Take this block from the previous frame, move it 3 pixels left, add this small difference.” To decode it, the earlier frame must already be decoded.
  • B frame (bi-directional frame): predicted from an earlier and a later frame. It is usually the smallest. To decode it, both of its reference frames must be decoded first, including the later one.

A GOP (group of pictures) is an I frame plus all the frames that follow it until the next I frame. A keyframe is an I frame where decoding can start from scratch (in H.264 this is an IDR frame: no later frame may look back past it). [Certain] The GOP length \(G\) is the keyframe interval. With x264 defaults, \(G\) is at most 250 frames, so at 30 fps a keyframe comes at least every 8.33 seconds; the encoder also inserts extra I frames at scene cuts. [Certain]

Because a B frame needs a later frame, the file stores frames in decode order, not in display order. For the display order I B B B P, the file stores I P B B B. Each frame therefore has two timestamps: the DTS (decode timestamp) and the PTS (presentation timestamp, when it is shown). Use the PTS to know when a frame happens in the video.

Now the key cost. To get frame \(j\) of a GOP (counting from the keyframe at position 0), the decoder must start at the keyframe. With only P frames, every frame depends on the one before it, so the decoder must decode frames \(0, 1, \dots, j\): that is \(j+1\) frames. A simple decode loop does the same even with B frames: it decodes everything in decode order until frame \(j\) comes out. If the target position is random, the average cost is:

\[ \E\big[\,\text{frames decoded for one random frame}\,\big] \;=\; \frac{1}{G}\sum_{j=0}^{G-1} (j+1) \;=\; \frac{G+1}{2} \tag{201.3} \]
Read the symbols
\(G\)
GOP length: frames from one keyframe to the next
\(j\)
position of the target frame inside its GOP (in decode order); 0 is the keyframe

For \(G = 250\), that is 125.5 frames decoded to return one frame. A smarter decoder can skip B frames that no other frame uses (non-reference B frames) and decode only the chain of I and P frames that the target needs. In our simple model below (B frames are never used as references), that minimum is smaller. Real H.264 encoders often do use some B frames as references (x264 does this by default with its “B-pyramid”), so the true minimum is between the two. [Likely] Explore the chain in Figure 2.

I frameneeded reference (minimum chain)selected framedecoded by a simple loop, not needed
Figure 2. One GOP. Top row: frames in display order (the order you watch them). Bottom row: the same frames in decode order (the order they are stored and decoded). Click any frame in the top row. Anchors (I and P frames) are placed every \(b+1\) frames and the last frame is always a P frame; in this simple model, B frames are never references.
How to read Figure 2

The default shows \(G = 16\) with 3 B frames between anchors, like x264’s default B-frame count. Frame 14 is a B frame selected. Its references are P12 (before it) and P15 (after it); P15 needs P12, P12 needs P8, and so on back to I0. So the minimum is 6 frames: I0, P4, P8, P12, P15 and B14. A simple decode loop goes through the bottom row from the left until B14 comes out: 16 frames. The readout also gives the averages over the GOP: the simple loop always averages \((G+1)/2\) (formula 201.3), whatever the number of B frames. Set B frames to 0: every frame now needs all frames before it.

Decoding is the hidden bottleneck

A Video-LLM keeps maybe 8 to 64 frames of a clip. People then think the cost is 8 to 64 frames. It is not: the decoder pays for every frame in each decode chain. Suppose we want \(k\) target frames at positions \(t_1 \lt t_2 \lt \dots \lt t_k\) in a video of \(N\) frames with keyframes every \(G\) frames. A naive reader seeks to each target separately: it jumps to the keyframe before \(t_i\) and decodes up to \(t_i\). With only P frames:

\[ C_{\text{naive}} \;=\; \sum_{i=1}^{k} \big( (t_i \bmod G) + 1 \big) \;\approx\; k \cdot \frac{G+1}{2} \tag{201.4} \]

A GOP-aware reader sorts the targets and decodes each GOP at most once, from its keyframe up to the last target inside it:

\[ C_{\text{smart}} \;=\; \sum_{g \,\in\, \text{GOPs with a target}} \Big( \max_{i:\ \lfloor t_i / G \rfloor = g} (t_i \bmod G) \;+\; 1 \Big) \;\le\; \min\!\big(N,\ C_{\text{naive}}\big) \tag{201.5} \]
Read the symbols
\(t_i\)
frame index of the \(i\)-th target frame; \(t_i \bmod G\) is its position inside its GOP
\(\lfloor t_i / G \rfloor\)
the number of the GOP that contains target \(i\)
\(N\)
all frames of the video: the cost of decoding everything once, in order

Two regimes follow. When the targets are few and far apart (one per GOP or less), each costs about \((G+1)/2\) frames, and seeking beats decoding everything. When there are many targets, the naive reader decodes the same GOP again and again, and its cost passes \(N\) at about \(k \approx 2N/(G+1)\) targets. For a 10-minute video at 30 fps (\(N = 18{,}000\)) and \(G = 250\), that break-even is about 143 targets. Sampling at 2 fps means 1,200 targets: the naive reader then decodes 150,600 frames, more than 8 times the whole video.

The cheapest of all is to decode only keyframes. Every I frame can be decoded alone, so one decode per kept frame. But you get the frames the encoder chose (every 8.3 s, plus scene cuts), not the frames you want.

Frame rate
naive: seek per target (201.4)GOP-aware single pass (201.5)decode all \(N\) frameskeyframes only
Figure 3. Frames the decoder must decode to return \(k\) uniformly spaced frames (formula 201.6), in the P-frames-only model of formulas (201.4) and (201.5). Both axes are logarithmic. The target slider moves in powers of two, from 1 to 4,096 frames. The decoder speed is your assumption: it changes the seconds, not the frame counts.
How to read Figure 3

Start at the default: a 10-minute 30 fps video, \(G = 250\), 32 targets. Each target lands in a different GOP, so the naive and the GOP-aware readers both decode 4,016 frames, about 4.5 times fewer than the 18,000 of a full pass. Now raise \(k\). The red naive line crosses the grey “decode all” line near \(k = 143\) and keeps rising; the blue GOP-aware line bends and stays below \(N\). Set \(G\) to 30 (a keyframe every second, common for streaming): seeking becomes cheap. The green keyframes line shows the cheapest possible reader: 72 frames, one per GOP. The jagged part at small \(k\) is real: it depends on where each target lands inside its GOP (at \(k = 6\) every target falls on a keyframe by chance).

Where does the decoding run? There are two common places.

  • CPU. Software decoders (ffmpeg’s libavcodec, used by PyAV, decord and torchvision) run on CPU cores. They use several threads per stream. The decoded frames then live in CPU memory, and the pipeline must resize them, convert YUV to RGB and copy them to the GPU. At 186.6 MB/s of RGB per 1080p30 stream, 100 streams need 18.7 GB/s, a large part of a PCIe 4.0 x16 link (about 32 GB/s per direction). [Likely]
  • GPU (NVDEC). NVIDIA GPUs have fixed-function video decoders called NVDEC, separate from the CUDA cores. The A100 has 5 NVDEC engines, the H100 has 7, the L4 has 4, the A10 and T4 have 2 each. [Certain: NVIDIA support matrix] Decoded frames stay in GPU memory, so only the compressed packets (0.63 MB per second at 5 Mbit/s) cross PCIe. Tools: ffmpeg with -hwaccel cuda, NVIDIA DALI, TorchCodec with a CUDA device. [Certain] NVDEC supports a fixed list of codecs and profiles per GPU generation; check the support matrix before you rely on it, for example for AV1. [Certain]
Worked example: decode versus the vision encoder

Take the Qwen2-VL vision encoder (32 layers, width 1,280) from session 197. Two 448×448 frames form one group of 1,024 patches and cost about 1.46 TFLOP, so 4.68 ms at the A100’s peak of 312 TFLOP/s.

Dense sampling. One second at 2 fps is one group: 4.68 ms of encoder time. Decoding that second of 1080p30 means 30 frames. Assume a decoder speed of 1,000 frames per second (on the 18-core laptop used to write this page, ffmpeg decoded our 1080p test file at roughly 900 to 1,200 frames per second with all cores, while other jobs were running; treat this as an order of magnitude). That is 30 ms, about 6 times the encoder bound.

Sparse sampling with a naive reader. 32 frames from a 10-minute video, \(G = 250\): 4,016 decoded frames, so 4.0 s of decoding against \(16 \times 4.68 = 75\) ms of encoder compute. Decoding is about 54 times larger. Real encoders run below peak, so the true ratio is smaller, but the order does not change.

Algorithm 1: a GOP-aware frame reader
  1. input: video file, target timestamps \(\tau_1, \dots, \tau_k\) in seconds
  2. read the stream index: PTS of every packet and which packets are keyframes // from the container, no decoding
  3. sort the targets; map each \(\tau_i\) to the frame whose PTS is closest // use time, not frame number: frame rates can vary
  4. group the targets by the keyframe that precedes them
  5. for each group, in file order:
  6.   seek to the group’s keyframe; decode forward
  7.   keep each decoded frame whose PTS matches a target; convert and resize only those // skip color conversion of the rest
  8.   stop after the last target of the group // formula (201.5)
  9. if the groups cover most GOPs: decode the whole file once instead
  10. return the kept frames with their PTS

Three more tricks are common in training pipelines. Re-encode the training set once with short GOPs (for example one keyframe per second), so every random clip is cheap to reach. Pre-extract frames at the training fps and store them as JPEG files or in a packed array, trading disk space for decode time. And decode at lower resolution: some decoders can skip work when you only need a thumbnail, and resizing on the GPU after NVDEC avoids a large copy. [Likely]

Sampling: which frames to keep

A model cannot read every frame, so we choose \(k\) of the \(N\) frames. This choice decides what the model can know. A short event between two sampled frames simply does not exist for the model. There are five common strategies.

1. Uniform (segment) sampling. Split the video into \(k\) equal segments and take the middle frame of each. Temporal Segment Networks (TSN) made this standard for action recognition; in training, a random frame inside each segment is taken instead of the middle, as data augmentation. [Certain]

\[ t_i \;=\; \Big\lfloor \Big(i + \tfrac12\Big) \frac{N}{k} \Big\rfloor, \qquad i = 0, 1, \dots, k-1 \tag{201.6} \]

Uniform sampling is simple, gives \(k\) frames for any length, and covers time evenly. But the gap between frames grows with the video: 32 frames over a 1-hour video means one frame every 112.5 seconds.

2. Fixed-rate (fps) sampling. Take a frame every \(1/f_s\) seconds, for example \(f_s = 2\). The gap is fixed, so a model can learn real speeds and durations, and timestamps are easy. But \(k = D \cdot f_s\) grows with the video, so long videos break the token budget. Qwen2-VL samples video at 2 frames per second. [Certain]

3. Keyframes only. Use the I frames from the file. This is the cheapest to decode (one decode per frame), and encoders put extra I frames at scene cuts, so they follow the content a little. But the count and positions are set by the encoder, not by your needs: a static 10-minute lecture may have 72 keyframes, a fast music video hundreds, and a file re-encoded with a fixed keyframe interval ignores the content completely.

4. Scene-based (shot-based) sampling. A shot is a run of frames from one continuous camera take; a cut is the jump between two shots. Detect cuts by comparing consecutive frames, then give every shot at least one frame. A simple detector compares small thumbnails:

\[ d_t \;=\; \big\lVert \mathbf{x}_t - \mathbf{x}_{t-1} \big\rVert, \qquad \text{cut at } t \ \text{ if } \ d_t \gt \theta \tag{201.7} \]
Read the symbols
\(\mathbf{x}_t\)
a cheap description of frame \(t\): a small thumbnail, a color histogram, or an embedding from a small network
\(d_t\)
the change from the previous frame, for example the mean absolute pixel difference
\(\theta\)
the cut threshold. Too low: camera motion and flashes look like cuts. Too high: soft cuts and fades are missed.

Tools such as PySceneDetect implement this with content and threshold detectors, plus a minimum shot length. [Certain] Scene-based sampling makes sure a 0.4-second insert shot is not missed. But it treats a long shot with a lot of action the same as a long static shot.

5. Adaptive (content-based) sampling. Put more frames where more changes. Accumulate the change over time and take frames at equal steps of the accumulated change:

\[ S(t) \;=\; \sum_{u=0}^{t} \big( \lambda + \min(d_u,\, c) \big), \qquad t_i \;=\; \min\Big\{\, t \;:\; S(t) \ge \Big(i + \tfrac12\Big) \frac{S(N-1)}{k} \Big\} \tag{201.8} \]
Read the symbols
\(S(t)\)
accumulated change up to frame \(t\); the frames are spread evenly along \(S\), not along time
\(\lambda\)
a small constant per frame, so that static parts still get some frames
\(c\)
a cap on one frame’s change, so that a single cut does not use up many frames

With \(c\) very small, every frame adds the same amount and this becomes uniform sampling. With \(\lambda\) small, the frames go where the motion is. The weak point is the opposite of uniform: a short static shot adds little change and can be skipped. More advanced methods choose frames by their relevance to the question (query-aware sampling, see session 210) or let the model ask for more frames (agents, see session 206).

How do we compare strategies without a model? One simple measure is coverage error: for every frame of the video, how far it is from the closest kept frame, in the same feature space as \(\mathbf{x}\):

\[ E \;=\; \frac{1}{N} \sum_{t=0}^{N-1} \min_{i} \big\lVert \mathbf{x}_t - \mathbf{x}_{t_i} \big\rVert \tag{201.9} \]

Low \(E\) means every moment of the video has a similar frame among the kept ones. Figure 4 uses a synthetic clip of 360 frames (12 s at 30 fps) with 7 shots. Each frame is a 3-number feature (shown as its color). Shot 2 lasts only 12 frames and shot 4 only 18 frames. Shot 5 has a fast camera pan in its middle.

Strategy
Figure 4. Four sampling strategies on a synthetic 360-frame clip. Top: one colored bar per frame, with the shot number above it. Middle: the change \(d_t\) from the previous frame (formula 201.7), cut off at 0.06 so the small changes are visible; tall spikes are cuts. The dashed line is the cut threshold \(\theta = 0.25\) (shown at the top because it is above the cut-off). Bottom: the kept frames. The table compares all four strategies at the current \(k\). Keyframes follow an x264-like rule: an I frame at every cut and at most 250 frames apart.
How to read Figure 4

At \(k = 8\), uniform sampling sees only 5 of 7 shots: the short shots 2 and 4 fall between samples. Keyframes see all 7 shots, but there are only 7 keyframes, so raising \(k\) does not help them, and their coverage error stays at 0.102 because they take the first frame of each shot and ignore the pan. Scene-based sampling sees all 7 shots with the lowest error at \(k = 8\) (0.034). Adaptive sampling puts several frames on the pan in shot 5 but often skips a short static shot; at \(k = 12\) it sees all 7 shots and has the lowest error (0.021). Now step \(k\) from 8 to 12 with uniform sampling: 5, 5, 7, 5 and 6 shots. Catching a short shot is luck of alignment. At large \(k\) all methods except keyframes become similar.

Algorithm 2: scene-based sampling with a frame budget
  1. input: decoded thumbnails \(\mathbf{x}_0, \dots, \mathbf{x}_{N-1}\) (low resolution, cheap), budget \(k\), threshold \(\theta\), minimum shot length \(L_{\min}\)
  2. compute \(d_t\) for all \(t\) (formula 201.7); mark a cut where \(d_t \gt \theta\) and the last cut is at least \(L_{\min}\) frames back
  3. shots \(\leftarrow\) runs of frames between cuts, with lengths \(L_1, \dots, L_S\)
  4. if \(k \lt S\): keep the middle frame of the \(k\) longest shots; return
  5. give every shot 1 frame; share the remaining \(k - S\) frames in proportion to \(L_s\) (largest remainders first)
  6. inside shot \(s\) with \(n_s\) frames: take the middles of \(n_s\) equal parts // formula (201.6) inside the shot
  7. return the frame indices and their PTS // keep timestamps for the model

Notice the cost: scene detection needs \(d_t\) for every frame, so it needs a full decode, often at low resolution or of keyframes and a fixed fraction of frames. In a large pipeline this is done once at ingest time and stored with the video. [Likely]

Tokens per second: the budget problem

Now the kept frames enter the model. A ViT cuts each frame into \(p \times p\) patches (see session 164). Many VLMs then merge each \(m \times m\) block of patches into one token, and some also merge \(t_p\) consecutive frames into one “temporal patch”. Qwen2-VL uses \(p = 14\), \(m = 2\) and \(t_p = 2\): its first layer is a 3D convolution over two frames, and a still image is treated as two identical frames. [Certain: paper and config.json] The tokens per second of video are then:

\[ \tau \;=\; \frac{f_s}{t_p} \cdot \Big\lfloor \frac{H'}{p\,m} \Big\rfloor \cdot \Big\lfloor \frac{W'}{p\,m} \Big\rfloor \quad \text{tokens per second} \tag{201.10} \]
Read the symbols
\(f_s\)
sampling rate: kept frames per second of video
\(H',\ W'\)
height and width of each frame after resizing, not of the original video
\(p\)
patch size of the ViT in pixels (14 for Qwen2-VL)
\(m\)
spatial merge: \(m \times m\) patches become one token (2 for Qwen2-VL)
\(t_p\)
temporal patch: consecutive frames merged into one token group (2 for Qwen2-VL, 1 for models that encode each frame alone)

With a video token budget \(B_v\), the longest video we can show at these settings, and the largest square frame side that fits a given duration, are:

\[ D_{\max} \;=\; \frac{B_v}{\tau}, \qquad s_{\text{fit}} \;=\; p\,m \,\Big\lfloor \sqrt{\frac{B_v\, t_p}{D\, f_s}} \Big\rfloor \tag{201.11} \]

Formula (201.11) shows the trade-off: for a fixed budget, the side of each frame shrinks with the square root of the duration. Doubling the length of the video means each frame side is divided by about 1.41. Qwen2-VL limits each video to 16,384 tokens and lowers the frame resolution of long videos to stay under that limit. [Certain] Try it in Figure 5.

Temporal patch \(t_p\)
Video token budget
Figure 5. Token cost of a video (formula 201.10) with patch size 14 and 2×2 merging, as in Qwen2-VL, against a video token budget (formula 201.11). The bar is on a log scale. The 16K budget is Qwen2-VL’s per-video limit; the others are common context sizes, before any room for text.
How to read Figure 5

The default is a 10-minute video at 2 fps and 448×448: 256 tokens per second and 153,600 tokens, 9.4 times the 16K budget. At 448×448, 16K tokens last only 64 seconds. To keep all 1,200 frames of the 10 minutes, each frame side must drop to 140 pixels (\(5 \times 5 = 25\) tokens per pair of frames): 15,000 tokens. Lower the sampling rate instead: at 0.25 fps, 448×448 fits 512 seconds. Switch the temporal patch to 1 frame: every number doubles.

Worked example: one minute, ten minutes, one hour

Qwen2-VL style (\(p = 14\), \(m = 2\), \(t_p = 2\), 2 fps, 448×448): each pair of frames gives \(32 \times 32 / 4 = 256\) tokens, and there is one pair per second, so \(\tau = 256\). One minute: 15,360 tokens, just under the 16,384 limit. Ten minutes: 153,600 tokens. To fit 10 minutes into 16,384 tokens at 2 fps: \(\sqrt{16{,}384 \times 2 / (600 \times 2)} = 5.2\), so 5 merged tokens per side, a 140×140 frame and 15,000 tokens. Frames that small cannot show text or faces well. Lowering the frame rate keeps the detail but loses events.

Gemini API. Google’s documentation counts about 263 tokens per second of video at the default sampling of 1 frame per second, plus 32 tokens per second of audio, and gives lower rates (about 100 per second) for a low-resolution setting. [Certain, as documented in 2026; the numbers change between model versions] At 263 + 32 = 295 tokens per second, a 1-million-token context holds about 56 minutes of video with audio, before any text.

The lesson. At about 250 tokens per second, the budget runs out quickly. Real systems answer with lower resolution per frame, lower fps, stronger token merging, or by retrieving only the relevant parts of a long video.

Algorithm 3: plan the frames for a token budget
  1. input: duration \(D\), budget \(B_v\), model constants \(p, m, t_p\), minimum useful side \(s_{\min}\) (for example 224), maximum fps \(f_{\max}\)
  2. \(f_s \leftarrow f_{\max}\); \(s \leftarrow s_{\text{fit}}(D, f_s, B_v)\) // formula (201.11)
  3. while \(s \lt s_{\min}\) and \(f_s\) can go lower: halve \(f_s\); recompute \(s\) // trade events for detail
  4. if \(s\) is still below \(s_{\min}\): switch to uniform sampling of \(k = \lfloor B_v\, t_p / \text{tokens per group at } s_{\min} \rfloor\) frames, or split the video and retrieve the relevant parts
  5. choose the frame positions (uniform, scene-based or adaptive) and read them with Algorithm 1
  6. return frames, their timestamps and the side \(s\) // pass timestamps so the model knows real time

Timestamps matter. With uniform sampling, the gap between frames depends on the video length, so the model cannot know real time unless you tell it. Qwen2.5-VL adds absolute time encoding so it can locate events to the second in long videos. [Certain] Simpler systems write the timestamp next to each frame as text.

Code: seek cost, sampling and token budget in PyTorch

The code below is deterministic and runs on a CPU in under a second. Part 1 counts decoded frames for the naive and GOP-aware readers (formulas 201.4 and 201.5). Part 2 builds a tiny synthetic video, detects cuts and compares three samplers. Part 3 counts tokens the Qwen2-VL way and finds the frame size that fits a budget.

import math
import torch
torch.manual_seed(0)

# 1) Seek cost: frames a decoder must decode to return k target frames
def decode_cost(targets, G, smart=True):
    """targets: sorted frame indices; G: keyframe interval (P-frames only).
    naive: every target restarts at its keyframe; smart: one pass per GOP."""
    if not smart:
        return sum(t % G + 1 for t in targets)
    last = {}
    for t in targets:
        last[t // G] = max(last.get(t // G, -1), t % G)
    return sum(v + 1 for v in last.values())

def uniform_indices(N, k):
    """Center of each of k equal segments (TSN-style)."""
    return [int((i + 0.5) * N / k) for i in range(k)]

N, G = 10 * 60 * 30, 250              # 10 minutes at 30 fps, x264 default keyint
for k in (8, 32, 1200):
    t = uniform_indices(N, k)
    print(f"k={k:5d}  naive {decode_cost(t, G, False):7d}  "
          f"smart {decode_cost(t, G):6d}  decode-all {N}")

# 2) A tiny synthetic video: 5 shots of different colors, plus noise
lengths = [40, 6, 30, 8, 36]                       # frames per shot
colors = torch.rand(len(lengths), 3, 1, 1)
frames = torch.cat([c.expand(L, 3, 32, 32) for c, L in zip(colors, lengths)])
frames = (frames + 0.02 * torch.randn_like(frames)).clamp(0, 1)   # (T, 3, H, W)
T = frames.shape[0]

def frame_diff(x):
    """Mean absolute difference between consecutive frames (d_0 = 0)."""
    small = torch.nn.functional.avg_pool2d(x, 4)    # cheap: compare 8x8 thumbnails
    d = (small[1:] - small[:-1]).abs().mean(dim=(1, 2, 3))
    return torch.cat([torch.zeros(1), d])

d = frame_diff(frames)
cuts = [0] + (d > 0.1).nonzero().flatten().tolist()
print("detected cuts:", cuts)

def scene_sample(cuts, T, k):
    """One frame per shot (middle), extra frames to long shots."""
    bounds = cuts + [T]
    L = torch.tensor([b - a for a, b in zip(bounds[:-1], bounds[1:])], dtype=torch.float)
    n = torch.ones(len(L), dtype=torch.long)
    for _ in range(k - len(L)):                      # give each extra frame to the
        n[torch.argmax(L / n)] += 1                  # shot with most frames per sample
    out = []
    for a, Ls, ns in zip(bounds[:-1], L.long().tolist(), n.tolist()):
        out += [a + int((j + 0.5) * Ls / ns) for j in range(ns)]
    return out

def adaptive_sample(d, k, lam=0.01, cap=0.05):
    """Equal steps of cumulative change: busy parts get more frames."""
    S = torch.cumsum(lam + d.clamp(max=cap), 0)
    targets = (torch.arange(k) + 0.5) * S[-1] / k
    return torch.searchsorted(S, targets).tolist()

shot = torch.repeat_interleave(torch.arange(len(lengths)), torch.tensor(lengths))
for name, idx in [("uniform", uniform_indices(T, 6)),
                  ("scene", scene_sample(cuts, T, 6)),
                  ("adaptive", adaptive_sample(d, 6))]:
    print(f"{name:8s} {idx}  shots seen: {len(set(shot[idx].tolist()))} of {len(lengths)}")

# 3) Token budget: Qwen2-VL-style counting (patch 14, merge 2x2, 2 frames per temporal patch)
def video_tokens(seconds, fps, side, patch=14, merge=2, t_patch=2):
    groups = math.ceil(seconds * fps / t_patch)
    per_group = (side // (patch * merge)) ** 2
    return groups * per_group

def fit_side(seconds, fps, budget, patch=14, merge=2, t_patch=2):
    """Largest square side (multiple of 28) that keeps the video under budget."""
    groups = math.ceil(seconds * fps / t_patch)
    cells = math.floor(math.sqrt(budget / groups))
    return cells * patch * merge

print("1 min at 2 fps, 448x448:", video_tokens(60, 2, 448), "tokens")
print("10 min at 2 fps under 16384 tokens -> side", fit_side(600, 2, 16384),
      "->", video_tokens(600, 2, fit_side(600, 2, 16384)), "tokens")

# Output (PyTorch 2.8, CPU):
# k=    8  naive    1008  smart   1008  decode-all 18000
# k=   32  naive    4016  smart   4016  decode-all 18000
# k= 1200  naive  150600  smart  17496  decode-all 18000
# detected cuts: [0, 40, 46, 76, 84]
# uniform  [10, 30, 50, 70, 90, 110]  shots seen: 3 of 5
# scene    [10, 30, 43, 61, 80, 102]  shots seen: 5 of 5
# adaptive [11, 32, 48, 70, 86, 109]  shots seen: 3 of 5
# 1 min at 2 fps, 448x448: 15360 tokens
# 10 min at 2 fps under 16384 tokens -> side 140 -> 15000 tokens

Part 2 shows the same pattern as Figure 4: with 6 frames for 5 shots, uniform and adaptive sampling both miss the 6-frame and 8-frame shots, and only the scene-based sampler sees all of them. (The scene sampler here gives extra frames to the shot with the most frames per sample; Algorithm 2 uses largest remainders. Both give one frame per shot first.) To read real frames, replace part 2’s synthetic tensor with a decoder such as TorchCodec, PyAV or decord, and pass timestamps rather than frame numbers.

Where this is used

ApplicationTypical frame choiceWhat dominates the cost
Video question answering with a Video-LLMuniform 8 to 64 frames, or 1 to 2 fps with lower resolution; timestamps in the prompttokens (prefill) and decode of the frames; see session 205
Action recognitionshort clips of 8 to 32 frames, uniform segments (TSN) or dense clips at a fixed stridedecoding of random clips during training; see session 203
Text-to-video retrievala few frames or one embedding per shot, computed once at indexingdecoding the whole library once; see session 208
Key-moment detection and temporal groundingfixed fps (1 to 2), because the answer is a time; short clips around candidateslong sequences of frame features; see session 207
Video fingerprinting and near-duplicate detectionkeyframes or fixed fps at very low resolutiondecode throughput over millions of videos; keyframe-only decoding helps; see session 209
Video object segmentation and trackingevery frame (or every second frame) at full resolutionno sampling shortcut: dense decode and per-frame compute; see session 204
Video-RAG over long videosshots detected at ingest, frames and captions per shot stored in an indexone full decode at ingest, then cheap retrieval; see session 210

Common mistakes

  • Seeking for every frame. A loop that calls “seek to time t, read one frame” for 1,000 timestamps can decode the same GOP hundreds of times (formula 201.4). Sort the targets and use one pass per GOP, or decode the file once.
  • Using frame numbers instead of timestamps. Phone videos often have a variable frame rate. Frame \(n\) is then not at time \(n/f\). Use the PTS of each frame for sampling and for the timestamps you give the model.
  • Ignoring rotation metadata. Phones store portrait video as landscape plus a rotation flag in the container. If the reader ignores it, the model sees sideways frames. [Likely]
  • Wrong color conversion. YUV to RGB needs the right matrix (BT.601 or BT.709) and range (limited 16–235 or full 0–255). A mismatch makes colors washed out or too dark, and the frames differ from what the encoder saw in training. [Likely]
  • Training and serving with different sampling. A model trained on 8 uniform frames and served at 1 fps sees a different time scale. Keep the sampler, fps and resolution the same, or train with several.
  • Fast but inaccurate seeking. Some readers jump to the nearest keyframe for speed and return that frame. Your “frame at 5.0 s” may be from 0.2 s or 8 s earlier. Check the PTS of what you get.
  • Counting tokens per frame and forgetting the temporal patch. With \(t_p = 2\), an odd number of frames is padded (Qwen2-VL repeats a frame), and the token count is per pair of frames, not per frame. [Likely]
  • Blaming the GPU. Low GPU use in a video job usually means the GPU is waiting for decoded frames. Profile the decoder first.

Interview questions

What are I, P and B frames, and why does the file store frames out of order?basic
An I frame is compressed alone and can be decoded alone. A P frame stores motion and a correction relative to an earlier frame. A B frame is predicted from an earlier and a later frame. Because a B frame needs a later frame, that later frame must be decoded first, so the file stores frames in decode order (for example I P B B B for the display order I B B B P). Each frame has a decode timestamp (DTS) and a presentation timestamp (PTS); use the PTS for time.
What is a GOP, and how does its length affect random access?basic
A GOP is a keyframe plus the frames that follow until the next keyframe. Decoding can start only at a keyframe, so getting frame \(j\) of a GOP means decoding the frames before it in decode order. For a random target the average is \((G+1)/2\) frames (formula 201.3): 125.5 frames for x264’s default maximum of 250. Short GOPs (for example one keyframe per second) make seeking cheap but make the file larger, because I frames are big.
You sample 32 frames from a 10-minute 30 fps video with keyframes every 250 frames. How many frames are decoded?core
The video has 18,000 frames and 72 GOPs. With uniform targets, each of the 32 falls in a different GOP, so each costs its position in the GOP plus one: 4,016 frames in total (about \(32 \times 125.5\)). That is 125 decoded frames per kept frame, but still 4.5 times less than decoding all 18,000. Keyframes only would cost one decode per frame, but the frames would be wherever the encoder put I frames.
When is decoding the whole video cheaper than seeking?core
A naive reader that seeks per target costs about \(k(G+1)/2\) frames; a full pass costs \(N\). They are equal at \(k \approx 2N/(G+1)\): about 143 targets for 18,000 frames and \(G = 250\). Above that, seeking repeats the same GOPs. A GOP-aware reader, which decodes each GOP once up to its last target, is never worse than a full pass, so it is the right default (formula 201.5).
Your video Video-LLM pipeline shows 15% GPU utilization. What do you check?core
Decoding first. Measure frames decoded per second and the decoded-to-kept ratio. Check for per-frame seeking, CPU threads per stream, YUV-to-RGB conversion and resizing on the CPU, and the copy of raw frames to the GPU (186.6 MB/s per 1080p30 stream). Fixes: a GOP-aware reader, GPU decoding with NVDEC so frames stay on the GPU, resizing on the GPU, more parallel decode workers, re-encoding the dataset with short GOPs, or pre-extracting frames. Then check data loader overlap (prefetching) so decoding runs while the GPU computes.
Compare uniform, fixed-fps, keyframe, scene-based and adaptive sampling.core
Uniform: fixed \(k\) for any length, even time coverage, but the gap grows with length and short shots can be missed. Fixed fps: constant time gap, good for timing and grounding, but \(k\) grows with length and breaks the token budget. Keyframes: cheapest to decode, but count and positions are set by the encoder. Scene-based: at least one frame per shot, so short shots are kept, but needs a full (cheap) pass for cut detection and treats busy and static shots alike. Adaptive: more frames where content changes, but can skip short static shots. In practice: scene detection at ingest plus uniform or adaptive allocation inside shots, with timestamps.
How many tokens does one minute of video cost in Qwen2-VL at 448×448 and 2 fps?basic
Patch 14 gives 32×32 patches per frame; 2×2 merging gives 256 tokens; the 3D patch merges 2 frames, so 256 tokens per pair of frames. At 2 fps that is one pair per second: 256 tokens per second, 15,360 per minute (formula 201.10). That is just under Qwen2-VL’s 16,384-token per-video limit, so longer videos get lower resolution.
You must fit a 10-minute video into 16K tokens. What are your options?advanced
At 2 fps and 448×448 it needs 153,600 tokens, 9.4 times too many. Options: (1) lower resolution: formula 201.11 gives a 140×140 frame at 2 fps, which loses text and small objects; (2) lower fps: 0.25 fps at 448 fits 512 seconds, so about 0.2 fps for 10 minutes, which loses short events; (3) stronger token merging or pooling per frame; (4) choose frames by content (scene-based or adaptive) instead of by time; (5) retrieve only the parts relevant to the question (Video-RAG) or let an agent look at parts on demand. Usually a mix: moderate fps, moderate resolution, and query-aware selection.
Why is NVDEC useful even if CPU decoding is fast enough?advanced
Three reasons. It frees CPU cores for the rest of the data pipeline (tokenization, augmentation, networking). Decoded frames stay in GPU memory, so only compressed packets cross PCIe instead of hundreds of MB/s of raw frames per stream. And resize and color conversion can run on the GPU right after. The limits: a fixed number of engines per GPU (5 on A100, 7 on H100, 4 on L4), a fixed list of supported codecs and profiles per generation, and GPU memory used by decoded frames.
Why should you pass timestamps to a Video-LLM?advanced
With uniform sampling, the gap between frames changes with video length: 8 frames of a 16-second clip are 2 s apart, 8 frames of a 16-minute video are 2 minutes apart. Without timestamps the model cannot tell speed, duration or when something happened, and it cannot answer “at what time” questions. Fixed-fps sampling fixes the gap, and models such as Qwen2.5-VL add absolute time encoding; simpler systems write each frame’s timestamp as text next to its tokens.

Exercises

  1. Using formula (201.1), compute the raw data rate of 4K video (3840×2160) at 60 fps in YUV 4:2:0. How much is that compressed at 20 Mbit/s?
  2. In Figure 2, set \(G = 16\) and B frames = 1. What is the minimum number of frames to decode B13, and how many does a simple decode loop decode?
  3. A 1-hour, 30 fps video has keyframes every 250 frames. You sample 64 frames uniformly with a naive reader. How many frames are decoded? Check with Figure 3.
  4. In Figure 4 with uniform sampling, step \(k\) from 8 to 12 and write down the number of shots seen. Why does the count go down when \(k\) goes up? Which \(k\) guarantees that every shot is seen?
  5. A model uses \(p = 14\), \(m = 2\), \(t_p = 1\), 1 fps and 448×448. How many tokens does one hour of video cost?
  6. With formula (201.11), what is the largest square frame for a 5-minute video at 1 fps and \(t_p = 2\) in a 32,768-token budget?
Answerscheck
1. \(3840 \times 2160 \times 1.5 \times 60 = 746.5\) MB/s, which is 5,972 Mbit/s; at 20 Mbit/s the ratio is about 299. 2. With 1 B frame, anchors are at 0, 2, 4, …, 14 and 15. B13 needs P12 and P14, so the chain is I0, P2, P4, …, P14 (8 anchors) plus B13: 9 frames. In decode order (I0 P2 B1 P4 B3 … P14 B13 P15), B13 is the 15th frame, so a simple loop decodes 14 frames before it and 15 in total. 3. \(N = 108{,}000\), 432 GOPs. The 64 targets fall in different GOPs; the sum of their positions plus one is 8,032 frames (about \(64 \times 125.5\)). 4. 5, 5, 7, 5, 6 shots for \(k = 8\) to 12. The kept frames are the centers of \(k\) equal segments, so they move when \(k\) changes; a 12-frame shot is caught only when one center happens to land inside it. A guarantee needs a gap \(N/k\) no larger than the shortest shot: \(360/k \le 12\), so \(k = 30\) or more. Scene-based sampling gets all shots with \(k = 7\). 5. 256 tokens per frame, 1 frame per second: \(256 \times 3{,}600 = 921{,}600\) tokens, almost a 1M context. 6. \(\sqrt{32{,}768 \times 2 / (300 \times 1)} = 14.8\), so 14 merged tokens per side and a 392×392 frame (150 groups of 196 tokens: \(150 \times 196 = 29{,}400\) tokens).

Key takeaways

  • A video is frames with time. Raw 1080p30 is 93 MB/s in YUV 4:2:0 and 187 MB/s in RGB; the file is about 100 times smaller because of compression (201.1, 201.2).
  • Codecs store I frames alone and P and B frames as changes. Decoding can start only at a keyframe, so one random frame costs about \((G+1)/2\) decoded frames (201.3).
  • The decoder pays for every frame in the decode chain, not for the frames you keep. Seeking per frame can cost more than decoding the whole file; a GOP-aware reader never does (201.4, 201.5).
  • Decoding is often the real bottleneck, larger than the vision encoder. Use GPU decoding (NVDEC), short GOPs for training data, or pre-extracted frames.
  • Uniform, fixed-fps, keyframe, scene-based and adaptive sampling miss different things. Short shots need scene detection; busy shots need adaptive allocation (201.6–201.9).
  • Tokens per second are set by fps, resolution, patch size and merging (201.10). At about 256 tokens per second, budgets run out in a minute, and frame size shrinks with the square root of duration (201.11).

Key formulas

\[ R_{\text{raw}} = W H\, b_{\text{px}}\, f, \qquad \E[\text{decoded per random frame}] = \frac{G+1}{2}, \qquad C_{\text{naive}} \approx k\,\frac{G+1}{2} \]
\[ t_i = \Big\lfloor \Big(i+\tfrac12\Big)\frac{N}{k} \Big\rfloor, \qquad \tau = \frac{f_s}{t_p}\Big\lfloor \frac{H'}{pm} \Big\rfloor \Big\lfloor \frac{W'}{pm} \Big\rfloor, \qquad s_{\text{fit}} = p\,m \Big\lfloor \sqrt{\frac{B_v\, t_p}{D f_s}} \Big\rfloor \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  2. Tang et al. (2023). Video Understanding with Large Language Models: A Survey
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  4. Tang et al. (2023). Video Understanding with Large Language Models: A Survey
  5. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  6. Bai et al. (2025). Qwen2.5-VL Technical Report
  7. Wang, Xiong, Wang, Qiao, Lin, Tang & Van Gool (2016). Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
  8. NVIDIA (2025). Video Encode and Decode GPU Support Matrix
  9. VideoLAN (2024). x264: free H.264/AVC encoder (command-line help, default settings)
  10. Google (2026). Gemini API documentation: understand and count tokens
  11. PyTorch team (2025). TorchCodec: media decoding and encoding for PyTorch, on CPU and GPU
  12. Castellano (2024). PySceneDetect: video scene cut detection and analysis tool