Video as data
Before a model can watch a video, someone must open the file, decode compressed frames, pick a few of them and turn them into tokens. This session explains what a video file really contains (frames, codecs, I, P and B frames), why decoding is often the slowest part of a video pipeline, how to choose which frames to keep, and how many tokens one second of video costs a vision-language model.
What you will learn
- What a video is as data: frames, frame rate, resolution, YUV 4:2:0 color, and why raw video is about 100 times larger than the file
- How codecs compress video with I, P and B frames, what a GOP is, and why decode order differs from display order
- Why random access into a video is expensive, how many frames a decoder must decode to return one frame, and when one sequential pass is cheaper than seeking
- Where decoding runs (CPU, NVDEC on the GPU) and why decoding can cost more than the vision encoder
- Frame sampling strategies: uniform, fixed fps, keyframes, scene-based and adaptive, and what each one misses
- Tokens per second of video: how patch size, token merging, temporal patches, fps and resolution set the token budget, with Qwen2-VL and Gemini numbers
Why this matters
A vision-language model (VLM) never sees a video file. It sees a few frames, cut into patches and turned into tokens. Between the file and the tokens there is a chain of steps: open the container, decode compressed frames, choose which frames to keep, resize them, and count how many tokens they will cost. Each step can be slow, wrong or expensive. This session explains that chain from the bottom up.
Three problems appear in every real video system, from a Video-LLM that answers questions to a search engine that indexes millions of clips:
- The data is huge. One second of 1080p video at 30 frames per second is 187 MB as RGB pixels. The file holds the same second in less than 1 MB, because it is compressed.
- Compression makes access expensive. Most frames in the file are stored as “changes from another frame”. To get one frame back, the decoder must first rebuild the frames it depends on. Picking 32 frames from a 10-minute video can mean decoding thousands of frames.
- Tokens are the scarce resource. A model that spends 256 tokens on each second of video fills a 16K-token budget in about one minute. So we must choose which frames, at which resolution, the model will see.
This is the first session of the Track 8 track. In session 196 we followed one image through a VLM; here the input is a stream of images with time. The next session, session 202, takes the sampled frames and asks how a network should model motion. Later sessions build on the budget problem: session 206 for hours of footage, session 207 for finding moments, and session 211 for running all of this at scale. The book chapter on video-language models (Noyan et al., chapter 9) and the survey by Tang et al. give the wider picture.
What a video is: frames, frame rate, resolution and color
A video is a sequence of still images called frames, shown quickly one after another. Three numbers describe it:
- Resolution \(W \times H\): the width and height of each frame in pixels. 1080p means 1920×1080; 4K (UHD) means 3840×2160.
- Frame rate \(f\): frames per second (fps). Film uses 24, TV and phones 25, 30 or 60.
- Duration \(D\): seconds. So the video has \(N = D \cdot f\) frames.
Pixels are usually not stored as RGB. Video uses YUV: one brightness channel (Y, called luma) and two color-difference channels (U and V, called chroma). Our eyes see detail in brightness much better than in color, so codecs store the color channels at lower resolution. In the common 4:2:0 format, U and V have half the width and half the height of Y. Each pixel then costs \(1 + \tfrac14 + \tfrac14 = 1.5\) bytes at 8 bits per sample, instead of 3 bytes for RGB. A vision encoder wants RGB, so the pipeline converts YUV to RGB at some point.
The raw data rate of a video, before compression, is:
- Read the symbols
- \(W,\ H\)
- frame width and height in pixels
- \(b_{\text{px}}\)
- bytes per pixel: 1.5 for 8-bit YUV 4:2:0, 3 for 8-bit RGB
- \(f\)
- frame rate in frames per second
A file stores the video at an encoded bit rate \(R_{\text{enc}}\) (for example 5 Mbit/s). The compression ratio compares the two:
For 1080p at 30 fps in YUV 4:2:0, \(R_{\text{raw}} = 1920 \times 1080 \times 1.5 \times 30 = 93.3\) MB/s, which is 746 Mbit/s. A 5 Mbit/s stream is then compressed about 149 times. The video lives in a container file (MP4, MKV, WebM, MOV). The container holds one or more streams (video, audio, subtitles) as small chunks called packets, plus timestamps and metadata such as rotation. The codec is the method that compresses each video stream: H.264 (also called AVC), H.265 (HEVC), VP9 or AV1. “MP4” says nothing about the codec; an MP4 file can hold H.264 or AV1.
Raw. One 1080p RGB frame is \(1920 \times 1080 \times 3 = 6{,}220{,}800\) bytes (6.2 MB). One hour at 30 fps has 108,000 frames, so 672 GB of RGB pixels. The same hour at 5 Mbit/s is a 2.25 GB file.
A real encode. We encoded 30 seconds of a synthetic 1080p, 30 fps test pattern (ffmpeg testsrc2) with the x264 encoder at its default settings. The file is 23,945,749 bytes, so 6.39 Mbit/s. The raw YUV 4:2:0 data is \(900 \times 3{,}110{,}400\) bytes = 2.80 GB, so the compression ratio (formula 201.2) is 117. Of the 900 frames, ffprobe reports 4 I frames, 419 P frames and 477 B frames. The I frames are at frames 0, 250, 500 and 750: one every 250 frames, which is the x264 default maximum keyframe interval. [Certain: measured with ffmpeg 9.0 and x264 defaults --keyint 250 --bframes 3 --scenecut 40]
Codecs and GOPs: I, P and B frames
Neighboring frames look almost the same. A codec uses this. It stores only a few frames completely and describes the others as changes. There are three frame types:
- I frame (intra frame): compressed alone, like a JPEG. It can be decoded without any other frame. It is the largest type.
- P frame (predicted frame): stores motion vectors and a small correction relative to an earlier frame. “Take this block from the previous frame, move it 3 pixels left, add this small difference.” To decode it, the earlier frame must already be decoded.
- B frame (bi-directional frame): predicted from an earlier and a later frame. It is usually the smallest. To decode it, both of its reference frames must be decoded first, including the later one.
A GOP (group of pictures) is an I frame plus all the frames that follow it until the next I frame. A keyframe is an I frame where decoding can start from scratch (in H.264 this is an IDR frame: no later frame may look back past it). [Certain] The GOP length \(G\) is the keyframe interval. With x264 defaults, \(G\) is at most 250 frames, so at 30 fps a keyframe comes at least every 8.33 seconds; the encoder also inserts extra I frames at scene cuts. [Certain]
Because a B frame needs a later frame, the file stores frames in decode order, not in display order. For the display order I B B B P, the file stores I P B B B. Each frame therefore has two timestamps: the DTS (decode timestamp) and the PTS (presentation timestamp, when it is shown). Use the PTS to know when a frame happens in the video.
Now the key cost. To get frame \(j\) of a GOP (counting from the keyframe at position 0), the decoder must start at the keyframe. With only P frames, every frame depends on the one before it, so the decoder must decode frames \(0, 1, \dots, j\): that is \(j+1\) frames. A simple decode loop does the same even with B frames: it decodes everything in decode order until frame \(j\) comes out. If the target position is random, the average cost is:
- Read the symbols
- \(G\)
- GOP length: frames from one keyframe to the next
- \(j\)
- position of the target frame inside its GOP (in decode order); 0 is the keyframe
For \(G = 250\), that is 125.5 frames decoded to return one frame. A smarter decoder can skip B frames that no other frame uses (non-reference B frames) and decode only the chain of I and P frames that the target needs. In our simple model below (B frames are never used as references), that minimum is smaller. Real H.264 encoders often do use some B frames as references (x264 does this by default with its “B-pyramid”), so the true minimum is between the two. [Likely] Explore the chain in Figure 2.
How to read Figure 2
The default shows \(G = 16\) with 3 B frames between anchors, like x264’s default B-frame count. Frame 14 is a B frame selected. Its references are P12 (before it) and P15 (after it); P15 needs P12, P12 needs P8, and so on back to I0. So the minimum is 6 frames: I0, P4, P8, P12, P15 and B14. A simple decode loop goes through the bottom row from the left until B14 comes out: 16 frames. The readout also gives the averages over the GOP: the simple loop always averages \((G+1)/2\) (formula 201.3), whatever the number of B frames. Set B frames to 0: every frame now needs all frames before it.
Decoding is the hidden bottleneck
A Video-LLM keeps maybe 8 to 64 frames of a clip. People then think the cost is 8 to 64 frames. It is not: the decoder pays for every frame in each decode chain. Suppose we want \(k\) target frames at positions \(t_1 \lt t_2 \lt \dots \lt t_k\) in a video of \(N\) frames with keyframes every \(G\) frames. A naive reader seeks to each target separately: it jumps to the keyframe before \(t_i\) and decodes up to \(t_i\). With only P frames:
A GOP-aware reader sorts the targets and decodes each GOP at most once, from its keyframe up to the last target inside it:
- Read the symbols
- \(t_i\)
- frame index of the \(i\)-th target frame; \(t_i \bmod G\) is its position inside its GOP
- \(\lfloor t_i / G \rfloor\)
- the number of the GOP that contains target \(i\)
- \(N\)
- all frames of the video: the cost of decoding everything once, in order
Two regimes follow. When the targets are few and far apart (one per GOP or less), each costs about \((G+1)/2\) frames, and seeking beats decoding everything. When there are many targets, the naive reader decodes the same GOP again and again, and its cost passes \(N\) at about \(k \approx 2N/(G+1)\) targets. For a 10-minute video at 30 fps (\(N = 18{,}000\)) and \(G = 250\), that break-even is about 143 targets. Sampling at 2 fps means 1,200 targets: the naive reader then decodes 150,600 frames, more than 8 times the whole video.
The cheapest of all is to decode only keyframes. Every I frame can be decoded alone, so one decode per kept frame. But you get the frames the encoder chose (every 8.3 s, plus scene cuts), not the frames you want.
How to read Figure 3
Start at the default: a 10-minute 30 fps video, \(G = 250\), 32 targets. Each target lands in a different GOP, so the naive and the GOP-aware readers both decode 4,016 frames, about 4.5 times fewer than the 18,000 of a full pass. Now raise \(k\). The red naive line crosses the grey “decode all” line near \(k = 143\) and keeps rising; the blue GOP-aware line bends and stays below \(N\). Set \(G\) to 30 (a keyframe every second, common for streaming): seeking becomes cheap. The green keyframes line shows the cheapest possible reader: 72 frames, one per GOP. The jagged part at small \(k\) is real: it depends on where each target lands inside its GOP (at \(k = 6\) every target falls on a keyframe by chance).
Where does the decoding run? There are two common places.
- CPU. Software decoders (ffmpeg’s libavcodec, used by PyAV, decord and torchvision) run on CPU cores. They use several threads per stream. The decoded frames then live in CPU memory, and the pipeline must resize them, convert YUV to RGB and copy them to the GPU. At 186.6 MB/s of RGB per 1080p30 stream, 100 streams need 18.7 GB/s, a large part of a PCIe 4.0 x16 link (about 32 GB/s per direction). [Likely]
- GPU (NVDEC). NVIDIA GPUs have fixed-function video decoders called NVDEC, separate from the CUDA cores. The A100 has 5 NVDEC engines, the H100 has 7, the L4 has 4, the A10 and T4 have 2 each. [Certain: NVIDIA support matrix] Decoded frames stay in GPU memory, so only the compressed packets (0.63 MB per second at 5 Mbit/s) cross PCIe. Tools: ffmpeg with
-hwaccel cuda, NVIDIA DALI, TorchCodec with a CUDA device. [Certain] NVDEC supports a fixed list of codecs and profiles per GPU generation; check the support matrix before you rely on it, for example for AV1. [Certain]
Take the Qwen2-VL vision encoder (32 layers, width 1,280) from session 197. Two 448×448 frames form one group of 1,024 patches and cost about 1.46 TFLOP, so 4.68 ms at the A100’s peak of 312 TFLOP/s.
Dense sampling. One second at 2 fps is one group: 4.68 ms of encoder time. Decoding that second of 1080p30 means 30 frames. Assume a decoder speed of 1,000 frames per second (on the 18-core laptop used to write this page, ffmpeg decoded our 1080p test file at roughly 900 to 1,200 frames per second with all cores, while other jobs were running; treat this as an order of magnitude). That is 30 ms, about 6 times the encoder bound.
Sparse sampling with a naive reader. 32 frames from a 10-minute video, \(G = 250\): 4,016 decoded frames, so 4.0 s of decoding against \(16 \times 4.68 = 75\) ms of encoder compute. Decoding is about 54 times larger. Real encoders run below peak, so the true ratio is smaller, but the order does not change.
- input: video file, target timestamps \(\tau_1, \dots, \tau_k\) in seconds
- read the stream index: PTS of every packet and which packets are keyframes // from the container, no decoding
- sort the targets; map each \(\tau_i\) to the frame whose PTS is closest // use time, not frame number: frame rates can vary
- group the targets by the keyframe that precedes them
- for each group, in file order:
- seek to the group’s keyframe; decode forward
- keep each decoded frame whose PTS matches a target; convert and resize only those // skip color conversion of the rest
- stop after the last target of the group // formula (201.5)
- if the groups cover most GOPs: decode the whole file once instead
- return the kept frames with their PTS
Three more tricks are common in training pipelines. Re-encode the training set once with short GOPs (for example one keyframe per second), so every random clip is cheap to reach. Pre-extract frames at the training fps and store them as JPEG files or in a packed array, trading disk space for decode time. And decode at lower resolution: some decoders can skip work when you only need a thumbnail, and resizing on the GPU after NVDEC avoids a large copy. [Likely]
Sampling: which frames to keep
A model cannot read every frame, so we choose \(k\) of the \(N\) frames. This choice decides what the model can know. A short event between two sampled frames simply does not exist for the model. There are five common strategies.
1. Uniform (segment) sampling. Split the video into \(k\) equal segments and take the middle frame of each. Temporal Segment Networks (TSN) made this standard for action recognition; in training, a random frame inside each segment is taken instead of the middle, as data augmentation. [Certain]
Uniform sampling is simple, gives \(k\) frames for any length, and covers time evenly. But the gap between frames grows with the video: 32 frames over a 1-hour video means one frame every 112.5 seconds.
2. Fixed-rate (fps) sampling. Take a frame every \(1/f_s\) seconds, for example \(f_s = 2\). The gap is fixed, so a model can learn real speeds and durations, and timestamps are easy. But \(k = D \cdot f_s\) grows with the video, so long videos break the token budget. Qwen2-VL samples video at 2 frames per second. [Certain]
3. Keyframes only. Use the I frames from the file. This is the cheapest to decode (one decode per frame), and encoders put extra I frames at scene cuts, so they follow the content a little. But the count and positions are set by the encoder, not by your needs: a static 10-minute lecture may have 72 keyframes, a fast music video hundreds, and a file re-encoded with a fixed keyframe interval ignores the content completely.
4. Scene-based (shot-based) sampling. A shot is a run of frames from one continuous camera take; a cut is the jump between two shots. Detect cuts by comparing consecutive frames, then give every shot at least one frame. A simple detector compares small thumbnails:
- Read the symbols
- \(\mathbf{x}_t\)
- a cheap description of frame \(t\): a small thumbnail, a color histogram, or an embedding from a small network
- \(d_t\)
- the change from the previous frame, for example the mean absolute pixel difference
- \(\theta\)
- the cut threshold. Too low: camera motion and flashes look like cuts. Too high: soft cuts and fades are missed.
Tools such as PySceneDetect implement this with content and threshold detectors, plus a minimum shot length. [Certain] Scene-based sampling makes sure a 0.4-second insert shot is not missed. But it treats a long shot with a lot of action the same as a long static shot.
5. Adaptive (content-based) sampling. Put more frames where more changes. Accumulate the change over time and take frames at equal steps of the accumulated change:
- Read the symbols
- \(S(t)\)
- accumulated change up to frame \(t\); the frames are spread evenly along \(S\), not along time
- \(\lambda\)
- a small constant per frame, so that static parts still get some frames
- \(c\)
- a cap on one frame’s change, so that a single cut does not use up many frames
With \(c\) very small, every frame adds the same amount and this becomes uniform sampling. With \(\lambda\) small, the frames go where the motion is. The weak point is the opposite of uniform: a short static shot adds little change and can be skipped. More advanced methods choose frames by their relevance to the question (query-aware sampling, see session 210) or let the model ask for more frames (agents, see session 206).
How do we compare strategies without a model? One simple measure is coverage error: for every frame of the video, how far it is from the closest kept frame, in the same feature space as \(\mathbf{x}\):
Low \(E\) means every moment of the video has a similar frame among the kept ones. Figure 4 uses a synthetic clip of 360 frames (12 s at 30 fps) with 7 shots. Each frame is a 3-number feature (shown as its color). Shot 2 lasts only 12 frames and shot 4 only 18 frames. Shot 5 has a fast camera pan in its middle.
How to read Figure 4
At \(k = 8\), uniform sampling sees only 5 of 7 shots: the short shots 2 and 4 fall between samples. Keyframes see all 7 shots, but there are only 7 keyframes, so raising \(k\) does not help them, and their coverage error stays at 0.102 because they take the first frame of each shot and ignore the pan. Scene-based sampling sees all 7 shots with the lowest error at \(k = 8\) (0.034). Adaptive sampling puts several frames on the pan in shot 5 but often skips a short static shot; at \(k = 12\) it sees all 7 shots and has the lowest error (0.021). Now step \(k\) from 8 to 12 with uniform sampling: 5, 5, 7, 5 and 6 shots. Catching a short shot is luck of alignment. At large \(k\) all methods except keyframes become similar.
- input: decoded thumbnails \(\mathbf{x}_0, \dots, \mathbf{x}_{N-1}\) (low resolution, cheap), budget \(k\), threshold \(\theta\), minimum shot length \(L_{\min}\)
- compute \(d_t\) for all \(t\) (formula 201.7); mark a cut where \(d_t \gt \theta\) and the last cut is at least \(L_{\min}\) frames back
- shots \(\leftarrow\) runs of frames between cuts, with lengths \(L_1, \dots, L_S\)
- if \(k \lt S\): keep the middle frame of the \(k\) longest shots; return
- give every shot 1 frame; share the remaining \(k - S\) frames in proportion to \(L_s\) (largest remainders first)
- inside shot \(s\) with \(n_s\) frames: take the middles of \(n_s\) equal parts // formula (201.6) inside the shot
- return the frame indices and their PTS // keep timestamps for the model
Notice the cost: scene detection needs \(d_t\) for every frame, so it needs a full decode, often at low resolution or of keyframes and a fixed fraction of frames. In a large pipeline this is done once at ingest time and stored with the video. [Likely]
Tokens per second: the budget problem
Now the kept frames enter the model. A ViT cuts each frame into \(p \times p\) patches (see session 164). Many VLMs then merge each \(m \times m\) block of patches into one token, and some also merge \(t_p\) consecutive frames into one “temporal patch”. Qwen2-VL uses \(p = 14\), \(m = 2\) and \(t_p = 2\): its first layer is a 3D convolution over two frames, and a still image is treated as two identical frames. [Certain: paper and config.json] The tokens per second of video are then:
- Read the symbols
- \(f_s\)
- sampling rate: kept frames per second of video
- \(H',\ W'\)
- height and width of each frame after resizing, not of the original video
- \(p\)
- patch size of the ViT in pixels (14 for Qwen2-VL)
- \(m\)
- spatial merge: \(m \times m\) patches become one token (2 for Qwen2-VL)
- \(t_p\)
- temporal patch: consecutive frames merged into one token group (2 for Qwen2-VL, 1 for models that encode each frame alone)
With a video token budget \(B_v\), the longest video we can show at these settings, and the largest square frame side that fits a given duration, are:
Formula (201.11) shows the trade-off: for a fixed budget, the side of each frame shrinks with the square root of the duration. Doubling the length of the video means each frame side is divided by about 1.41. Qwen2-VL limits each video to 16,384 tokens and lowers the frame resolution of long videos to stay under that limit. [Certain] Try it in Figure 5.
How to read Figure 5
The default is a 10-minute video at 2 fps and 448×448: 256 tokens per second and 153,600 tokens, 9.4 times the 16K budget. At 448×448, 16K tokens last only 64 seconds. To keep all 1,200 frames of the 10 minutes, each frame side must drop to 140 pixels (\(5 \times 5 = 25\) tokens per pair of frames): 15,000 tokens. Lower the sampling rate instead: at 0.25 fps, 448×448 fits 512 seconds. Switch the temporal patch to 1 frame: every number doubles.
Qwen2-VL style (\(p = 14\), \(m = 2\), \(t_p = 2\), 2 fps, 448×448): each pair of frames gives \(32 \times 32 / 4 = 256\) tokens, and there is one pair per second, so \(\tau = 256\). One minute: 15,360 tokens, just under the 16,384 limit. Ten minutes: 153,600 tokens. To fit 10 minutes into 16,384 tokens at 2 fps: \(\sqrt{16{,}384 \times 2 / (600 \times 2)} = 5.2\), so 5 merged tokens per side, a 140×140 frame and 15,000 tokens. Frames that small cannot show text or faces well. Lowering the frame rate keeps the detail but loses events.
Gemini API. Google’s documentation counts about 263 tokens per second of video at the default sampling of 1 frame per second, plus 32 tokens per second of audio, and gives lower rates (about 100 per second) for a low-resolution setting. [Certain, as documented in 2026; the numbers change between model versions] At 263 + 32 = 295 tokens per second, a 1-million-token context holds about 56 minutes of video with audio, before any text.
The lesson. At about 250 tokens per second, the budget runs out quickly. Real systems answer with lower resolution per frame, lower fps, stronger token merging, or by retrieving only the relevant parts of a long video.
- input: duration \(D\), budget \(B_v\), model constants \(p, m, t_p\), minimum useful side \(s_{\min}\) (for example 224), maximum fps \(f_{\max}\)
- \(f_s \leftarrow f_{\max}\); \(s \leftarrow s_{\text{fit}}(D, f_s, B_v)\) // formula (201.11)
- while \(s \lt s_{\min}\) and \(f_s\) can go lower: halve \(f_s\); recompute \(s\) // trade events for detail
- if \(s\) is still below \(s_{\min}\): switch to uniform sampling of \(k = \lfloor B_v\, t_p / \text{tokens per group at } s_{\min} \rfloor\) frames, or split the video and retrieve the relevant parts
- choose the frame positions (uniform, scene-based or adaptive) and read them with Algorithm 1
- return frames, their timestamps and the side \(s\) // pass timestamps so the model knows real time
Timestamps matter. With uniform sampling, the gap between frames depends on the video length, so the model cannot know real time unless you tell it. Qwen2.5-VL adds absolute time encoding so it can locate events to the second in long videos. [Certain] Simpler systems write the timestamp next to each frame as text.
Code: seek cost, sampling and token budget in PyTorch
The code below is deterministic and runs on a CPU in under a second. Part 1 counts decoded frames for the naive and GOP-aware readers (formulas 201.4 and 201.5). Part 2 builds a tiny synthetic video, detects cuts and compares three samplers. Part 3 counts tokens the Qwen2-VL way and finds the frame size that fits a budget.
import math import torch torch.manual_seed(0) # 1) Seek cost: frames a decoder must decode to return k target frames def decode_cost(targets, G, smart=True): """targets: sorted frame indices; G: keyframe interval (P-frames only). naive: every target restarts at its keyframe; smart: one pass per GOP.""" if not smart: return sum(t % G + 1 for t in targets) last = {} for t in targets: last[t // G] = max(last.get(t // G, -1), t % G) return sum(v + 1 for v in last.values()) def uniform_indices(N, k): """Center of each of k equal segments (TSN-style).""" return [int((i + 0.5) * N / k) for i in range(k)] N, G = 10 * 60 * 30, 250 # 10 minutes at 30 fps, x264 default keyint for k in (8, 32, 1200): t = uniform_indices(N, k) print(f"k={k:5d} naive {decode_cost(t, G, False):7d} " f"smart {decode_cost(t, G):6d} decode-all {N}") # 2) A tiny synthetic video: 5 shots of different colors, plus noise lengths = [40, 6, 30, 8, 36] # frames per shot colors = torch.rand(len(lengths), 3, 1, 1) frames = torch.cat([c.expand(L, 3, 32, 32) for c, L in zip(colors, lengths)]) frames = (frames + 0.02 * torch.randn_like(frames)).clamp(0, 1) # (T, 3, H, W) T = frames.shape[0] def frame_diff(x): """Mean absolute difference between consecutive frames (d_0 = 0).""" small = torch.nn.functional.avg_pool2d(x, 4) # cheap: compare 8x8 thumbnails d = (small[1:] - small[:-1]).abs().mean(dim=(1, 2, 3)) return torch.cat([torch.zeros(1), d]) d = frame_diff(frames) cuts = [0] + (d > 0.1).nonzero().flatten().tolist() print("detected cuts:", cuts) def scene_sample(cuts, T, k): """One frame per shot (middle), extra frames to long shots.""" bounds = cuts + [T] L = torch.tensor([b - a for a, b in zip(bounds[:-1], bounds[1:])], dtype=torch.float) n = torch.ones(len(L), dtype=torch.long) for _ in range(k - len(L)): # give each extra frame to the n[torch.argmax(L / n)] += 1 # shot with most frames per sample out = [] for a, Ls, ns in zip(bounds[:-1], L.long().tolist(), n.tolist()): out += [a + int((j + 0.5) * Ls / ns) for j in range(ns)] return out def adaptive_sample(d, k, lam=0.01, cap=0.05): """Equal steps of cumulative change: busy parts get more frames.""" S = torch.cumsum(lam + d.clamp(max=cap), 0) targets = (torch.arange(k) + 0.5) * S[-1] / k return torch.searchsorted(S, targets).tolist() shot = torch.repeat_interleave(torch.arange(len(lengths)), torch.tensor(lengths)) for name, idx in [("uniform", uniform_indices(T, 6)), ("scene", scene_sample(cuts, T, 6)), ("adaptive", adaptive_sample(d, 6))]: print(f"{name:8s} {idx} shots seen: {len(set(shot[idx].tolist()))} of {len(lengths)}") # 3) Token budget: Qwen2-VL-style counting (patch 14, merge 2x2, 2 frames per temporal patch) def video_tokens(seconds, fps, side, patch=14, merge=2, t_patch=2): groups = math.ceil(seconds * fps / t_patch) per_group = (side // (patch * merge)) ** 2 return groups * per_group def fit_side(seconds, fps, budget, patch=14, merge=2, t_patch=2): """Largest square side (multiple of 28) that keeps the video under budget.""" groups = math.ceil(seconds * fps / t_patch) cells = math.floor(math.sqrt(budget / groups)) return cells * patch * merge print("1 min at 2 fps, 448x448:", video_tokens(60, 2, 448), "tokens") print("10 min at 2 fps under 16384 tokens -> side", fit_side(600, 2, 16384), "->", video_tokens(600, 2, fit_side(600, 2, 16384)), "tokens") # Output (PyTorch 2.8, CPU): # k= 8 naive 1008 smart 1008 decode-all 18000 # k= 32 naive 4016 smart 4016 decode-all 18000 # k= 1200 naive 150600 smart 17496 decode-all 18000 # detected cuts: [0, 40, 46, 76, 84] # uniform [10, 30, 50, 70, 90, 110] shots seen: 3 of 5 # scene [10, 30, 43, 61, 80, 102] shots seen: 5 of 5 # adaptive [11, 32, 48, 70, 86, 109] shots seen: 3 of 5 # 1 min at 2 fps, 448x448: 15360 tokens # 10 min at 2 fps under 16384 tokens -> side 140 -> 15000 tokens
Part 2 shows the same pattern as Figure 4: with 6 frames for 5 shots, uniform and adaptive sampling both miss the 6-frame and 8-frame shots, and only the scene-based sampler sees all of them. (The scene sampler here gives extra frames to the shot with the most frames per sample; Algorithm 2 uses largest remainders. Both give one frame per shot first.) To read real frames, replace part 2’s synthetic tensor with a decoder such as TorchCodec, PyAV or decord, and pass timestamps rather than frame numbers.
Where this is used
| Application | Typical frame choice | What dominates the cost |
|---|---|---|
| Video question answering with a Video-LLM | uniform 8 to 64 frames, or 1 to 2 fps with lower resolution; timestamps in the prompt | tokens (prefill) and decode of the frames; see session 205 |
| Action recognition | short clips of 8 to 32 frames, uniform segments (TSN) or dense clips at a fixed stride | decoding of random clips during training; see session 203 |
| Text-to-video retrieval | a few frames or one embedding per shot, computed once at indexing | decoding the whole library once; see session 208 |
| Key-moment detection and temporal grounding | fixed fps (1 to 2), because the answer is a time; short clips around candidates | long sequences of frame features; see session 207 |
| Video fingerprinting and near-duplicate detection | keyframes or fixed fps at very low resolution | decode throughput over millions of videos; keyframe-only decoding helps; see session 209 |
| Video object segmentation and tracking | every frame (or every second frame) at full resolution | no sampling shortcut: dense decode and per-frame compute; see session 204 |
| Video-RAG over long videos | shots detected at ingest, frames and captions per shot stored in an index | one full decode at ingest, then cheap retrieval; see session 210 |
Common mistakes
- Seeking for every frame. A loop that calls “seek to time t, read one frame” for 1,000 timestamps can decode the same GOP hundreds of times (formula 201.4). Sort the targets and use one pass per GOP, or decode the file once.
- Using frame numbers instead of timestamps. Phone videos often have a variable frame rate. Frame \(n\) is then not at time \(n/f\). Use the PTS of each frame for sampling and for the timestamps you give the model.
- Ignoring rotation metadata. Phones store portrait video as landscape plus a rotation flag in the container. If the reader ignores it, the model sees sideways frames. [Likely]
- Wrong color conversion. YUV to RGB needs the right matrix (BT.601 or BT.709) and range (limited 16–235 or full 0–255). A mismatch makes colors washed out or too dark, and the frames differ from what the encoder saw in training. [Likely]
- Training and serving with different sampling. A model trained on 8 uniform frames and served at 1 fps sees a different time scale. Keep the sampler, fps and resolution the same, or train with several.
- Fast but inaccurate seeking. Some readers jump to the nearest keyframe for speed and return that frame. Your “frame at 5.0 s” may be from 0.2 s or 8 s earlier. Check the PTS of what you get.
- Counting tokens per frame and forgetting the temporal patch. With \(t_p = 2\), an odd number of frames is padded (Qwen2-VL repeats a frame), and the token count is per pair of frames, not per frame. [Likely]
- Blaming the GPU. Low GPU use in a video job usually means the GPU is waiting for decoded frames. Profile the decoder first.
Interview questions
What are I, P and B frames, and why does the file store frames out of order?basic
What is a GOP, and how does its length affect random access?basic
You sample 32 frames from a 10-minute 30 fps video with keyframes every 250 frames. How many frames are decoded?core
When is decoding the whole video cheaper than seeking?core
Your video Video-LLM pipeline shows 15% GPU utilization. What do you check?core
Compare uniform, fixed-fps, keyframe, scene-based and adaptive sampling.core
How many tokens does one minute of video cost in Qwen2-VL at 448×448 and 2 fps?basic
You must fit a 10-minute video into 16K tokens. What are your options?advanced
Why is NVDEC useful even if CPU decoding is fast enough?advanced
Why should you pass timestamps to a Video-LLM?advanced
Exercises
- Using formula (201.1), compute the raw data rate of 4K video (3840×2160) at 60 fps in YUV 4:2:0. How much is that compressed at 20 Mbit/s?
- In Figure 2, set \(G = 16\) and B frames = 1. What is the minimum number of frames to decode B13, and how many does a simple decode loop decode?
- A 1-hour, 30 fps video has keyframes every 250 frames. You sample 64 frames uniformly with a naive reader. How many frames are decoded? Check with Figure 3.
- In Figure 4 with uniform sampling, step \(k\) from 8 to 12 and write down the number of shots seen. Why does the count go down when \(k\) goes up? Which \(k\) guarantees that every shot is seen?
- A model uses \(p = 14\), \(m = 2\), \(t_p = 1\), 1 fps and 448×448. How many tokens does one hour of video cost?
- With formula (201.11), what is the largest square frame for a 5-minute video at 1 fps and \(t_p = 2\) in a 32,768-token budget?
Answerscheck
Key takeaways
- A video is frames with time. Raw 1080p30 is 93 MB/s in YUV 4:2:0 and 187 MB/s in RGB; the file is about 100 times smaller because of compression (201.1, 201.2).
- Codecs store I frames alone and P and B frames as changes. Decoding can start only at a keyframe, so one random frame costs about \((G+1)/2\) decoded frames (201.3).
- The decoder pays for every frame in the decode chain, not for the frames you keep. Seeking per frame can cost more than decoding the whole file; a GOP-aware reader never does (201.4, 201.5).
- Decoding is often the real bottleneck, larger than the vision encoder. Use GPU decoding (NVDEC), short GOPs for training data, or pre-extracted frames.
- Uniform, fixed-fps, keyframe, scene-based and adaptive sampling miss different things. Short shots need scene detection; busy shots need adaptive allocation (201.6–201.9).
- Tokens per second are set by fps, resolution, patch size and merging (201.10). At about 256 tokens per second, budgets run out in a minute, and frame size shrinks with the square root of duration (201.11).
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Tang et al. (2023). Video Understanding with Large Language Models: A Survey
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Tang et al. (2023). Video Understanding with Large Language Models: A Survey
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Wang, Xiong, Wang, Qiao, Lin, Tang & Van Gool (2016). Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
- NVIDIA (2025). Video Encode and Decode GPU Support Matrix
- VideoLAN (2024). x264: free H.264/AVC encoder (command-line help, default settings)
- Google (2026). Gemini API documentation: understand and count tokens
- PyTorch team (2025). TorchCodec: media decoding and encoding for PyTorch, on CPU and GPU
- Castellano (2024). PySceneDetect: video scene cut detection and analysis tool