Long video
A one-hour video is hundreds of thousands of visual tokens, far more than a Video-LLM can read. This session shows four ways to handle it: a longer context, merging tokens that repeat over time, a memory bank for streaming video, and an agent that searches for the right moments and keeps captions as a compact memory.
What you will learn
- How to count the token budget of a video, N = L · f · n, and the KV-cache memory it needs, with real numbers for a Qwen2-7B-shaped model
- Long context transfer (LongVA): extend the language model on text, then read about 2,000 frames, and why accuracy still plateaus
- Token compression across frames: weighted token merging (ToMe) and LongVU’s frame dropping, question-guided resolution and spatial token compression
- Memory banks for streaming video (MovieChat) and how VideoLLM-online learns when to speak with a streaming EOS loss
- Agentic search (VideoAgent): why uniform sampling misses short events, and how an LLM picks the segments to look at
- Captions as a compressed memory of the video (LLoVi), what they keep and what they lose
Why this matters
A Video-LLM from session 205 reads 8, 16 or 64 frames and answers a question. That works for a 30-second clip. It breaks for a 2-hour football match, a 1-hour lecture or a day of camera footage. The question “when did the goalkeeper first touch the ball?” may depend on two seconds somewhere in the middle, and a model that sees 16 frames will most likely never see them.
The problem is a token budget. Every frame becomes tens or hundreds of visual tokens. A long video becomes hundreds of thousands of tokens, which is more than the context of most models and costs many gigabytes of memory. This session is about the four ways people deal with it:
- Make the context bigger, so more frames fit (LongVA).
- Compress the tokens: merge or drop tokens that repeat over time (token merging, LongVU).
- Keep a memory of fixed size and read the video as a stream (MovieChat, VideoLLM-online).
- Search: let the model choose which parts to look at, and store what it saw as text captions (VideoAgent, LLoVi).
These ideas are not only for chat. The same tools power key-moment detection in sports (session 207), video search (session 208) and Video-RAG (session 210). Figure 1 shows the four strategies side by side.
The token budget of a long video
Start with simple counting. A video of length \(L\) seconds, sampled at \(f\) frames per second, with \(n\) visual tokens per frame, gives
- Read the symbols
- \(N\)
- the number of visual tokens the language model must read
- \(L\)
- video length in seconds
- \(f\)
- how many frames per second we keep (the sampling rate, not the camera’s rate)
- \(n\)
- tokens per frame after the vision encoder and connector: 576 for a 24×24 CLIP grid, 144 after 2×2 pooling, 1 if we keep only one summary token
Each of these tokens also lives in the KV cache: the stored keys and values of every past token, in every layer (see session 80). Its size is
- Read the symbols
- \(2\)
- one key and one value per token
- \(n_{\text{layer}}\)
- number of transformer layers
- \(n_{\text{kv}}\)
- number of key/value heads (fewer than query heads with grouped-query attention)
- \(d_{\text{head}}\)
- size of one head
- \(b\)
- bytes per number: 2 for bf16
Qwen2-7B has 28 layers, 4 key/value heads and head size \(3584/28 = 128\) (from its config.json) [Certain]. In bf16 one token costs \(2 \cdot 28 \cdot 4 \cdot 128 \cdot 2 = 57{,}344\) bytes, about 56 KB.
One hour at 1 fps is 3,600 frames. With 144 tokens per frame, (206.1) gives \(N = 3600 \cdot 144 = 518{,}400\) tokens. The KV cache is \(518{,}400 \cdot 57{,}344 = 29.7\) GB, more than the 14 GB or so of the model weights. With the full 576 tokens per frame it is 2,073,600 tokens and 118.9 GB. The model’s normal context is 32,768 tokens, which holds only \(32768/144 \approx 227\) frames: under 4 minutes at 1 fps.
Attention cost grows even faster: the score matrix in each layer has \(N^2\) entries, so 4 times the frames means 16 times the attention work.
Figure 2 lets you change the three factors and see where the token count lands against common context sizes.
How to read Figure 2
The default is one hour, 1 fps, 144 tokens per frame: 518,400 tokens and 29.7 GB of KV cache, past every context line. Move tokens per frame down to 1: 3,600 tokens, which fits even a 4K context. That is the setting VideoLLM-online uses for most experiments (one token per frame). Then raise frames per second to 2 and the length to 30 minutes: still 3,600 tokens. Each factor multiplies, so halving any one of them halves the budget.
There are two ways out. Keep all tokens and make the model handle them (next section), or send fewer tokens (the sections after it).
Strategy 1: a longer context (LongVA)
LongVA (Long Video Assistant) asks a simple question: if the language model can read a 200K-token book, can it read a 200K-token video? The authors call this long context transfer [Certain]:
- Take Qwen2-7B-Instruct and continue pretraining it on long text only, until it handles 224K tokens. The RoPE base is raised to \(10^9\) (Qwen2’s config uses \(10^6\)); the training is about 900M tokens of SlimPajama at 224K length, the longest that fit on 8 A100-80G GPUs.
- Align it with images using short image–text data, as in LLaVA. No video training at all.
- At test time, feed many frames. Each frame gives 576 CLIP tokens, pooled 2×2 to 144, and all frames are laid out like one very wide image (“UniRes” encoding): \(N\) frames give \(144N\) tokens.
Why does raising the RoPE base help? In session 14 each pair of dimensions \(i\) of a query or key is rotated by an angle that grows with position \(m\):
- Read the symbols
- \(m\)
- the position of the token in the sequence
- \(\theta_i\)
- the rotation speed of dimension pair \(i\)
- \(\beta\)
- the RoPE base, a fixed number in the config
- \(d\)
- the head size (128 here)
A bigger base makes the slow dimensions rotate even more slowly, so far-apart positions still get different, smooth angles that the model has learned to read. For \(d = 128\), the slowest pair repeats after \(2\pi \beta^{126/128}\) positions: about 5.1 million with \(\beta = 10^6\) and about 4.5 billion with \(\beta = 10^9\).
To test it, the paper builds V-NIAH (visual needle in a haystack): one unusual frame with a question about it is hidden in hours of video sampled at 1 fps. LongVA finds the needle almost perfectly up to about 2,000 frames, which is over 200K visual tokens, although it was trained at 224K tokens, about 1,555 frames [Certain]. On Video-MME (videos up to an hour) its score rises from 47.9 with 8 frames to 52.6 with 128 frames and then stops rising (51.8 at 384 frames) [Certain].
That plateau is the lesson. A longer context lets the model see more, but it does not make it use more, and the cost is still the full KV cache of Figure 2. So most systems also reduce the tokens.
Strategy 2: merge tokens that repeat over time
Most of a video does not change from one second to the next. The wall, the floor and the sky are the same; only a person or a ball moves. If a token at position \(p\) in frame \(t\) is almost the same as the token at the same position in frame \(t-1\), we can drop it or merge it, and lose little. The measure of “almost the same” is cosine similarity:
It is 1 for vectors that point the same way, 0 for unrelated ones.
Token merging (ToMe)
Token Merging (ToMe) was made for image ViTs, and video models reuse it. In each layer it splits the tokens into two sets A and B, links every token in A to its most similar token in B, and merges the \(r\) most similar pairs by a weighted average [Certain]:
- Read the symbols
- \(\mathbf{x}_a, \mathbf{x}_b\)
- the two tokens being merged
- \(s_a, s_b\)
- their sizes: how many original tokens each one already stands for (1 at the start)
- \(s_{\text{new}}\)
- the size of the merged token. ToMe also adds \(\log s\) to the attention scores of that token, so a token that stands for 10 patches still gets the weight of 10
The weights matter: without them, a token that already stands for 100 frames of sky would count as much as one new token, and the average would drift.
LongVU: compress in three steps
LongVU fits hour-long videos sampled at 1 fps into an 8K context of Qwen2-7B. It reduces tokens in three steps [Certain]:
- Temporal reduction. Split frames into windows of 8. Compute DINOv2 features (DINOv2 is good at fine visual similarity) and remove frames that are very similar to the others in their window. On average about 45.9% of frames remain.
- Cross-modal query. If the remaining frames still do not fit at full size (144 tokens), keep the \(N_h\) frames most related to the question at 144 tokens and pool the rest to 64 tokens. The number that fits is
- Read the symbols
- \(L_{\max}\)
- the context length (8,192)
- \(L_q\)
- the number of text tokens in the question
- \(T\)
- the number of frames left after step 1
- \(144, 64\)
- tokens per frame at full (12×12) and reduced (8×8) size
- Spatial token compression (STC). If even 64 tokens per frame is too much, use windows of \(K = 8\) frames. The first frame keeps all its tokens. In each later frame, drop a token if it is too similar to the token at the same place in the first frame:
In the paper this removes about 40.4% of the tokens on average [Certain].
A 10-minute video at 1 fps has 600 frames. If 45.9% survive step 1, about 275 remain. At 64 tokens each that is \(275 \cdot 64 = 17{,}600\) tokens, more than 8,192, so (206.6) gives \(N_h = 0\): every frame is pooled, and STC must remove more.
A shorter case: \(T = 100\) frames, a 100-token question. Then \(N_h = \lfloor (8192 - 100 - 6400) / 80 \rfloor = \lfloor 21.15 \rfloor = 21\). The 21 frames most related to the question get 144 tokens; the other 79 get 64. Total: \(21 \cdot 144 + 79 \cdot 64 + 100 = 8{,}180\) tokens, just under 8,192.
- input: frames sampled at 1 fps, question \(q\), context limit \(L_{\max}\)
- split frames into windows of 8; in each, drop frames whose mean DINOv2 similarity to the others is high // step 1
- encode the remaining \(T\) frames: 144 tokens each
- if \(144\,T + L_q \gt L_{\max}\) then
- score each frame against \(q\); compute \(N_h\) by (206.6) // step 2
- keep the top \(N_h\) frames at 144 tokens, pool the others to 64
- if \(64\,T + L_q \gt L_{\max}\) then
- for each window of 8 frames: keep the first frame; drop tokens with \(s \gt 0.8\) to it (206.7) // step 3
- feed tokens in time order, then \(q\), to the LLM
Figure 3 runs the two kinds of reduction on a toy video: 48 frames, 16 tokens each, three scenes, a small object that moves one place every 4 frames, and one “needle” frame (frame 37) where four tokens show something unusual.
How to read Figure 3
At \(\theta = 0.9\), frame dropping keeps 14 frames (224 of 768 tokens): one each time the object moves, at each scene change, and the needle. Lower \(\theta\) to 0.5: only 3 frames survive (one per scene) and the needle is lost, because 10 of its 16 tokens are the same as in frame 32, so its mean similarity to frame 32 is about 0.63, above the threshold. Now switch to STC: at \(\theta = 0.5\) it keeps 140 tokens, fewer than frame dropping at 0.9, and still keeps all 4 needle tokens, because it judges each token on its own. Token-level decisions keep small changes that a frame-level average hides.
Strategy 3: memory banks and streaming
Compression still needs the whole video before it starts. A live camera, a sports broadcast or a robot has no “whole video”: frames keep coming. A streaming video LLM reads one frame at a time and must answer at any moment. It needs a memory bank: a store of fixed size that summarizes the past.
MovieChat: short-term and long-term memory
MovieChat borrows the human memory model of Atkinson and Shiffrin [Certain]. Frames are read in windows of 16. Each frame becomes 32 tokens (from a Q-Former, see session 178). The newest frames go into a short-term memory of 18 frames that works as a first-in, first-out queue. When it is full, its frames are consolidated: the adjacent pair with the highest similarity is merged by a weighted average, again and again, until a few representative frames remain, and these go into the long-term memory. In formula form, with memory slots \(\mathbf{m}_1, \dots, \mathbf{m}_S\) and capacity \(C\):
- Read the symbols
- \(\mathbf{m}_i\)
- memory slot \(i\): the (weighted) average of the frames it stands for
- \(S\)
- the number of slots in use
- \(C\)
- the fixed capacity
- \(i^\star\)
- the most similar pair of neighbours; only neighbours are merged, so time order is kept
The paper reports that this handles more than 10K frames on a 24 GB GPU [Certain]. It answers in two modes: global (only long-term memory, for questions about the whole video) and breakpoint (long-term memory plus short-term memory plus the current frame, for “what is happening now?”).
Why merge similar neighbours instead of dropping the oldest frame? Figure 4 shows the difference on a stream of 24 frames with four scenes: kitchen (frames 0–6), a 2-frame doorbell event (7–8), cooking (9–16) and eating (17–23).
How to read Figure 4
Press Play or Next to feed one frame at a time. With “Merge similar” and \(C = 5\), the kitchen frames collapse into one slot, but the doorbell keeps its own slot to the end, because it is not similar to its neighbours. At frame 23 the memory is kitchen 0–6, doorbell 7–8, cooking 9–16, eating 17–22 and eating 23. Switch to “Drop oldest”: at the end only frames 19–23 remain, and the doorbell is gone. A FIFO memory remembers only the recent past; a merging memory remembers every distinct event, at lower detail.
VideoLLM-online: when should the model speak?
A streaming assistant has a second problem: it must decide when to talk. If you ask “tell me when the pasta is done”, the answer must come at the right frame, not at the end. VideoLLM-online solves this with its LIVE framework (Learning-In-Video-strEam) [Certain]:
- Frames arrive at 2 fps. In the main experiments each frame is only 1 token (the CLIP class token); the demo model uses 10 tokens per frame (1 plus a 3×3 pooled grid). With 1 token per frame, half an hour at 2 fps is 3,600 tokens and fits in a 4,096 context.
- After every frame the model predicts either text (start speaking) or a special EOS token that means “stay silent, wait for the next frame”.
- Training adds a streaming loss to the normal language-model loss: at frames where no answer is due, the model is taught to predict EOS.
- Read the symbols
- \(P_j\)
- the model’s next-token distribution at position \(j\)
- \(l_j\)
- 1 if position \(j\) is in a text answer (normal language-model loss), else 0
- \(f_j\)
- 1 if position \(j\) is the last token of a frame where the model should stay silent, else 0
- \(w\)
- the weight of the streaming term (1.0 by default)
At inference, the model speaks when \(P[\text{EOS}]\) falls below a threshold; the paper found values between 0.5 and 0.8 work much better than no threshold [Certain]. Two engineering tricks make it fast: the KV cache is kept across the whole stream (old frames are never re-encoded), and the vision encoder runs in parallel with the LLM and pushes frame tokens into a queue. The paper reports over 10 fps on an A100 for a 5-minute video, and in its ablation 18.2 GB of memory and 13.5 fps against 34.4 GB and 1.5 fps for an interleaved baseline [Certain].
- input: a stream of frames, capacity \(C\), speak threshold \(\theta\)
- memory \(\leftarrow\) empty; KV cache \(\leftarrow\) the system prompt
- for each new frame \(t\) do
- encode the frame to a few tokens; append them to the memory and the KV cache
- while memory size \(\gt C\): merge the most similar neighbours (206.8) // MovieChat-style
- if the user asked something: add the question tokens
- \(p \leftarrow P[\text{EOS} \mid \text{memory}, \text{dialog}]\)
- if \(p \lt \theta\) then generate an answer and append it to the cache // VideoLLM-online-style
- else stay silent and wait for the next frame
A real system must also keep the KV cache consistent with a merged memory: when slots merge, their cached keys and values must be recomputed or dropped. MovieChat merges before the LLM sees the tokens; VideoLLM-online keeps very few tokens per frame instead of merging.
Strategy 4: agentic search
A person who must answer “what did the chef add after the onions?” about a 1-hour cooking video does not watch the whole hour. They skim, jump to the part that looks right, look closely, and stop when they are sure. VideoAgent makes an LLM do the same [Certain]:
- Sample 5 frames uniformly and caption them with a vision-language model.
- Give the captions and the question to the LLM (GPT-4). It answers and rates its own confidence: 1 (not enough information), 2 (partial) or 3 (enough).
- If confidence is low, the LLM says which segments (gaps between the frames seen so far) to search and writes a text query for each, for example “the chef pours something into the pan”.
- A CLIP model scores the frames in each chosen segment against the query and returns the best one. Caption it, add it, and go back to step 2.
Here the LLM is an agent: it plans, calls tools (the captioner and the CLIP retriever), and decides when to stop (see session 154). On EgoSchema (3-minute egocentric videos, over 5,000 multiple-choice questions) VideoAgent reached 54.1% zero-shot with 8.4 frames per question on average, and 71.3% on NExT-QA with 8.2 frames [Certain]. On a 500-question subset it scored 60.2% with 8.4 frames, against 59.6% for uniform sampling with 180 frames [Certain]. Accuracy stopped improving after 3 rounds.
Why does searching beat uniform sampling? Take an event of length \(d\) in a video of length \(L\), and \(k\) frames on an evenly spaced grid with a random start. The chance that at least one frame lands inside the event is
- Read the symbols
- \(k\)
- number of frames sampled
- \(d\)
- length of the event, in seconds
- \(L\)
- length of the video, in seconds
Eight frames and a 10-second event in a 1-hour video give \(8 \cdot 10 / 3600 = 0.022\): about a 2% chance. Uniform sampling cannot find short events in long videos; it can only describe the average scene. A retriever that scores every frame cheaply (one CLIP embedding per frame, computed once) and sends only the best few to the expensive captioner and LLM turns the problem around: its success depends on how well the query matches, not on luck.
How to read Figure 5
At the default (\(d = 10\), \(k = 8\), \(\sigma = 0.4\)) uniform sampling hits the event in 2.5% of trials, close to formula (206.10)’s 2.2%; retrieval hits it in 95%. Raise \(\sigma\) to 0.6: the retriever is now confused by look-alike frames and drops to about 50%. Set \(\sigma = 0\): retrieval is perfect. Raise \(d\) to 120: uniform sampling improves, because long events are easy to hit. Search pays off when events are short and the retriever is good; both depend on the task.
Captions as a compressed memory
VideoAgent never shows frames to the LLM. It shows captions: short text descriptions written by a vision-language model. Text is a very strong compression of video. LLoVi (a simple LLM framework for long-range video question answering) made this the whole method [Certain]: cut the video into 1-second clips, caption each clip with a captioner (LaViLa on EgoSchema), put all captions in time order into an LLM, and ask the question. A two-step prompt helps: first ask the LLM to summarize the captions with the question in mind (about 500 words), then answer from the summary. On EgoSchema’s full test set this raised accuracy from 42.8% to 52.2% with GPT-3.5 [Certain].
We can measure the compression as a ratio of token counts:
- Read the symbols
- \(\rho\)
- how many times fewer tokens the captions use than the visual tokens
- \(\Delta\)
- seconds of video per caption
- \(c\)
- text tokens per caption
Suppose one caption per 4 seconds and about 25 tokens per caption (illustrative values, not from a paper). One hour gives \(3600/4 = 900\) captions and \(900 \cdot 25 = 22{,}500\) text tokens. The visual budget at 1 fps and 144 tokens per frame is 518,400, so \(\rho = 518400 / 22500 \approx 23\). The captions fit a 32K context with room to spare; the frames do not fit at all.
Captions also have practical strengths: they can be stored in a normal text search index, any LLM can read them (even one with no vision), and they can be written once and reused for many questions. That is the base of Video-RAG (session 210).
Their weakness is that a caption keeps only what the captioner chose to say. “A man walks into a kitchen” loses the color of his shirt, the brand on the box, the exact count of eggs and small motions. Questions about fine detail need the frames. Good systems mix both: captions (or embeddings) to find the right moment, then real frames of that moment for the final answer.
Code: budget, token compression, a memory bank and a streaming rule
The code below is plain PyTorch on the CPU and is deterministic (fixed seed). It implements (206.1)–(206.2), STC (206.7), a merging memory bank (206.5, 206.8), the uniform hit probability (206.10) and a VideoLLM-online-style speak rule. The printed values are in the comments.
import torch
import torch.nn.functional as F
torch.manual_seed(0)
# 1) Token budget and KV-cache size (formulas 206.1 and 206.2)
def video_tokens(minutes, fps, tok_per_frame):
frames = round(minutes * 60 * fps)
return frames, frames * tok_per_frame
def kv_bytes(n_tokens, layers=28, kv_heads=4, head_dim=128, bytes_per=2):
return 2 * layers * kv_heads * head_dim * bytes_per * n_tokens # Qwen2-7B shape, bf16
frames, N = video_tokens(60, 1, 144)
print(frames, N, round(kv_bytes(N) / 1e9, 1)) # 3600 518400 29.7
# 2) Spatial token compression across time (formula 206.7)
def stc(video, theta=0.8, window=8):
"""video: (T, P, d). Keep all tokens of the first frame of each window;
in later frames drop a token if it is too similar to the same position
in that first frame."""
T, P, d = video.shape
keep = torch.ones(T, P, dtype=torch.bool)
for t in range(T):
w0 = (t // window) * window
if t != w0:
sim = F.cosine_similarity(video[t], video[w0], dim=-1) # (P,)
keep[t] = sim <= theta
return keep
T, P, d = 16, 16, 32
static = torch.randn(1, P, d).expand(T, P, d).clone()
video = static + 0.1 * torch.randn(T, P, d) # a nearly still camera
video[5, 3] = torch.randn(d) # one token changes at t=5
keep = stc(video)
print(int(keep.sum()), "of", T * P, "tokens kept") # 33 of 256 tokens kept
print(bool(keep[5, 3])) # True: the change survives
# 3) A memory bank that merges the most similar neighbours (formulas 206.5, 206.8)
class MemoryBank:
def __init__(self, capacity):
self.cap, self.mem, self.size = capacity, [], []
def add(self, x):
self.mem.append(x.clone()); self.size.append(1)
while len(self.mem) > self.cap:
sims = [F.cosine_similarity(self.mem[i], self.mem[i + 1], dim=0)
for i in range(len(self.mem) - 1)]
i = int(torch.stack(sims).argmax())
a, b = self.size[i], self.size[i + 1]
self.mem[i] = (a * self.mem[i] + b * self.mem[i + 1]) / (a + b)
self.size[i] = a + b
del self.mem[i + 1], self.size[i + 1]
scenes = torch.randn(3, 64)
stream = [scenes[0]] * 10 + [scenes[1]] * 2 + [scenes[2]] * 10 # a short event in the middle
bank = MemoryBank(capacity=4)
for x in stream:
bank.add(x + 0.05 * torch.randn(64))
print(bank.size) # [10, 1, 1, 10]: the 2-frame event keeps its own slots
# 4) Uniform sampling vs. a short event (formula 206.10)
def p_hit_uniform(k, d, L):
return min(1.0, k * d / L)
print(round(p_hit_uniform(8, 10, 3600), 4)) # 0.0222
# 5) Streaming: speak only when P(not EOS) is high enough (VideoLLM-online style)
def should_speak(logits, eos_id, theta=0.5):
p = torch.softmax(logits, dim=-1)
return bool(p[eos_id] < theta)
logits = torch.tensor([2.0, 0.5, 0.1, 1.0]) # token 0 is EOS ("stay silent")
print(should_speak(logits, eos_id=0)) # False: p(EOS)=0.57, stay silent
Notice the memory bank result: two scenes of 10 frames each end in one slot each, while the 2-frame event in the middle keeps its own slots. That is the same behavior as Figure 4.
Where this is used
| Application | Long-video problem | Typical approach |
|---|---|---|
| Sports highlights and key-moment detection | Find seconds-long events (goals, fouls) in a 2-hour broadcast | Dense cheap features over the whole match, then a model reads only candidate windows (session 207) |
| Video-RAG over a media library | Answer questions over thousands of hours | Captions and frame embeddings in an index; retrieve segments; a VLM reads the retrieved frames (session 210) |
| Text-to-video retrieval | Match a query to a moment, not to a whole file | Per-segment embeddings, retrieval, then re-ranking (session 208) |
| Live assistants (cooking, repair, AR glasses) | A stream with no end; answers must come on time | Streaming LLM with few tokens per frame, a kept KV cache and a speak/silent decision (VideoLLM-online) |
| Security and safety cameras | Days of mostly static footage | Drop frames that do not change (206.7), memory of events, alerts on change |
| Meeting and lecture summarization | One hour of slides and speech | Captions plus speech transcript as text memory; LLM summarizes; frames only for slides with figures |
| Fingerprinting and near-duplicate detection | Compare long videos quickly | Compact per-segment signatures; merging similar frames first reduces the work (session 209) |
| Long-video benchmarks (Video-MME, EgoSchema, MLVU) | Measure whether a model uses the whole video | Report frames, tokens and context next to accuracy; check needle tests (V-NIAH) |
Common mistakes
- Counting frames, not tokens. “The model reads 64 frames” means nothing without tokens per frame. 64 frames at 576 tokens is 36,864 tokens; at 1 token it is 64. Always compute (206.1).
- Forgetting the KV cache. A model that “supports 224K context” still needs (206.2) bytes of memory per token. Hour-long video at 144 tokens per frame is about 30 GB of cache for a 7B model.
- Uniform sampling for short events. Eight frames of a one-hour video hit a 10-second event about 2% of the time (206.10). Use retrieval or dense cheap features to find candidates first.
- Merging by a frame-level average. A small change (a ball, a needle frame) hardly moves the mean similarity of a whole frame. Decide per token (STC) or keep a minimum number of frames per segment (Figure 3).
- Unweighted merging. Averaging merged tokens without their sizes (206.5) lets one new token pull a slot that stands for 100 frames. Keep the counts.
- Dropping the oldest frames in a stream. FIFO memory forgets early events completely; questions about the start of the video then fail (Figure 4).
- Trusting captions for fine detail. Counts, colors, text on screen and small motions are often missing from captions. Go back to the frames of the retrieved moment.
- Losing time. After merging or dropping tokens, the model must still know when each token came from. Keep timestamps (as text or in the position encoding), or questions like “what happened after…?” break.
Interview questions
How many visual tokens is a one-hour video, and what does it cost in memory?basic
Name the main strategies for long video understanding and their trade-offs.basic
What is long context transfer in LongVA?core
How does token merging work, and why are the merged tokens weighted?core
Explain LongVU’s three compression steps.core
Why can uniform frame sampling fail on long videos? Give a number.basic
How does a streaming video LLM decide when to talk?core
Compare a FIFO memory with a merging memory bank.core
How does VideoAgent work, and what are its costs?advanced
When are captions a good memory for video, and when are they not?advanced
Design a system that answers questions about 2-hour football matches with an 8K-context VLM.advanced
Exercises
- A 45-minute lecture is sampled at 0.5 fps with 64 tokens per frame. How many tokens? Does it fit a 32K context?
- Using (206.2), how many GB of KV cache does a 128K-token context need in the Qwen2-7B shape (bf16)?
- In LongVU, \(T = 80\) frames and a 50-token question. Compute \(N_h\) for \(L_{\max} = 8192\).
- Two merged tokens have sizes 3 and 1 and values \(x_a = 2.0\), \(x_b = 6.0\) (one dimension). What is the merged value with and without size weights?
- In Figure 4, with “Merge similar”, set \(C = 3\) and step to frame 23. Which scenes share a slot, and does the doorbell survive? Explain why from (206.8).
- How many uniform frames are needed to hit a 5-second event in a 2-hour video with probability at least 0.5?
- With VideoLLM-online’s 10 tokens per frame at 2 fps, how long a video fits in a 4,096-token context (ignore text)?
Answerscheck
Key takeaways
- Count tokens first: \(N = L f n\) (206.1). One hour at 1 fps and 144 tokens per frame is 518,400 tokens and about 30 GB of KV cache for a 7B model (206.2).
- Long context transfer (LongVA) shows that a text-extended LLM can read about 2,000 frames, but accuracy plateaus and the memory cost stays.
- Token compression drops what repeats over time: weighted merging (206.5), LongVU’s frame dropping, question-guided resolution (206.6) and per-token STC (206.7). Decide per token so short events survive.
- Streaming models keep a fixed-size memory; merging similar neighbours (206.8) keeps every distinct event, FIFO keeps only the recent past. VideoLLM-online learns when to speak with a streaming EOS loss (206.9).
- Uniform sampling misses short events (206.10); agentic search (VideoAgent) looks at about 8 chosen frames and matches uniform sampling with 180.
- Captions compress video about 20× or more (206.11) and make it searchable; use them to find the moment, then look at the frames.
Key formulas
References for this session
- Zhang et al. (2024). Long Context Transfer from Language to Vision
- Shen et al. (2024). LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
- Wang et al. (2024). VideoAgent: Long-form Video Understanding with Large Language Model as Agent
- Chen et al. (2024). VideoLLM-online: Online Video Large Language Model for Streaming Video
- Zhang et al. (2024). Long Context Transfer from Language to Vision
- Shen et al. (2024). LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
- Bolya et al. (2022). Token Merging: Your ViT But Faster
- Song, Chai, Wang, Zhang, Zhou, Wu, Chi, Guo, Ye, Zhang, Lu, Hwang & Wang (2023). MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
- Chen et al. (2024). VideoLLM-online: Online Video Large Language Model for Streaming Video
- Wang et al. (2024). VideoAgent: Long-form Video Understanding with Large Language Model as Agent
- Zhang, Lu, Islam, Wang, Yu, Bansal & Bertasius (2023). A Simple LLM Framework for Long-Range Video Question-Answering
- Mangalam, Akshulakov & Malik (2023). EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding
- Fu et al. (2024). Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models