Video encoders: from 3D CNNs to video transformers
A video encoder turns a clip of frames into vectors that know about both appearance and motion. This session follows the main designs in order: 3D CNNs (C3D, I3D with filters inflated from image models, SlowFast with two frame rates), video transformers (ViViT and TimeSformer, with attention split into space and time), masked video pretraining (VideoMAE, which hides 90% of the clip), and video–text encoders such as InternVideo2 that feed today’s video LLMs.
What you will learn
- Why averaging per-frame features loses motion and order, and what a 3D convolution adds
- How I3D inflates a 2D image network into 3D and keeps the ImageNet weights useful (the boring-video fixed point)
- How SlowFast splits a clip into a slow, wide pathway for appearance and a fast, thin pathway for motion
- How ViViT and TimeSformer turn a clip into tokens, and why divided space–time attention is about 7.7× cheaper than joint attention for 8 frames
- Why VideoMAE can hide 90% of the tokens, and why the mask must be a tube through time
- How InternVideo2 trains a 6B video encoder in three stages: distillation, video–text contrastive learning, and next-token prediction
- How to choose a video encoder for classification, retrieval, key-moment detection and video LLMs
Why this matters
A video is a stack of images, but a video model that only looks at images misses the most important thing: what changes. “Opening a door” and “closing a door” have almost the same frames, in a different order. A video encoder is the network that turns a clip into vectors that carry both appearance (what is there) and motion (what happens).
Every video system in this track starts with one. Action recognition (session 203), text-to-video retrieval (session 208), key-moment detection (session 207) and video LLMs (session 205) all put a video encoder in front. The choice of encoder decides three things: how well the system sees motion, how many tokens (and so how much compute) one second of video costs, and whether the features already speak the language of text.
The history is short and each step fixes one problem of the step before:
- 3D CNNs (C3D, I3D, SlowFast, 2015–2019): convolutions that slide over time as well as space. I3D found a way to reuse ImageNet weights; SlowFast found a cheap way to see fast motion.
- Video transformers (ViViT, TimeSformer, 2021): a clip becomes a long sequence of patch tokens. Full attention over all of them is too expensive, so attention is split into a time part and a space part.
- Masked video pretraining (VideoMAE, 2022): learn from unlabeled video by hiding 90% of the tokens and rebuilding them.
- Video–text encoders (UMT, InternVideo2, 2023–2024): train the encoder so that its output lines up with text, then plug it into an LLM.
In session 201 we saw that video is expensive before any model runs: decoding, sampling and the token budget. This session is about the model that consumes the sampled frames. Session 203 then uses these encoders on real tasks.
The baseline: a 2D network on each frame
The simplest video model takes an image network \(g\) (a ResNet or a ViT), runs it on each sampled frame \(x_1, \dots, x_T\), averages the frame features, and classifies the average:
- Read the symbols
- \(V = (x_1, \dots, x_T)\)
- the clip: \(T\) frames, each an image of size \(H \times W \times 3\)
- \(g\)
- the image network, applied to each frame on its own
- \(h\)
- the classifier head (often one linear layer)
This is called late fusion: frames meet only at the very end. It is cheap, it reuses strong image models, and on datasets where the scene gives the answer (“playing piano”, “surfing”) it works well. But look at formula (202.1): a sum does not care about order. Shuffle the frames, or play the clip backwards, and the output is exactly the same. So the model cannot tell “opening” from “closing”, or “pushing something left to right” from “right to left”. Datasets such as Something-Something were built with exactly such labels, where one frame is not enough. [Certain]
Two early fixes:
- Two-stream networks (Simonyan and Zisserman, 2014): a second 2D network reads optical flow, the per-pixel motion between neighboring frames, computed by a classic algorithm. Motion now enters as an input. The price is computing flow for every frame, which is slow. [Certain]
- 3D convolutions: let the filters themselves look at several frames. This is the next section.
3D convolution: filters that slide through time
A 2D convolution slides a \(k \times k\) filter over the image. A 3D convolution slides a \(k_t \times k \times k\) filter over the clip: \(k_t\) frames deep, \(k\) pixels high and wide. One output value now depends on a small cube of space and time, so the filter can respond to an edge that moves.
- Read the symbols
- \(x_c(t, i, j)\)
- input channel \(c\) at frame \(t\), row \(i\), column \(j\)
- \(W_{o,c}(\tau, u, v)\)
- filter weight for output channel \(o\), input channel \(c\), time offset \(\tau\) and space offsets \(u, v\)
- \(y_o(t, i, j)\)
- the output feature map, which still has a time axis
The cost grows with the time size of the filter and with the number of frames:
where \(T' \times H' \times W'\) is the size of the output. C3D (Tran et al., 2015) was a VGG-like network built only from \(3 \times 3 \times 3\) filters, trained from scratch on a large sports dataset. [Certain] It showed that 3D filters learn motion, but training from scratch needs a lot of labeled video, and there was no ImageNet for video yet.
A \(3 \times 3\) convolution with 64 input and 64 output channels has \(3 \cdot 3 \cdot 64 \cdot 64 = 36{,}864\) weights. The \(3 \times 3 \times 3\) version has \(110{,}592\): three times more.
On an 8-frame feature map of \(56 \times 56\), the 3D layer costs \(2 \cdot 27 \cdot 64 \cdot 64 \cdot 8 \cdot 56 \cdot 56 = 5.55\) GFLOPs. Running the 2D layer on each of the 8 frames costs \(1.85\) GFLOPs. Again three times more, for one layer.
A factorized block, a \(1 \times 3 \times 3\) spatial filter followed by a \(3 \times 1 \times 1\) temporal filter with the same width, has \(36{,}864 + 12{,}288 = 49{,}152\) weights, 44% of the full 3D filter. This is the idea of R(2+1)D (Tran et al., 2018), which also found it more accurate than full 3D filters. [Certain] It is the convolution version of the “split space and time” idea that comes back in video transformers.
Figure 1 runs formula (202.2) on a toy clip, one output value at a time. The clip has 4 frames of 6×6 pixels and one channel; a 2×2 bright square moves one pixel to the right per frame. The default kernel is a hand-made motion detector: it adds the pixels on a line that moves right by one pixel per frame and subtracts the pixels on a line that moves left. A bar that does not move gives exactly 0, because both lines see it. No 2D filter can do this, since a 2D filter never sees two frames.
- for each output position \((t', i', j')\) of the \(T' \times H' \times W'\) grid:
- cube \(\leftarrow x[s t' : s t' + k_t,\ s i' : s i' + k,\ s j' : s j' + k]\) stride \(s\)
- for each output channel \(o\): \(y_o(t', i', j') \leftarrow b_o + \sum W_o \odot \text{cube}\) (202.2)
- return \(y\): \(T'H'W'\) positions \(\times\ C_{\text{out}}\) channels with \(s =\) kernel size these are the \(n\) tokens of (202.6)
How to read Figure 1
Press Play, or Step through it. Each output position takes two steps: first the cube is gathered (line 2), then the weighted sum is written (line 3). With the right-motion detector and the square moving right, the largest output is +2; with the square moving left it is −2; with a square that does not move, every output is exactly 0. Switch to “inflated edge ÷ 3” (the I3D weights of formula 202.4, explained in the next section): on the still square it gives ±3, the same as the 2D Sobel filter; on the moving square the response is smeared and its peak drops to 2. Now pick “Tubelet embedding”: the cubes no longer overlap, every pixel is read exactly once, and 2×2×2 tubelets give \(2 \cdot 3 \cdot 3 = 18\) tokens of 2 numbers each: a mean brightness (channel 1) and a “last frame minus first frame” value (channel 2). The second channel is not 0 only where the square moves inside a tubelet. With 1×2×2 tubelets (one frame deep) it is always 0: a token that sees one frame cannot carry motion.
I3D: inflate an image network into a video network
Carreira and Zisserman (2017) asked a practical question: image architectures and their ImageNet weights took years to develop. Can we reuse both? Their answer, I3D (Inflated 3D ConvNet), has two parts. [Certain]
Inflate the architecture. Take a 2D network (they used Inception-v1 with batch norm) and give every filter and pooling kernel a time dimension: \(N \times N\) becomes \(N \times N \times N\). The layer structure stays the same.
Inflate the weights. Turn a picture into a “boring video” by repeating it in every frame. A good 3D network started from ImageNet should give the same answer on this boring video as the 2D network gives on the picture. The paper calls this the boring-video fixed point. Because convolution is linear, this is easy: copy the 2D filter into each of the \(N\) time slices and divide by \(N\):
On a boring video every frame gives the same 2D response, and the average of \(N\) equal numbers is that number. Pointwise nonlinearities and pooling then keep the outputs unchanged, so the whole network gives the ImageNet answer at initialization. From there, training on video teaches the filters to use time. [Certain]
Other details matter too. I3D does not pool over time in the first two max-pool layers (kernels \(1 \times 3 \times 3\)), so early layers keep the fine time detail at 25 fps. It trains on 64-frame snippets and tests on whole videos by averaging predictions over time. And it still uses two streams: one I3D on RGB frames and one on optical flow (TV-L1), with the two predictions averaged. [Certain] The paper also re-evaluated video architectures on the then-new Kinetics dataset (400 human action classes, more than 400 clips per class), which made large-scale video pretraining possible. After Kinetics pretraining, I3D reached 80.9% on HMDB-51 and 98.0% on UCF-101. [Certain]
How to read Figure 2
Start with the boring video and “Repeat ÷ N”: the two response maps are identical, for any \(N\). That is formula (202.4). Switch to “Repeat, no ÷ N”: the 3D response is \(N\) times too large, so every layer after it would see inputs far outside what it was trained on. Now pick “Moving edge”: the edge moves one pixel per frame. The inflated filter averages \(N\) frames, so its response is spread and its peak drops (to \(2/3\) of the 2D peak for \(N = 3\)). The 3D filter no longer matches any single frame: training on video must now teach it to track the motion. “Central frame” puts the 2D filter only in the middle time slice and zeros elsewhere; it also passes the boring-video test, and it is exactly a per-frame 2D network at the start. ViViT found it works better for transformers (next sections).
- input: a pretrained 2D network, a time size \(N_\ell\) for each layer
- for each convolution with weight \(W^{2D}\) of shape \((C_{\text{out}}, C_{\text{in}}, k, k)\) do
- create \(W^{3D}\) of shape \((C_{\text{out}}, C_{\text{in}}, N_\ell, k, k)\)
- \(W^{3D}(\tau) \leftarrow W^{2D} / N_\ell\) for every \(\tau\) // formula (202.4)
- copy the bias and the batch-norm statistics unchanged
- for each pooling layer do give it a time size (often 1 in early layers, so time is not reduced too early)
- check: on a clip that repeats one image, the 3D network output equals the 2D network output on that image
- fine-tune on video clips (for example 64-frame snippets of Kinetics)
SlowFast: one pathway for “what”, one for “how it moves”
A 3D CNN treats space and time the same way. But they are not the same. The identity of things in a scene changes slowly: a person is still a person 0.5 seconds later. Motion can be fast: a hand clapping, a ball hit. Feichtenhofer et al. (2019) built SlowFast on this observation, with two pathways that read the same clip at two frame rates. [Certain]
- Slow pathway: few frames, many channels. In the default model it takes \(T = 4\) frames from a 64-frame clip (one every \(\tau = 16\) frames). It is a 3D ResNet whose early layers are really 2D: it uses temporal filters (time size above 1) only in the last two stages, res4 and res5. The paper found that temporal filters in early layers hurt accuracy when frames are far apart: an object moves too much between two sampled frames for a small filter to connect them. [Certain]
- Fast pathway: \(\alpha\) times more frames (\(\alpha = 8\), so 32 frames, one every 2) but only a fraction \(\beta = 1/8\) of the channels. It has temporal filters in every block and never downsamples time. [Certain]
- Lateral connections send Fast features into the Slow pathway after several stages. The shapes differ, so the Fast features are first reshaped. The default is a time-strided convolution (kernel \(5 \times 1^2\), stride \(\alpha\) in time, \(2\beta C\) output channels), and the result is concatenated to the Slow features. [Certain]
- Read the symbols
- \(\{T, S^2, C\}\)
- feature shape: \(T\) time steps, \(S \times S\) spatial positions, \(C\) channels
- \(\alpha\)
- speed ratio: how many times more frames the Fast pathway reads
- \(\beta\)
- channel ratio: the Fast pathway is \(\beta\) times thinner
Why \(\alpha\beta^2\)? A convolution’s FLOPs grow with the number of frames and with input channels times output channels (formula 202.3). The Fast pathway has \(\alpha\) times more frames and \(\beta\) times fewer channels on both sides. With \(\alpha = 8\), \(\beta = 1/8\), the estimate is \(8/64 = 1/8\). The measured numbers in the paper are a bit higher, because the Fast pathway uses temporal filters everywhere: for the ResNet-50 4×16 model, the Slow pathway alone costs 27.3 GFLOPs and the Fast pathway alone 6.4 GFLOPs, so Fast is about 19% of the total (the paper says “∼20%”). [Certain]
The two pathways need each other. On Kinetics-400 (ResNet-50, 4×16), Slow alone reaches 72.6% top-1, Fast alone only 51.7%, and SlowFast with time-strided lateral connections 75.6%, a 3.0-point gain for about 9 more GFLOPs (36.1 in total). [Certain] Explore the sampling in Figure 3.
How to read Figure 3
The default (\(\tau = 16\), \(\alpha = 8\), \(\beta = 1/8\)) gives 4 Slow frames and 32 Fast frames, with Fast costing about 1/8 of Slow. Raise \(\beta\) to 1/4: Fast gets twice as wide and four times as expensive (the paper measured 54.5 GFLOPs in total for \(\beta = 1/4\), against 36.1 for \(1/8\)). Lower \(\tau\): Slow sees more frames but costs more. The speed ratio cannot exceed the Slow stride here, because the Fast pathway cannot read more than every raw frame.
From 3D CNNs to video transformers: tokens from a clip
A Vision Transformer (session 164) cuts an image into \(P \times P\) patches and treats each patch as a token. A 224×224 image with \(P = 16\) gives \(14 \times 14 = 196\) tokens. For video, ViViT (Arnab et al., 2021) studied two ways to make tokens: [Certain]
- Uniform frame sampling: patchify each sampled frame as in ViT, and put all frames’ tokens in one sequence. Time is mixed only by the transformer.
- Tubelet embedding: cut the clip into small 3D blocks of \(t \times h \times w\) pixels (a tubelet) and project each to a vector. This is a 3D convolution with kernel = stride = \((t, h, w)\). Tokens now carry a little motion from the start, and a tubelet of length \(t = 2\) halves the number of tokens. Figure 1, in tubelet mode, runs this on a toy clip.
Example: 16 frames at 224×224 with tubelets of \(2 \times 16 \times 16\) give \(8 \cdot 14 \cdot 14 = 1{,}568\) tokens. That is the VideoMAE setting. 32 frames with the same tubelet (the ViViT setting) give 3,136 tokens.
How do you start a tubelet filter from an image ViT? Inflation (formula 202.4) is one option. ViViT added central frame initialization: put the 2D patch filter in the middle time slice and zeros elsewhere, so at the start the model is exactly a per-frame ViT. On Kinetics (ViViT-B, joint attention), central frame reached 79.2% top-1, filter inflation 77.6%, random initialization 73.2%, and uniform frame sampling 78.5%. [Certain] Figure 2 shows both initializations side by side.
Now the problem. Self-attention compares every token with every other token (session 17). With \(N\) patches per frame and \(F\) time steps there are \(NF\) tokens, and joint space–time attention costs
where \(d\) is the model width (the \(4\) counts \(QK^\top\) and the weighted sum of values, 2 FLOPs per multiply-add each). The cost grows with the square of the clip length and of the frame area. Doubling the frames makes attention four times more expensive.
Divided space–time attention (TimeSformer)
Bertasius et al. (2021) proposed TimeSformer, a ViT for video that compares several ways to split attention. The best one is divided space–time attention: inside each block, first a temporal attention where each patch looks only at the patches in the same position in the other frames, then a spatial attention where it looks only at the patches of its own frame, then the MLP. [Certain]
- Read the symbols
- \(\mathbf{z}_{p,t}\)
- the token for patch position \(p\) in frame \(t\)
- \(\mathbf{q}, \mathbf{k}, \mathbf{v}\)
- query, key and value, computed from the layer-normalized input; the temporal and spatial attentions have separate weight matrices
- \(\mathbf{q}', \mathbf{k}', \mathbf{v}'\)
- computed from \(\mathbf{z}'\), the output of the temporal step
Each query now meets \(F\) keys in the first step and \(N\) in the second, instead of \(NF\). Summed over all tokens:
(The paper counts the class token too: \(NF + 1\) comparisons per patch for joint attention, \(N + F + 2\) for divided.) Does a patch lose information? No: after the temporal step, each token in frame \(t\) already carries what its position saw in all frames; the spatial step then mixes those tokens. Two hops reach every token of the clip. Figure 5 lets you check this cell by cell.
\(N = 196\) patches per frame, \(F = 8\) frames, so \(NF = 1{,}568\) tokens. Joint attention: each query meets 1,568 keys. Divided: \(8 + 196 = 204\) keys. The ratio is \(1568 / 204 = 7.7\times\) (formula 202.9).
With \(d = 768\) (ViT-Base), joint attention costs \(4 \cdot 1568^2 \cdot 768 = 7.55\) GFLOPs per layer; divided costs \(4 \cdot 1568 \cdot 204 \cdot 768 = 0.98\) GFLOPs. The linear layers (QKV, output projection and MLP, about \(24\, d^2\) FLOPs per token) cost \(22.2\) GFLOPs per layer either way. So at 8 frames, joint attention is only 25% of the layer cost.
At 96 frames (the TimeSformer-L setting) the picture flips: 18,816 tokens, and joint attention costs 1,088 GFLOPs per layer, 80% of the layer. One FP16 attention matrix for one head would take \(18816^2 \cdot 2 = 0.71\) GB. Divided attention needs \(96 + 196 = 292\) keys per query.
The paper’s comparison on Kinetics-400 / Something-Something V2 (ViT-Base, 8 frames): [Certain]
| Attention | Parameters | K400 top-1 | SSv2 top-1 |
|---|---|---|---|
| Space only (no time mixing) | 85.9M | 76.9 | 36.6 |
| Joint space–time | 85.9M | 77.4 | 58.5 |
| Divided space–time | 121.4M | 78.0 | 59.5 |
| Sparse local global | 121.4M | 75.9 | 56.3 |
| Axial (time, then width, then height) | 156.8M | 73.5 | 56.2 |
Two lessons sit in this table. First, on Kinetics, “space only” is just 0.5 points behind joint attention: many Kinetics classes can be read from appearance. On SSv2, where motion direction decides the label, space only collapses to 36.6%. Always test a video model on a motion-heavy dataset. Second, divided attention is better and cheaper. Joint attention ran out of GPU memory at 448-pixel frames or at 32 frames; divided attention scaled to TimeSformer-HR (16 × 448 × 448) and TimeSformer-L (96 frames at 224, sampled at 1/4 of the frame rate). [Certain] Axial attention splits further (time, then height, then width) and is the cheapest per query, but here it was the least accurate.
How to read Figure 4
Start at 8 frames, 224 px, patch 16: 1,568 tokens and a joint/divided ratio of 7.7×. Slide to 96 frames: joint attention grows by \(12^2 = 144\times\), divided by much less, and the readout shows attention taking most of the layer cost. Choose 448 px: four times more patches per frame. Choose tubelet \(t = 2\): half the time steps, and every cost goes down. “Space only” is the cheapest but cannot see motion at all; axial is cheaper than divided but less accurate in the TimeSformer study.
The other ViViT designs, with measured cost
ViViT compared four ways to place time inside the transformer, all with ViViT-B and tubelets of 16×16×2: [Certain]
- Model 1, joint space–time attention: all tokens in one sequence (as above).
- Model 2, factorised encoder: a spatial transformer runs on each frame (or tubelet slice) and gives one vector per time step; a second, temporal transformer runs on those vectors. This is late fusion done well: time mixing happens only at the top.
- Model 3, factorised self-attention: spatial then temporal attention inside every block, like TimeSformer with the order swapped (ViViT found the order makes no difference with its initialization). The temporal attention weights start at zero, so the block starts as the pretrained image block.
- Model 4, factorised dot-product: half of the heads attend over space and half over time, with the same parameters as Model 1.
| ViViT-B, 16×2 tubelets | K400 top-1 | Epic Kitchens | GFLOPs | Params | Runtime (TPU-v3) |
|---|---|---|---|---|---|
| Model 1: joint | 80.0 | 43.1 | 455.2 | 88.9M | 58.9 ms |
| Model 2: factorised encoder | 78.8 | 43.7 | 284.4 | 115.1M | 17.4 ms |
| Model 3: factorised self-attention | 77.4 | 39.1 | 372.3 | 117.3M | 31.7 ms |
| Model 4: factorised dot-product | 76.3 | 39.5 | 277.1 | 88.9M | 22.9 ms |
So in ViViT’s own setting, joint attention was the most accurate on Kinetics, and the factorised encoder was more than 3× faster with only 1.2 points less. The two papers disagree on which factorization wins (TimeSformer found divided attention better than joint). The safe conclusion: the order of splits, the data and the initialization all matter, and the winner depends on the budget. [Likely]
Later video transformers put back the bias that CNNs had. MViT (Fan et al., 2021) is a multiscale transformer: it starts with many tokens and few channels and pools tokens stage by stage, like a CNN pyramid. Video Swin (Liu et al., 2021) computes attention only inside local 3D windows that shift between layers. [Certain] Both trade global attention for locality and speed.
- input: tokens \(\mathbf{z}\) (\(F = 4\) frames \(\times\ N = 16\) patches, width \(d = 2\))
- for each attention step of the block (divided: time, then space):
- \(\mathbf{q} \leftarrow W_q \mathbf{z}_{\text{query}}\); \(\mathbf{k}_j \leftarrow W_k \mathbf{z}_j\), \(\mathbf{v}_j \leftarrow \mathbf{z}_j\) for the keys of this step
- \(s_j \leftarrow \mathbf{q} \cdot \mathbf{k}_j / \sqrt{d}\)
- \(\alpha \leftarrow \softmax(s)\)
- \(\mathbf{z}_{\text{query}} \leftarrow \mathbf{z}_{\text{query}} + \sum_j \alpha_j \mathbf{v}_j\) (202.8); every token does the same, in parallel
- return \(\mathbf{z}\) the MLP of (202.8) is left out here
How to read Figure 5
Press Play on “Divided”. In the time step the query compares itself with the 4 tokens at the same position in the 4 frames, one by one; the softmax turns the 4 scores into weights, and the query becomes the weighted sum (plus itself, the residual). Every other token does the same at the same moment. In the space step the query compares itself with the 16 tokens of its own frame, row by row. Watch the outlines: each key of the space step brings in its whole time line, so after the 20 comparisons all 64 tokens can influence the output. The readout counts this live: 20 comparisons per query and \(64 \times 20 = 1{,}280\) for the layer, against \(64^2 = 4{,}096\) for joint attention, a ratio of \(64/20 = 3.2\) from formula (202.9). “Space only” stops at 16 tokens and never leaves the frame, which is why it fails on SSv2. “Axial” uses three hops of 4 keys (12 comparisons) and still reaches all 64 tokens. “Joint” reads all 64 keys in one step. Click any cell to choose another query; the animation restarts.
- input: tokens \(Z\) of shape \((B, F \cdot N, d)\), in frame-major order
- \(U \leftarrow \mathrm{LN}(Z)\) reshaped to \((B \cdot N, F, d)\) // one short sequence per patch position
- \(Z \leftarrow Z + \text{reshape back}\big(\mathrm{MHA}_{\text{time}}(U)\big)\) // first line of (202.8): \(F\) keys per query
- \(U \leftarrow \mathrm{LN}(Z)\) reshaped to \((B \cdot F, N, d)\) // one sequence per frame
- \(Z \leftarrow Z + \text{reshape back}\big(\mathrm{MHA}_{\text{space}}(U)\big)\) // second line: \(N\) keys per query
- \(Z \leftarrow Z + \mathrm{MLP}(\mathrm{LN}(Z))\)
- return \(Z\) // the class token (not shown) joins the spatial step of every frame
VideoMAE: learn from unlabeled video by hiding 90% of it
All the models above were trained with labels (Kinetics) on top of ImageNet weights. Labels for video are slow and expensive to make. Masked autoencoding (session 165) learns from raw data: hide most of the input, and train the model to rebuild it. Image MAE hides 75% of the patches. VideoMAE (Tong et al., 2022) brings this to video with two changes. [Certain]
1. An extremely high masking ratio: 90% to 95%. Video is redundant: neighboring frames are almost the same, so the information per token is much lower than in an image. With 75% masking the task is too easy. VideoMAE takes 16 frames (stride \(\tau = 4\) on Kinetics, 2 on SSv2), cuts them into \(2 \times 16 \times 16\) cubes (1,568 tokens), and hides 90%. [Certain]
2. Tube masking. If you mask patches at random, a hidden patch is often visible in the next frame at the same place. The model can then copy it instead of understanding anything. The paper calls this information leakage. Tube masking draws one random spatial mask and applies it to every frame, so a hidden position is hidden for the whole clip:
- Read the symbols
- \(M_{t,p} = 1\)
- the token at position \(p\) in time step \(t\) is hidden
- \(\rho\)
- the masking ratio, for example 0.9
- \(N\)
- patch positions per time step (196 at 224 px with 16-pixel patches)
Only the visible tokens go into the encoder, a plain ViT with joint space–time attention (joint attention is affordable because so few tokens remain). A small decoder gets the encoded visible tokens plus learned mask tokens at the hidden positions, and predicts the pixels of the hidden cubes. The loss is the mean squared error on the hidden cubes only, against pixel values normalized inside each cube:
where \(\Omega\) is the set of hidden tokens, \(\mathbf{x}_i\) the pixels of cube \(i\), and \(\hat{\mathbf{x}}_i\) the decoder’s prediction.
16 frames \(\to\) 8 time steps of \(14 \times 14 = 196\) positions: 1,568 tokens. The reference code hides \(\lfloor 0.9 \cdot 196 \rfloor = 176\) positions per time step, so \(20 \times 8 = 160\) tokens are visible: 10.2% of the clip. The encoder’s linear layers do about 10% of the work of a full pass, and its attention about \((160/1568)^2 = 1\%\). At 95% masking only 80 tokens are left.
The ablations on SSv2 (ViT-B, 800 epochs): tube masking at 75% gives 68.0%, tube at 90% gives 69.6%, random masking at 90% gives 68.3%, and frame masking (hide 14 of 16 frames) gives 61.5%. Training from scratch without pretraining gives 32.6%. [Certain]
How to read Figure 6
With tube masking, no hidden token is ever copyable, at any ratio: every hidden position is hidden in every frame. Switch to random masking at 90%: more than 40% of the hidden tokens are visible at the same spot in some other frame, so the reconstruction task leaks. Frame masking hides whole frames (7 of 8 at 90%), and almost every hidden static token can be copied from the one visible frame; only the moving object must be predicted. This matches the paper’s order: tube is best, then random, then frame.
Two results made VideoMAE important. First, it is data-efficient: it works on datasets of only 3k–4k videos, without extra data. Pretrained and fine-tuned only on HMDB-51 (3.5k training videos), it reached 62.6%. [Certain] Second, it is cheap to pretrain because the encoder sees so little: on SSv2 it trained 3.2× faster in wall-clock time than a MoCo v3 contrastive baseline (19.5 vs 61.7 hours on 64 V100 GPUs) and was more accurate (69.6% vs 54.2%). [Certain] VideoMAE V2 (Wang et al., 2023) added dual masking (the decoder also sees only a subset of tokens) and scaled the encoder to a billion-parameter ViT-g. [Certain] V-JEPA (Bardes et al., 2024) keeps the masking idea but predicts the hidden regions in feature space instead of pixels. [Certain]
- input: a batch of clips \(V\) of shape \((B, 3, 16, 224, 224)\), masking ratio \(\rho = 0.9\)
- \(X \leftarrow \mathrm{TubeletEmbed}(V) + \text{pos}\) // \((B, 1568, d)\), formula (202.6)
- draw one spatial mask \(m\) per clip with \(\lfloor \rho \cdot 196 \rfloor\) hidden positions; repeat it over the 8 time steps // (202.10)
- \(H \leftarrow \mathrm{Encoder}(X[\text{visible}])\) // 160 tokens, joint attention
- \(Y \leftarrow \mathrm{Decoder}\big(\text{scatter}(H) \cup \{\text{mask token} + \text{pos at hidden places}\}\big)\)
- \(\mathcal{L} \leftarrow \mathrm{MSE}\big(Y[\text{hidden}],\ \mathrm{norm}(\text{cubes of } V)[\text{hidden}]\big)\) // (202.11)
- update encoder and decoder; after pretraining, keep only the encoder and fine-tune it with all tokens
Video–text encoders: features that speak the language of text
A classifier needs features that separate 400 actions. A video LLM or a retrieval system needs more: features that line up with any sentence. The tool is the same as CLIP for images (session 166): a video encoder and a text encoder trained so that matching (video, caption) pairs have a high cosine similarity and non-matching pairs a low one. With \(B\) pairs in a batch, normalized embeddings \(\mathbf{v}_i\) and \(\mathbf{t}_i\), and temperature \(\tau\):
The first term asks each video to pick its own caption among the \(B\) captions; the second asks each caption to pick its video.
Take a batch of 3 and \(\tau = 0.1\). Video 1 has cosine similarities 0.62, 0.20, 0.15 with captions 1, 2, 3. The logits are 6.2, 2.0, 1.5, and the softmax gives 0.976 to its own caption. Its loss term is \(-\ln 0.976 = 0.024\). With a full similarity matrix of rows (0.62, 0.20, 0.15), (0.25, 0.55, 0.30), (0.10, 0.35, 0.48), the video-to-text loss is 0.135, the text-to-video loss 0.122, and their average 0.129.
The simple way to build a video–text encoder is to start from CLIP and pool frame features over time; CLIP4Clip did this for retrieval (session 208). Its weakness is the one from formula (202.1): pooled frame features are weak on motion. The newer recipe trains a true video encoder and aligns it with text. Two steps led there. UMT (Unmasked Teacher, Li et al., 2023) masks most video tokens, like VideoMAE, but instead of rebuilding pixels it trains the visible tokens to match the features of a CLIP image teacher: semantic targets, fast convergence, and features that are friendly to text. [Certain] InternVideo2 scales this idea up.
InternVideo2: a 6B video encoder in three stages
InternVideo2 (Wang et al., 2024) trains a ViT video encoder with 6B parameters (48 layers, width 3,200; 8 frames at 224 px with 14×14 patches, so \(8 \times 256 = 2{,}048\) tokens, and an attention-pooling head that outputs a 768-dimensional clip vector). [Certain] Training has three stages, each adding something the previous one lacks:
- Unmasked token distillation. Hide 80% of the tokens. Feed the full clip to two frozen teachers: InternViT-6B (the vision encoder of the InternVL multimodal model, for semantics) and VideoMAE V2-g (for motion). Train the student so that its visible tokens match the teachers’ tokens at the same positions, through small projection heads that are dropped afterwards:
\[ \mathcal{L}_{1} = \frac{1}{Z} \sum_{p\, \in\, \text{visible}} \Big( \alpha_1 \big\lVert f(V)_p - h(V)_p \big\rVert^2 + \alpha_2 \big\lVert f(V)_p - g(V)_p \big\rVert^2 \Big) \tag{202.13} \]Here \(f\) is the student, \(h\) the InternViT teacher, \(g\) the VideoMAE V2 teacher, and \(Z\) a normalizer. In practice they align several of the last layers (the last 6 of InternViT, the last 4 of VideoMAE V2, and InternViT’s final output token). [Certain]
- Video–audio–speech–text alignment. Add an audio encoder (a 12-layer transformer initialized from BEATs, 90M parameters) and a text encoder from BERT-Large. Train with contrastive loss (202.12), a video–text matching loss and a masked language loss, on image–text, video–text and video–audio–speech–text pairs whose captions were fused from what is seen, heard and said. [Certain]
- Next-token prediction. Connect the encoder to an LLM and train on video instruction and question–answer data, so the encoder is also updated by the LLM’s loss. [Certain]
The training data has 402M entries: 2M videos for stage 1, 50M video–text pairs, 50M video–audio–speech–text pairs, and 300M image–text pairs. The paper reports strong results on more than 60 video and audio tasks. [Certain] The point for design: each stage gives one property. Distillation from a motion teacher gives temporal sense; contrastive alignment gives text-ready features for retrieval and zero-shot classification; next-token training makes the features useful to an LLM (session 205).
Choosing a video encoder
| You need | Good choice | Why |
|---|---|---|
| Fast action classification on a GPU budget | SlowFast or X3D (3D CNN), or a small fine-tuned VideoMAE ViT-B | Mature, cheap per clip, strong on motion; easy to fine-tune with a small labeled set |
| Fine-grained motion (direction, sports techniques) | Video transformer pretrained with VideoMAE, fine-tuned; or a temporal-aware encoder such as InternVideo2 | Space-only or CLIP-pooled features fail on motion (SSv2: 36.6% for space-only TimeSformer) |
| Text-to-video search, zero-shot labels | A video–text encoder (InternVideo2 stage 2, UMT, or CLIP with temporal pooling as a baseline) | The embedding must live in the same space as text, formula (202.12) |
| Input to a video LLM | A video–text encoder or an image encoder applied per frame plus token compression | The LLM needs text-aligned tokens; the token budget per second decides cost (session 206) |
| Long videos (minutes to hours) | Divided or windowed attention, short clips encoded separately, then a temporal model on top | Joint attention grows with the square of the number of frames (formula 202.7) |
| Pretraining on your own unlabeled footage | VideoMAE-style masked pretraining | Works with a few thousand videos, no labels |
Putting it together in PyTorch
The script below builds the four key pieces of this session at toy sizes: an I3D inflation with the boring-video check, a tubelet embedding, a divided space–time block, tube masking, and the symmetric video–text contrastive loss. It runs on a CPU in about a second and prints the same numbers every time.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) I3D inflation: a 2D conv becomes a 3D conv, formula (202.4) def inflate(conv2d, kt): c3 = nn.Conv3d(conv2d.in_channels, conv2d.out_channels, kernel_size=(kt, *conv2d.kernel_size), padding=(kt // 2, *conv2d.padding), bias=conv2d.bias is not None) with torch.no_grad(): w = conv2d.weight.unsqueeze(2).repeat(1, 1, kt, 1, 1) / kt # repeat N times, divide by N c3.weight.copy_(w) if conv2d.bias is not None: c3.bias.copy_(conv2d.bias) return c3 conv2d = nn.Conv2d(3, 8, 3, padding=1) conv3d = inflate(conv2d, kt=3) img = torch.randn(1, 3, 32, 32) boring = img.unsqueeze(2).repeat(1, 1, 5, 1, 1) # (B, C, T, H, W): same frame 5 times y2, y3 = conv2d(img), conv3d(boring) print("boring-video check:", (y3[:, :, 2] - y2).abs().max().item()) # middle frame: about 1e-6 # 2) Tubelet embedding (ViViT / VideoMAE): a Conv3d with kernel = stride = (2, 16, 16) d = 96 embed = nn.Conv3d(3, d, kernel_size=(2, 16, 16), stride=(2, 16, 16)) video = torch.randn(1, 3, 16, 224, 224) tok = embed(video) # (1, d, 8, 14, 14) B, D, Tp, Hp, Wp = tok.shape x = tok.flatten(2).transpose(1, 2) # (1, 1568, d), formula (202.6) print("tokens:", x.shape) # 3) Divided space-time attention block (TimeSformer), formula (202.8) class DividedBlock(nn.Module): def __init__(self, d, heads): super().__init__() self.n1, self.n2, self.n3 = nn.LayerNorm(d), nn.LayerNorm(d), nn.LayerNorm(d) self.t_attn = nn.MultiheadAttention(d, heads, batch_first=True) self.s_attn = nn.MultiheadAttention(d, heads, batch_first=True) self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) def forward(self, x, T, N): # x: (B, T*N, d), frame-major B, _, d = x.shape h = self.n1(x).reshape(B, T, N, d).transpose(1, 2).reshape(B * N, T, d) h = self.t_attn(h, h, h, need_weights=False)[0] # attend over T frames, same patch x = x + h.reshape(B, N, T, d).transpose(1, 2).reshape(B, T * N, d) h = self.n2(x).reshape(B * T, N, d) h = self.s_attn(h, h, h, need_weights=False)[0] # attend over N patches, same frame x = x + h.reshape(B, T * N, d) return x + self.mlp(self.n3(x)) blk = DividedBlock(d, heads=4) out = blk(x, Tp, Hp * Wp) print("divided block out:", out.shape) N = Hp * Wp print("keys per query: joint", Tp * N, " divided", Tp + N) # 4) Tube masking (VideoMAE), formula (202.10): one spatial mask, shared by all frames def tube_mask(T, N, ratio, g): m = int(ratio * N) # masked positions per frame keep = torch.randperm(N, generator=g)[: N - m] # visible positions mask = torch.ones(T, N, dtype=torch.bool); mask[:, keep] = False return mask.flatten() # True = masked g = torch.Generator().manual_seed(0) mask = tube_mask(Tp, N, 0.9, g) visible = x[:, ~mask] # only these go to the encoder print("visible tokens:", visible.shape[1], "of", mask.numel()) # 5) Symmetric video-text contrastive loss, formula (202.12) v = F.normalize(torch.randn(4, 64), dim=-1); t = F.normalize(v + 0.3 * torch.randn(4, 64), dim=-1) logits = v @ t.T / 0.07 labels = torch.arange(4) loss = (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) / 2 print("contrastive loss:", round(loss.item(), 4))
Output (PyTorch 2.8, CPU):
boring-video check: 1.430511474609375e-06 tokens: torch.Size([1, 1568, 96]) divided block out: torch.Size([1, 1568, 96]) keys per query: joint 1568 divided 204 visible tokens: 160 of 1568 contrastive loss: 0.0315
The boring-video check compares the middle frame. The first and last frames differ from the 2D output, because zero padding in time puts empty frames next to them: inflation is exact only where the filter sees real frames.
Where this is used
| Application | Encoder role | Typical choice |
|---|---|---|
| Action recognition (Kinetics, sports moves) | Clip vector → classifier; average over several clips for a whole video | SlowFast, VideoMAE fine-tuned, InternVideo2 (session 203) |
| Text-to-video retrieval | Clip vector in a shared space with text, searched with a vector index | Video–text encoder, CLIP4Clip as a baseline (session 208) |
| Key-moment detection and temporal grounding | A feature per short clip along the timeline; a head predicts start and end | Frozen clip features (I3D and SlowFast features were the standard; now video–text features) (session 207) |
| Video fingerprinting and near-duplicates | Compact per-segment vectors that survive re-encoding and crops | Frame-level image features plus temporal alignment; video encoders when motion matters (session 209) |
| Video segmentation and tracking | Dense per-patch features over time | Image backbones with memory (SAM 2) more than clip encoders (session 204) |
| Video LLMs and Video-RAG | Visual tokens for the LLM, or clip embeddings for retrieval | Video–text encoder or per-frame image encoder with token merging (session 205, session 210) |
Common mistakes
- Judging a video model only on Kinetics. Appearance solves much of Kinetics; space-only attention is 0.5 points behind joint there but 22 points behind on SSv2. Test on a motion-heavy set.
- Inflating without dividing by N. The responses become N times too large and the pretrained batch-norm statistics no longer fit. Use repeat ÷ N or central frame initialization.
- Mixing up frame-major and patch-major order when reshaping. In a divided block, the reshape to \((B \cdot N, F, d)\) must group tokens of the same position. A wrong transpose still runs, and silently attends over the wrong tokens.
- Random masking in masked video pretraining. Hidden patches leak from neighboring frames, and the model learns to copy. Use tube masking (or motion-aware masks) and a high ratio.
- Forgetting the sampling stride. A model trained on 16 frames with stride 4 (2.1 s at 30 fps) sees a different speed of motion if you feed stride 1 at test time. Keep the training sampling, or train with several strides.
- Using pooled CLIP frame features for motion questions. They are excellent for “what is in the video” and weak for “which way did it move”.
- Counting only parameters. TimeSformer has 121.4M parameters but its paper reports 0.59 TFLOPs per inference against 1.97 for SlowFast 8×8 R50 with 34.6M parameters. [Certain] Compare FLOPs, tokens and latency at your clip length.
Interview questions
Why can a per-frame image model with average pooling not tell “opening a door” from “closing a door”?basic
How does I3D initialize 3D filters from ImageNet weights, and why does it work?core
Explain SlowFast. Why is the Fast pathway thin?core
What is divided space–time attention and how much does it save?core
Tubelet embedding vs uniform frame sampling: what is the difference?basic
Why does VideoMAE use 90% masking when image MAE uses 75%, and why tube masking?core
How much cheaper is VideoMAE pretraining than a full forward pass, and where does the saving come from?advanced
Why are VideoMAE features not enough for a video LLM or retrieval, and how does InternVideo2 fix it?advanced
You must build text-to-video search over a sports archive. Which encoder, and what are the trade-offs?advanced
Joint attention on 96 frames: estimate the cost and what you would do instead.advanced
SlowFast uses temporal filters only in res4 and res5 of the Slow pathway. Why?core
Exercises
- A clip has 32 frames at 224×224. How many tokens does a ViT with \(16 \times 16\) patches produce with uniform frame sampling? With tubelets of \(2 \times 16 \times 16\)? With \(4 \times 16 \times 16\)?
- For the 16-tubelet case of exercise 1 (\(t = 2\)), compute the joint/divided attention ratio of formula (202.9).
- Compute the parameters of a \(3 \times 3 \times 3\) convolution with 128 input and 256 output channels, and of its \(1 \times 3 \times 3\) + \(3 \times 1 \times 1\) factorization with a middle width of 256.
- A 2D filter has weights summing to 2.0. After I3D inflation with \(N = 5\), what is the sum of the weights in each time slice and in total?
- SlowFast with \(\alpha = 4\), \(\beta = 1/4\): estimate the Fast/Slow compute ratio. How many frames does each pathway read from a 64-frame clip with \(\tau = 16\)?
- VideoMAE with masking ratio 0.95 on 16 frames at 224 px: how many tokens does the encoder see? (Use the per-frame rounding of the reference code.)
- In the toy clip of Figure 6 (8 frames), with random masking at ratio 0.9, estimate the chance that a hidden static token is visible at the same place in at least one of the other 7 frames.
Answerscheck
Key takeaways
- Averaging per-frame features (202.1) ignores order; video encoders add time with 3D filters (202.2), temporal attention or optical flow.
- I3D inflates 2D filters by repeating them and dividing by \(N\) (202.4), so ImageNet weights give the same output on a still video; SlowFast reads appearance slowly with many channels and motion fast with few, at about \(\alpha\beta^2\) extra cost (202.5).
- Video transformers turn clips into \(n = \lfloor T/t \rfloor \lfloor H/h \rfloor \lfloor W/w \rfloor\) tokens (202.6). Joint attention costs \((NF)^2\) pairs (202.7); divided attention (202.8) costs \(NF(N + F)\), 7.7× less at 8 frames (202.9), without losing reach.
- VideoMAE pretrains on unlabeled video by hiding 90% of the tokens with a tube mask (202.10) and rebuilding normalized pixels (202.11): data-efficient and cheap, but features need fine-tuning.
- Video–text encoders add contrastive alignment (202.12); InternVideo2 distills semantic and motion teachers (202.13), aligns with audio, speech and text, then trains with an LLM.
Key formulas
References for this session
- Carreira & Zisserman (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
- Feichtenhofer et al. (2018). SlowFast Networks for Video Recognition
- Arnab et al. (2021). ViViT: A Video Vision Transformer
- Bertasius et al. (2021). Is Space-Time Attention All You Need for Video Understanding?
- Tong et al. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
- Wang et al. (2024). InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
- Kay et al. (2017). The Kinetics Human Action Video Dataset
- Simonyan & Zisserman (2014). Two-Stream Convolutional Networks for Action Recognition in Videos
- Tran et al. (2015). Learning Spatiotemporal Features with 3D Convolutional Networks
- Tran et al. (2018). A Closer Look at Spatiotemporal Convolutions for Action Recognition
- Feichtenhofer (2020). X3D: Expanding Architectures for Efficient Video Recognition
- Fan et al. (2021). Multiscale Vision Transformers
- Liu et al. (2021). Video Swin Transformer
- Wang et al. (2023). VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
- Li et al. (2023). Unmasked Teacher: Towards Training-Efficient Video Foundation Models
- Bardes et al. (2024). Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA)
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision