LLM & VLM courseBlog
202Track 8: Video and multimodal systems · session 2 of 21

Video encoders: from 3D CNNs to video transformers

A video encoder turns a clip of frames into vectors that know about both appearance and motion. This session follows the main designs in order: 3D CNNs (C3D, I3D with filters inflated from image models, SlowFast with two frame rates), video transformers (ViViT and TimeSformer, with attention split into space and time), masked video pretraining (VideoMAE, which hides 90% of the clip), and video–text encoders such as InternVideo2 that feed today’s video LLMs.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Session 164 (the Vision Transformer), session 165 (MAE), session 166 (CLIP), and session 201 (video as data)

What you will learn

  • Why averaging per-frame features loses motion and order, and what a 3D convolution adds
  • How I3D inflates a 2D image network into 3D and keeps the ImageNet weights useful (the boring-video fixed point)
  • How SlowFast splits a clip into a slow, wide pathway for appearance and a fast, thin pathway for motion
  • How ViViT and TimeSformer turn a clip into tokens, and why divided space–time attention is about 7.7× cheaper than joint attention for 8 frames
  • Why VideoMAE can hide 90% of the tokens, and why the mask must be a tube through time
  • How InternVideo2 trains a 6B video encoder in three stages: distillation, video–text contrastive learning, and next-token prediction
  • How to choose a video encoder for classification, retrieval, key-moment detection and video LLMs

Why this matters

A video is a stack of images, but a video model that only looks at images misses the most important thing: what changes. “Opening a door” and “closing a door” have almost the same frames, in a different order. A video encoder is the network that turns a clip into vectors that carry both appearance (what is there) and motion (what happens).

Every video system in this track starts with one. Action recognition (session 203), text-to-video retrieval (session 208), key-moment detection (session 207) and video LLMs (session 205) all put a video encoder in front. The choice of encoder decides three things: how well the system sees motion, how many tokens (and so how much compute) one second of video costs, and whether the features already speak the language of text.

The history is short and each step fixes one problem of the step before:

  • 3D CNNs (C3D, I3D, SlowFast, 2015–2019): convolutions that slide over time as well as space. I3D found a way to reuse ImageNet weights; SlowFast found a cheap way to see fast motion.
  • Video transformers (ViViT, TimeSformer, 2021): a clip becomes a long sequence of patch tokens. Full attention over all of them is too expensive, so attention is split into a time part and a space part.
  • Masked video pretraining (VideoMAE, 2022): learn from unlabeled video by hiding 90% of the tokens and rebuilding them.
  • Video–text encoders (UMT, InternVideo2, 2023–2024): train the encoder so that its output lines up with text, then plug it into an LLM.

In session 201 we saw that video is expensive before any model runs: decoding, sampling and the token budget. This session is about the model that consumes the sampled frames. Session 203 then uses these encoders on real tasks.

The baseline: a 2D network on each frame

The simplest video model takes an image network \(g\) (a ResNet or a ViT), runs it on each sampled frame \(x_1, \dots, x_T\), averages the frame features, and classifies the average:

\[ f(V) = h\!\left( \frac{1}{T} \sum_{t=1}^{T} g(x_t) \right) \tag{202.1} \]
Read the symbols
\(V = (x_1, \dots, x_T)\)
the clip: \(T\) frames, each an image of size \(H \times W \times 3\)
\(g\)
the image network, applied to each frame on its own
\(h\)
the classifier head (often one linear layer)

This is called late fusion: frames meet only at the very end. It is cheap, it reuses strong image models, and on datasets where the scene gives the answer (“playing piano”, “surfing”) it works well. But look at formula (202.1): a sum does not care about order. Shuffle the frames, or play the clip backwards, and the output is exactly the same. So the model cannot tell “opening” from “closing”, or “pushing something left to right” from “right to left”. Datasets such as Something-Something were built with exactly such labels, where one frame is not enough. [Certain]

Two early fixes:

  • Two-stream networks (Simonyan and Zisserman, 2014): a second 2D network reads optical flow, the per-pixel motion between neighboring frames, computed by a classic algorithm. Motion now enters as an input. The price is computing flow for every frame, which is slow. [Certain]
  • 3D convolutions: let the filters themselves look at several frames. This is the next section.

3D convolution: filters that slide through time

A 2D convolution slides a \(k \times k\) filter over the image. A 3D convolution slides a \(k_t \times k \times k\) filter over the clip: \(k_t\) frames deep, \(k\) pixels high and wide. One output value now depends on a small cube of space and time, so the filter can respond to an edge that moves.

\[ y_{o}(t, i, j) = b_o + \sum_{c=1}^{C_{\text{in}}} \sum_{\tau, u, v} W_{o, c}(\tau, u, v)\; x_c(t + \tau,\ i + u,\ j + v) \tag{202.2} \]
Read the symbols
\(x_c(t, i, j)\)
input channel \(c\) at frame \(t\), row \(i\), column \(j\)
\(W_{o,c}(\tau, u, v)\)
filter weight for output channel \(o\), input channel \(c\), time offset \(\tau\) and space offsets \(u, v\)
\(y_o(t, i, j)\)
the output feature map, which still has a time axis

The cost grows with the time size of the filter and with the number of frames:

\[ \text{params} = k_t\, k^2\, C_{\text{in}} C_{\text{out}}, \qquad \text{FLOPs} = 2\, k_t\, k^2\, C_{\text{in}} C_{\text{out}} \cdot T' H' W' \tag{202.3} \]

where \(T' \times H' \times W'\) is the size of the output. C3D (Tran et al., 2015) was a VGG-like network built only from \(3 \times 3 \times 3\) filters, trained from scratch on a large sports dataset. [Certain] It showed that 3D filters learn motion, but training from scratch needs a lot of labeled video, and there was no ImageNet for video yet.

Worked example: one layer, 2D vs 3D

A \(3 \times 3\) convolution with 64 input and 64 output channels has \(3 \cdot 3 \cdot 64 \cdot 64 = 36{,}864\) weights. The \(3 \times 3 \times 3\) version has \(110{,}592\): three times more.

On an 8-frame feature map of \(56 \times 56\), the 3D layer costs \(2 \cdot 27 \cdot 64 \cdot 64 \cdot 8 \cdot 56 \cdot 56 = 5.55\) GFLOPs. Running the 2D layer on each of the 8 frames costs \(1.85\) GFLOPs. Again three times more, for one layer.

A factorized block, a \(1 \times 3 \times 3\) spatial filter followed by a \(3 \times 1 \times 1\) temporal filter with the same width, has \(36{,}864 + 12{,}288 = 49{,}152\) weights, 44% of the full 3D filter. This is the idea of R(2+1)D (Tran et al., 2018), which also found it more accurate than full 3D filters. [Certain] It is the convolution version of the “split space and time” idea that comes back in video transformers.

Figure 1 runs formula (202.2) on a toy clip, one output value at a time. The clip has 4 frames of 6×6 pixels and one channel; a 2×2 bright square moves one pixel to the right per frame. The default kernel is a hand-made motion detector: it adds the pixels on a line that moves right by one pixel per frame and subtracts the pixels on a line that moves left. A bar that does not move gives exactly 0, because both lines see it. No 2D filter can do this, since a 2D filter never sees two frames.

pixel = 1 (the moving square)current window (cube)positive weight or outputnegative weight or output
  1. for each output position \((t', i', j')\) of the \(T' \times H' \times W'\) grid:
  2.   cube \(\leftarrow x[s t' : s t' + k_t,\ s i' : s i' + k,\ s j' : s j' + k]\) stride \(s\)
  3.   for each output channel \(o\): \(y_o(t', i', j') \leftarrow b_o + \sum W_o \odot \text{cube}\) (202.2)
  4. return \(y\): \(T'H'W'\) positions \(\times\ C_{\text{out}}\) channels with \(s =\) kernel size these are the \(n\) tokens of (202.6)
Figure 1. A 3D convolution computed in your browser, one output value per step. Top: the 4 input frames; the blue boxes are the cube that the kernel reads now (\(k_t = 3\) frames deep). Bottom left: the kernel weights, one 3×3 slice per time offset \(\tau\). Bottom right: the output maps, which still have a time axis (\(T' = 4 - 3 + 1 = 2\)). The box under the pseudo-code shows formula (202.2) with the real numbers of the current step (bias \(b = 0\)). In tubelet mode the same loop runs with stride = kernel size and two output channels, and each output position becomes one token.
How to read Figure 1

Press Play, or Step through it. Each output position takes two steps: first the cube is gathered (line 2), then the weighted sum is written (line 3). With the right-motion detector and the square moving right, the largest output is +2; with the square moving left it is −2; with a square that does not move, every output is exactly 0. Switch to “inflated edge ÷ 3” (the I3D weights of formula 202.4, explained in the next section): on the still square it gives ±3, the same as the 2D Sobel filter; on the moving square the response is smeared and its peak drops to 2. Now pick “Tubelet embedding”: the cubes no longer overlap, every pixel is read exactly once, and 2×2×2 tubelets give \(2 \cdot 3 \cdot 3 = 18\) tokens of 2 numbers each: a mean brightness (channel 1) and a “last frame minus first frame” value (channel 2). The second channel is not 0 only where the square moves inside a tubelet. With 1×2×2 tubelets (one frame deep) it is always 0: a token that sees one frame cannot carry motion.

I3D: inflate an image network into a video network

Carreira and Zisserman (2017) asked a practical question: image architectures and their ImageNet weights took years to develop. Can we reuse both? Their answer, I3D (Inflated 3D ConvNet), has two parts. [Certain]

Inflate the architecture. Take a 2D network (they used Inception-v1 with batch norm) and give every filter and pooling kernel a time dimension: \(N \times N\) becomes \(N \times N \times N\). The layer structure stays the same.

Inflate the weights. Turn a picture into a “boring video” by repeating it in every frame. A good 3D network started from ImageNet should give the same answer on this boring video as the 2D network gives on the picture. The paper calls this the boring-video fixed point. Because convolution is linear, this is easy: copy the 2D filter into each of the \(N\) time slices and divide by \(N\):

\[ W^{3D}(\tau, u, v) = \frac{1}{N}\, W^{2D}(u, v), \quad \tau = 1, \dots, N \quad\Longrightarrow\quad \sum_{\tau=1}^{N} \frac{1}{N}\, W^{2D} * x = W^{2D} * x \ \text{ when every frame is } x \tag{202.4} \]

On a boring video every frame gives the same 2D response, and the average of \(N\) equal numbers is that number. Pointwise nonlinearities and pooling then keep the outputs unchanged, so the whole network gives the ImageNet answer at initialization. From there, training on video teaches the filters to use time. [Certain]

Other details matter too. I3D does not pool over time in the first two max-pool layers (kernels \(1 \times 3 \times 3\)), so early layers keep the fine time detail at 25 fps. It trains on 64-frame snippets and tests on whole videos by averaging predictions over time. And it still uses two streams: one I3D on RGB frames and one on optical flow (TV-L1), with the two predictions averaged. [Certain] The paper also re-evaluated video architectures on the then-new Kinetics dataset (400 human action classes, more than 400 clips per class), which made large-scale video pretraining possible. After Kinetics pretraining, I3D reached 80.9% on HMDB-51 and 98.0% on UCF-101. [Certain]

Figure 2. Inflating a 2D edge filter (a 3×3 Sobel filter that finds vertical edges) into a 3D filter. Top: five 7×7 input frames. Bottom: the 2D filter’s response on the middle frame, and the 3D filter’s response at the same frame. The readout compares their peaks.
How to read Figure 2

Start with the boring video and “Repeat ÷ N”: the two response maps are identical, for any \(N\). That is formula (202.4). Switch to “Repeat, no ÷ N”: the 3D response is \(N\) times too large, so every layer after it would see inputs far outside what it was trained on. Now pick “Moving edge”: the edge moves one pixel per frame. The inflated filter averages \(N\) frames, so its response is spread and its peak drops (to \(2/3\) of the 2D peak for \(N = 3\)). The 3D filter no longer matches any single frame: training on video must now teach it to track the motion. “Central frame” puts the 2D filter only in the middle time slice and zeros elsewhere; it also passes the boring-video test, and it is exactly a per-frame 2D network at the start. ViViT found it works better for transformers (next sections).

Algorithm 1: inflate a pretrained 2D network into a 3D network (I3D)
  1. input: a pretrained 2D network, a time size \(N_\ell\) for each layer
  2. for each convolution with weight \(W^{2D}\) of shape \((C_{\text{out}}, C_{\text{in}}, k, k)\) do
  3.   create \(W^{3D}\) of shape \((C_{\text{out}}, C_{\text{in}}, N_\ell, k, k)\)
  4.   \(W^{3D}(\tau) \leftarrow W^{2D} / N_\ell\) for every \(\tau\) // formula (202.4)
  5.   copy the bias and the batch-norm statistics unchanged
  6. for each pooling layer do give it a time size (often 1 in early layers, so time is not reduced too early)
  7. check: on a clip that repeats one image, the 3D network output equals the 2D network output on that image
  8. fine-tune on video clips (for example 64-frame snippets of Kinetics)

SlowFast: one pathway for “what”, one for “how it moves”

A 3D CNN treats space and time the same way. But they are not the same. The identity of things in a scene changes slowly: a person is still a person 0.5 seconds later. Motion can be fast: a hand clapping, a ball hit. Feichtenhofer et al. (2019) built SlowFast on this observation, with two pathways that read the same clip at two frame rates. [Certain]

  • Slow pathway: few frames, many channels. In the default model it takes \(T = 4\) frames from a 64-frame clip (one every \(\tau = 16\) frames). It is a 3D ResNet whose early layers are really 2D: it uses temporal filters (time size above 1) only in the last two stages, res4 and res5. The paper found that temporal filters in early layers hurt accuracy when frames are far apart: an object moves too much between two sampled frames for a small filter to connect them. [Certain]
  • Fast pathway: \(\alpha\) times more frames (\(\alpha = 8\), so 32 frames, one every 2) but only a fraction \(\beta = 1/8\) of the channels. It has temporal filters in every block and never downsamples time. [Certain]
  • Lateral connections send Fast features into the Slow pathway after several stages. The shapes differ, so the Fast features are first reshaped. The default is a time-strided convolution (kernel \(5 \times 1^2\), stride \(\alpha\) in time, \(2\beta C\) output channels), and the result is concatenated to the Slow features. [Certain]
\[ \text{Slow: } \{T,\ S^2,\ C\}, \qquad \text{Fast: } \{\alpha T,\ S^2,\ \beta C\}, \qquad \frac{\text{FLOPs}_{\text{Fast}}}{\text{FLOPs}_{\text{Slow}}} \approx \alpha \beta^2 \tag{202.5} \]
Read the symbols
\(\{T, S^2, C\}\)
feature shape: \(T\) time steps, \(S \times S\) spatial positions, \(C\) channels
\(\alpha\)
speed ratio: how many times more frames the Fast pathway reads
\(\beta\)
channel ratio: the Fast pathway is \(\beta\) times thinner

Why \(\alpha\beta^2\)? A convolution’s FLOPs grow with the number of frames and with input channels times output channels (formula 202.3). The Fast pathway has \(\alpha\) times more frames and \(\beta\) times fewer channels on both sides. With \(\alpha = 8\), \(\beta = 1/8\), the estimate is \(8/64 = 1/8\). The measured numbers in the paper are a bit higher, because the Fast pathway uses temporal filters everywhere: for the ResNet-50 4×16 model, the Slow pathway alone costs 27.3 GFLOPs and the Fast pathway alone 6.4 GFLOPs, so Fast is about 19% of the total (the paper says “∼20%”). [Certain]

The two pathways need each other. On Kinetics-400 (ResNet-50, 4×16), Slow alone reaches 72.6% top-1, Fast alone only 51.7%, and SlowFast with time-strided lateral connections 75.6%, a 3.0-point gain for about 9 more GFLOPs (36.1 in total). [Certain] Explore the sampling in Figure 3.

Slow pathway frames and channelsFast pathway frames and channels
Figure 3. SlowFast sampling on a 64-frame clip. Each tick is one raw frame. The Slow pathway reads one frame every \(\tau\); the Fast pathway reads one every \(\tau/\alpha\). The bars on the right show channel width (64 channels for Slow in the first stage). The compute ratio uses the estimate \(\alpha\beta^2\) from formula (202.5).
How to read Figure 3

The default (\(\tau = 16\), \(\alpha = 8\), \(\beta = 1/8\)) gives 4 Slow frames and 32 Fast frames, with Fast costing about 1/8 of Slow. Raise \(\beta\) to 1/4: Fast gets twice as wide and four times as expensive (the paper measured 54.5 GFLOPs in total for \(\beta = 1/4\), against 36.1 for \(1/8\)). Lower \(\tau\): Slow sees more frames but costs more. The speed ratio cannot exceed the Slow stride here, because the Fast pathway cannot read more than every raw frame.

From 3D CNNs to video transformers: tokens from a clip

A Vision Transformer (session 164) cuts an image into \(P \times P\) patches and treats each patch as a token. A 224×224 image with \(P = 16\) gives \(14 \times 14 = 196\) tokens. For video, ViViT (Arnab et al., 2021) studied two ways to make tokens: [Certain]

  • Uniform frame sampling: patchify each sampled frame as in ViT, and put all frames’ tokens in one sequence. Time is mixed only by the transformer.
  • Tubelet embedding: cut the clip into small 3D blocks of \(t \times h \times w\) pixels (a tubelet) and project each to a vector. This is a 3D convolution with kernel = stride = \((t, h, w)\). Tokens now carry a little motion from the start, and a tubelet of length \(t = 2\) halves the number of tokens. Figure 1, in tubelet mode, runs this on a toy clip.
\[ n = \left\lfloor \frac{T}{t} \right\rfloor \cdot \left\lfloor \frac{H}{h} \right\rfloor \cdot \left\lfloor \frac{W}{w} \right\rfloor \tag{202.6} \]

Example: 16 frames at 224×224 with tubelets of \(2 \times 16 \times 16\) give \(8 \cdot 14 \cdot 14 = 1{,}568\) tokens. That is the VideoMAE setting. 32 frames with the same tubelet (the ViViT setting) give 3,136 tokens.

How do you start a tubelet filter from an image ViT? Inflation (formula 202.4) is one option. ViViT added central frame initialization: put the 2D patch filter in the middle time slice and zeros elsewhere, so at the start the model is exactly a per-frame ViT. On Kinetics (ViViT-B, joint attention), central frame reached 79.2% top-1, filter inflation 77.6%, random initialization 73.2%, and uniform frame sampling 78.5%. [Certain] Figure 2 shows both initializations side by side.

Now the problem. Self-attention compares every token with every other token (session 17). With \(N\) patches per frame and \(F\) time steps there are \(NF\) tokens, and joint space–time attention costs

\[ \text{pairs}_{\text{joint}} = (NF)^2, \qquad \text{attention FLOPs per layer} \approx 4\, (NF)^2\, d \tag{202.7} \]

where \(d\) is the model width (the \(4\) counts \(QK^\top\) and the weighted sum of values, 2 FLOPs per multiply-add each). The cost grows with the square of the clip length and of the frame area. Doubling the frames makes attention four times more expensive.

Divided space–time attention (TimeSformer)

Bertasius et al. (2021) proposed TimeSformer, a ViT for video that compares several ways to split attention. The best one is divided space–time attention: inside each block, first a temporal attention where each patch looks only at the patches in the same position in the other frames, then a spatial attention where it looks only at the patches of its own frame, then the MLP. [Certain]

\[ \begin{aligned} \mathbf{z}'_{p,t} &= \mathbf{z}_{p,t} + \mathrm{Attn}_{\text{time}}\big(\mathbf{q}_{p,t},\ \{\mathbf{k}_{p,t'}, \mathbf{v}_{p,t'}\}_{t'=1}^{F}\big) \\ \mathbf{z}''_{p,t} &= \mathbf{z}'_{p,t} + \mathrm{Attn}_{\text{space}}\big(\mathbf{q}'_{p,t},\ \{\mathbf{k}'_{p',t}, \mathbf{v}'_{p',t}\}_{p'=1}^{N}\big) \\ \mathbf{z}^{\text{out}}_{p,t} &= \mathbf{z}''_{p,t} + \mathrm{MLP}\big(\mathrm{LN}(\mathbf{z}''_{p,t})\big) \end{aligned} \tag{202.8} \]
Read the symbols
\(\mathbf{z}_{p,t}\)
the token for patch position \(p\) in frame \(t\)
\(\mathbf{q}, \mathbf{k}, \mathbf{v}\)
query, key and value, computed from the layer-normalized input; the temporal and spatial attentions have separate weight matrices
\(\mathbf{q}', \mathbf{k}', \mathbf{v}'\)
computed from \(\mathbf{z}'\), the output of the temporal step

Each query now meets \(F\) keys in the first step and \(N\) in the second, instead of \(NF\). Summed over all tokens:

\[ \text{pairs}_{\text{divided}} = NF\,(N + F), \qquad \frac{\text{pairs}_{\text{joint}}}{\text{pairs}_{\text{divided}}} = \frac{NF}{N + F} \tag{202.9} \]

(The paper counts the class token too: \(NF + 1\) comparisons per patch for joint attention, \(N + F + 2\) for divided.) Does a patch lose information? No: after the temporal step, each token in frame \(t\) already carries what its position saw in all frames; the spatial step then mixes those tokens. Two hops reach every token of the clip. Figure 5 lets you check this cell by cell.

Worked example: TimeSformer on 8 frames at 224×224

\(N = 196\) patches per frame, \(F = 8\) frames, so \(NF = 1{,}568\) tokens. Joint attention: each query meets 1,568 keys. Divided: \(8 + 196 = 204\) keys. The ratio is \(1568 / 204 = 7.7\times\) (formula 202.9).

With \(d = 768\) (ViT-Base), joint attention costs \(4 \cdot 1568^2 \cdot 768 = 7.55\) GFLOPs per layer; divided costs \(4 \cdot 1568 \cdot 204 \cdot 768 = 0.98\) GFLOPs. The linear layers (QKV, output projection and MLP, about \(24\, d^2\) FLOPs per token) cost \(22.2\) GFLOPs per layer either way. So at 8 frames, joint attention is only 25% of the layer cost.

At 96 frames (the TimeSformer-L setting) the picture flips: 18,816 tokens, and joint attention costs 1,088 GFLOPs per layer, 80% of the layer. One FP16 attention matrix for one head would take \(18816^2 \cdot 2 = 0.71\) GB. Divided attention needs \(96 + 196 = 292\) keys per query.

The paper’s comparison on Kinetics-400 / Something-Something V2 (ViT-Base, 8 frames): [Certain]

AttentionParametersK400 top-1SSv2 top-1
Space only (no time mixing)85.9M76.936.6
Joint space–time85.9M77.458.5
Divided space–time121.4M78.059.5
Sparse local global121.4M75.956.3
Axial (time, then width, then height)156.8M73.556.2

Two lessons sit in this table. First, on Kinetics, “space only” is just 0.5 points behind joint attention: many Kinetics classes can be read from appearance. On SSv2, where motion direction decides the label, space only collapses to 36.6%. Always test a video model on a motion-heavy dataset. Second, divided attention is better and cheaper. Joint attention ran out of GPU memory at 448-pixel frames or at 32 frames; divided attention scaled to TimeSformer-HR (16 × 448 × 448) and TimeSformer-L (96 frames at 224, sampled at 1/4 of the frame rate). [Certain] Axial attention splits further (time, then height, then width) and is the cheapest per query, but here it was the least accurate.

Figure 4. The cost of attention for a ViT-Base video encoder (\(d = 768\)). Bars show query–key pairs per layer on a log scale (each grid line is 10× more). Class tokens are ignored, so the numbers are a little below the paper’s \(NF + 1\) and \(N + F + 2\).
How to read Figure 4

Start at 8 frames, 224 px, patch 16: 1,568 tokens and a joint/divided ratio of 7.7×. Slide to 96 frames: joint attention grows by \(12^2 = 144\times\), divided by much less, and the readout shows attention taking most of the layer cost. Choose 448 px: four times more patches per frame. Choose tubelet \(t = 2\): half the time steps, and every cost goes down. “Space only” is the cheapest but cannot see motion at all; axial is cheaper than divided but less accurate in the TimeSformer study.

The other ViViT designs, with measured cost

ViViT compared four ways to place time inside the transformer, all with ViViT-B and tubelets of 16×16×2: [Certain]

  • Model 1, joint space–time attention: all tokens in one sequence (as above).
  • Model 2, factorised encoder: a spatial transformer runs on each frame (or tubelet slice) and gives one vector per time step; a second, temporal transformer runs on those vectors. This is late fusion done well: time mixing happens only at the top.
  • Model 3, factorised self-attention: spatial then temporal attention inside every block, like TimeSformer with the order swapped (ViViT found the order makes no difference with its initialization). The temporal attention weights start at zero, so the block starts as the pretrained image block.
  • Model 4, factorised dot-product: half of the heads attend over space and half over time, with the same parameters as Model 1.
ViViT-B, 16×2 tubeletsK400 top-1Epic KitchensGFLOPsParamsRuntime (TPU-v3)
Model 1: joint80.043.1455.288.9M58.9 ms
Model 2: factorised encoder78.843.7284.4115.1M17.4 ms
Model 3: factorised self-attention77.439.1372.3117.3M31.7 ms
Model 4: factorised dot-product76.339.5277.188.9M22.9 ms

So in ViViT’s own setting, joint attention was the most accurate on Kinetics, and the factorised encoder was more than 3× faster with only 1.2 points less. The two papers disagree on which factorization wins (TimeSformer found divided attention better than joint). The safe conclusion: the order of splits, the data and the initialization all matter, and the winner depends on the budget. [Likely]

Later video transformers put back the bias that CNNs had. MViT (Fan et al., 2021) is a multiscale transformer: it starts with many tokens and few channels and pools tokens stage by stage, like a CNN pyramid. Video Swin (Liu et al., 2021) computes attention only inside local 3D windows that shift between layers. [Certain] Both trade global attention for locality and speed.

query token (click any cell to move it)keys of the current attention step (darker = larger weight \(\alpha\))keys of earlier stepsthe moving object
  1. input: tokens \(\mathbf{z}\) (\(F = 4\) frames \(\times\ N = 16\) patches, width \(d = 2\))
  2. for each attention step of the block (divided: time, then space):
  3.   \(\mathbf{q} \leftarrow W_q \mathbf{z}_{\text{query}}\);  \(\mathbf{k}_j \leftarrow W_k \mathbf{z}_j\), \(\mathbf{v}_j \leftarrow \mathbf{z}_j\) for the keys of this step
  4.   \(s_j \leftarrow \mathbf{q} \cdot \mathbf{k}_j / \sqrt{d}\)
  5.   \(\alpha \leftarrow \softmax(s)\)
  6.   \(\mathbf{z}_{\text{query}} \leftarrow \mathbf{z}_{\text{query}} + \sum_j \alpha_j \mathbf{v}_j\) (202.8); every token does the same, in parallel
  7. return \(\mathbf{z}\) the MLP of (202.8) is left out here
Figure 5. One attention block computed in your browser on a toy clip of \(F = 4\) frames with \(N = 16\) patches each (64 tokens, each a 2-number vector). A bright bar moves one column to the right per frame. Each attention step has its own small weight matrices \(W_q, W_k\); the values are the tokens themselves and there is no layer norm, so you can check every number by hand. The box shows the live values of the current step: the query vector, the scores, the softmax weights and the new token. Outlined cells are the tokens that can already influence the query’s output.
How to read Figure 5

Press Play on “Divided”. In the time step the query compares itself with the 4 tokens at the same position in the 4 frames, one by one; the softmax turns the 4 scores into weights, and the query becomes the weighted sum (plus itself, the residual). Every other token does the same at the same moment. In the space step the query compares itself with the 16 tokens of its own frame, row by row. Watch the outlines: each key of the space step brings in its whole time line, so after the 20 comparisons all 64 tokens can influence the output. The readout counts this live: 20 comparisons per query and \(64 \times 20 = 1{,}280\) for the layer, against \(64^2 = 4{,}096\) for joint attention, a ratio of \(64/20 = 3.2\) from formula (202.9). “Space only” stops at 16 tokens and never leaves the frame, which is why it fails on SSv2. “Axial” uses three hops of 4 keys (12 comparisons) and still reaches all 64 tokens. “Joint” reads all 64 keys in one step. Click any cell to choose another query; the animation restarts.

Algorithm 2: one divided space–time block (TimeSformer)
  1. input: tokens \(Z\) of shape \((B, F \cdot N, d)\), in frame-major order
  2. \(U \leftarrow \mathrm{LN}(Z)\) reshaped to \((B \cdot N, F, d)\) // one short sequence per patch position
  3. \(Z \leftarrow Z + \text{reshape back}\big(\mathrm{MHA}_{\text{time}}(U)\big)\) // first line of (202.8): \(F\) keys per query
  4. \(U \leftarrow \mathrm{LN}(Z)\) reshaped to \((B \cdot F, N, d)\) // one sequence per frame
  5. \(Z \leftarrow Z + \text{reshape back}\big(\mathrm{MHA}_{\text{space}}(U)\big)\) // second line: \(N\) keys per query
  6. \(Z \leftarrow Z + \mathrm{MLP}(\mathrm{LN}(Z))\)
  7. return \(Z\) // the class token (not shown) joins the spatial step of every frame

VideoMAE: learn from unlabeled video by hiding 90% of it

All the models above were trained with labels (Kinetics) on top of ImageNet weights. Labels for video are slow and expensive to make. Masked autoencoding (session 165) learns from raw data: hide most of the input, and train the model to rebuild it. Image MAE hides 75% of the patches. VideoMAE (Tong et al., 2022) brings this to video with two changes. [Certain]

1. An extremely high masking ratio: 90% to 95%. Video is redundant: neighboring frames are almost the same, so the information per token is much lower than in an image. With 75% masking the task is too easy. VideoMAE takes 16 frames (stride \(\tau = 4\) on Kinetics, 2 on SSv2), cuts them into \(2 \times 16 \times 16\) cubes (1,568 tokens), and hides 90%. [Certain]

2. Tube masking. If you mask patches at random, a hidden patch is often visible in the next frame at the same place. The model can then copy it instead of understanding anything. The paper calls this information leakage. Tube masking draws one random spatial mask and applies it to every frame, so a hidden position is hidden for the whole clip:

\[ M_{t,p} = m_p \ \text{ for all } t, \qquad m_p \in \{0, 1\}, \qquad \sum_{p=1}^{N} m_p = \lfloor \rho N \rfloor \tag{202.10} \]
Read the symbols
\(M_{t,p} = 1\)
the token at position \(p\) in time step \(t\) is hidden
\(\rho\)
the masking ratio, for example 0.9
\(N\)
patch positions per time step (196 at 224 px with 16-pixel patches)

Only the visible tokens go into the encoder, a plain ViT with joint space–time attention (joint attention is affordable because so few tokens remain). A small decoder gets the encoded visible tokens plus learned mask tokens at the hidden positions, and predicts the pixels of the hidden cubes. The loss is the mean squared error on the hidden cubes only, against pixel values normalized inside each cube:

\[ \mathcal{L}_{\text{MAE}} = \frac{1}{|\Omega|} \sum_{i \in \Omega} \big\lVert \hat{\mathbf{x}}_i - \mathrm{norm}(\mathbf{x}_i) \big\rVert_2^2 \tag{202.11} \]

where \(\Omega\) is the set of hidden tokens, \(\mathbf{x}_i\) the pixels of cube \(i\), and \(\hat{\mathbf{x}}_i\) the decoder’s prediction.

Worked example: how much the encoder actually sees

16 frames \(\to\) 8 time steps of \(14 \times 14 = 196\) positions: 1,568 tokens. The reference code hides \(\lfloor 0.9 \cdot 196 \rfloor = 176\) positions per time step, so \(20 \times 8 = 160\) tokens are visible: 10.2% of the clip. The encoder’s linear layers do about 10% of the work of a full pass, and its attention about \((160/1568)^2 = 1\%\). At 95% masking only 80 tokens are left.

The ablations on SSv2 (ViT-B, 800 epochs): tube masking at 75% gives 68.0%, tube at 90% gives 69.6%, random masking at 90% gives 68.3%, and frame masking (hide 14 of 16 frames) gives 61.5%. Training from scratch without pretraining gives 32.6%. [Certain]

hidden tokenvisible moving objecthidden but copyable from the same spot in another frame
Figure 6. Masking a toy clip: 8 time steps of 8×8 tokens (512 tokens). The background is static; a 2×2 object moves one token to the right per step. A hidden token is “copyable” when the same position is visible in another frame with the same content, so the model could fill it in without understanding anything.
How to read Figure 6

With tube masking, no hidden token is ever copyable, at any ratio: every hidden position is hidden in every frame. Switch to random masking at 90%: more than 40% of the hidden tokens are visible at the same spot in some other frame, so the reconstruction task leaks. Frame masking hides whole frames (7 of 8 at 90%), and almost every hidden static token can be copied from the one visible frame; only the moving object must be predicted. This matches the paper’s order: tube is best, then random, then frame.

Two results made VideoMAE important. First, it is data-efficient: it works on datasets of only 3k–4k videos, without extra data. Pretrained and fine-tuned only on HMDB-51 (3.5k training videos), it reached 62.6%. [Certain] Second, it is cheap to pretrain because the encoder sees so little: on SSv2 it trained 3.2× faster in wall-clock time than a MoCo v3 contrastive baseline (19.5 vs 61.7 hours on 64 V100 GPUs) and was more accurate (69.6% vs 54.2%). [Certain] VideoMAE V2 (Wang et al., 2023) added dual masking (the decoder also sees only a subset of tokens) and scaled the encoder to a billion-parameter ViT-g. [Certain] V-JEPA (Bardes et al., 2024) keeps the masking idea but predicts the hidden regions in feature space instead of pixels. [Certain]

Algorithm 3: one VideoMAE pretraining step
  1. input: a batch of clips \(V\) of shape \((B, 3, 16, 224, 224)\), masking ratio \(\rho = 0.9\)
  2. \(X \leftarrow \mathrm{TubeletEmbed}(V) + \text{pos}\) // \((B, 1568, d)\), formula (202.6)
  3. draw one spatial mask \(m\) per clip with \(\lfloor \rho \cdot 196 \rfloor\) hidden positions; repeat it over the 8 time steps // (202.10)
  4. \(H \leftarrow \mathrm{Encoder}(X[\text{visible}])\) // 160 tokens, joint attention
  5. \(Y \leftarrow \mathrm{Decoder}\big(\text{scatter}(H) \cup \{\text{mask token} + \text{pos at hidden places}\}\big)\)
  6. \(\mathcal{L} \leftarrow \mathrm{MSE}\big(Y[\text{hidden}],\ \mathrm{norm}(\text{cubes of } V)[\text{hidden}]\big)\) // (202.11)
  7. update encoder and decoder; after pretraining, keep only the encoder and fine-tune it with all tokens

Video–text encoders: features that speak the language of text

A classifier needs features that separate 400 actions. A video LLM or a retrieval system needs more: features that line up with any sentence. The tool is the same as CLIP for images (session 166): a video encoder and a text encoder trained so that matching (video, caption) pairs have a high cosine similarity and non-matching pairs a low one. With \(B\) pairs in a batch, normalized embeddings \(\mathbf{v}_i\) and \(\mathbf{t}_i\), and temperature \(\tau\):

\[ \mathcal{L}_{\text{con}} = -\frac{1}{2B} \sum_{i=1}^{B} \left[ \log \frac{e^{\mathbf{v}_i^\top \mathbf{t}_i / \tau}}{\sum_{j} e^{\mathbf{v}_i^\top \mathbf{t}_j / \tau}} + \log \frac{e^{\mathbf{t}_i^\top \mathbf{v}_i / \tau}}{\sum_{j} e^{\mathbf{t}_i^\top \mathbf{v}_j / \tau}} \right] \tag{202.12} \]

The first term asks each video to pick its own caption among the \(B\) captions; the second asks each caption to pick its video.

Worked example: one row of the contrastive loss

Take a batch of 3 and \(\tau = 0.1\). Video 1 has cosine similarities 0.62, 0.20, 0.15 with captions 1, 2, 3. The logits are 6.2, 2.0, 1.5, and the softmax gives 0.976 to its own caption. Its loss term is \(-\ln 0.976 = 0.024\). With a full similarity matrix of rows (0.62, 0.20, 0.15), (0.25, 0.55, 0.30), (0.10, 0.35, 0.48), the video-to-text loss is 0.135, the text-to-video loss 0.122, and their average 0.129.

The simple way to build a video–text encoder is to start from CLIP and pool frame features over time; CLIP4Clip did this for retrieval (session 208). Its weakness is the one from formula (202.1): pooled frame features are weak on motion. The newer recipe trains a true video encoder and aligns it with text. Two steps led there. UMT (Unmasked Teacher, Li et al., 2023) masks most video tokens, like VideoMAE, but instead of rebuilding pixels it trains the visible tokens to match the features of a CLIP image teacher: semantic targets, fast convergence, and features that are friendly to text. [Certain] InternVideo2 scales this idea up.

InternVideo2: a 6B video encoder in three stages

InternVideo2 (Wang et al., 2024) trains a ViT video encoder with 6B parameters (48 layers, width 3,200; 8 frames at 224 px with 14×14 patches, so \(8 \times 256 = 2{,}048\) tokens, and an attention-pooling head that outputs a 768-dimensional clip vector). [Certain] Training has three stages, each adding something the previous one lacks:

  1. Unmasked token distillation. Hide 80% of the tokens. Feed the full clip to two frozen teachers: InternViT-6B (the vision encoder of the InternVL multimodal model, for semantics) and VideoMAE V2-g (for motion). Train the student so that its visible tokens match the teachers’ tokens at the same positions, through small projection heads that are dropped afterwards:
    \[ \mathcal{L}_{1} = \frac{1}{Z} \sum_{p\, \in\, \text{visible}} \Big( \alpha_1 \big\lVert f(V)_p - h(V)_p \big\rVert^2 + \alpha_2 \big\lVert f(V)_p - g(V)_p \big\rVert^2 \Big) \tag{202.13} \]
    Here \(f\) is the student, \(h\) the InternViT teacher, \(g\) the VideoMAE V2 teacher, and \(Z\) a normalizer. In practice they align several of the last layers (the last 6 of InternViT, the last 4 of VideoMAE V2, and InternViT’s final output token). [Certain]
  2. Video–audio–speech–text alignment. Add an audio encoder (a 12-layer transformer initialized from BEATs, 90M parameters) and a text encoder from BERT-Large. Train with contrastive loss (202.12), a video–text matching loss and a masked language loss, on image–text, video–text and video–audio–speech–text pairs whose captions were fused from what is seen, heard and said. [Certain]
  3. Next-token prediction. Connect the encoder to an LLM and train on video instruction and question–answer data, so the encoder is also updated by the LLM’s loss. [Certain]

The training data has 402M entries: 2M videos for stage 1, 50M video–text pairs, 50M video–audio–speech–text pairs, and 300M image–text pairs. The paper reports strong results on more than 60 video and audio tasks. [Certain] The point for design: each stage gives one property. Distillation from a motion teacher gives temporal sense; contrastive alignment gives text-ready features for retrieval and zero-shot classification; next-token training makes the features useful to an LLM (session 205).

Figure 7. The three stages of InternVideo2. Stage 1 distills two teachers into the video encoder on visible tokens only. Stage 2 adds audio and text encoders and aligns everything through text. Stage 3 connects the encoder to an LLM and trains with next-token prediction.

Choosing a video encoder

You needGood choiceWhy
Fast action classification on a GPU budgetSlowFast or X3D (3D CNN), or a small fine-tuned VideoMAE ViT-BMature, cheap per clip, strong on motion; easy to fine-tune with a small labeled set
Fine-grained motion (direction, sports techniques)Video transformer pretrained with VideoMAE, fine-tuned; or a temporal-aware encoder such as InternVideo2Space-only or CLIP-pooled features fail on motion (SSv2: 36.6% for space-only TimeSformer)
Text-to-video search, zero-shot labelsA video–text encoder (InternVideo2 stage 2, UMT, or CLIP with temporal pooling as a baseline)The embedding must live in the same space as text, formula (202.12)
Input to a video LLMA video–text encoder or an image encoder applied per frame plus token compressionThe LLM needs text-aligned tokens; the token budget per second decides cost (session 206)
Long videos (minutes to hours)Divided or windowed attention, short clips encoded separately, then a temporal model on topJoint attention grows with the square of the number of frames (formula 202.7)
Pretraining on your own unlabeled footageVideoMAE-style masked pretrainingWorks with a few thousand videos, no labels

Putting it together in PyTorch

The script below builds the four key pieces of this session at toy sizes: an I3D inflation with the boring-video check, a tubelet embedding, a divided space–time block, tube masking, and the symmetric video–text contrastive loss. It runs on a CPU in about a second and prints the same numbers every time.

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# 1) I3D inflation: a 2D conv becomes a 3D conv, formula (202.4)
def inflate(conv2d, kt):
    c3 = nn.Conv3d(conv2d.in_channels, conv2d.out_channels,
                   kernel_size=(kt, *conv2d.kernel_size),
                   padding=(kt // 2, *conv2d.padding), bias=conv2d.bias is not None)
    with torch.no_grad():
        w = conv2d.weight.unsqueeze(2).repeat(1, 1, kt, 1, 1) / kt   # repeat N times, divide by N
        c3.weight.copy_(w)
        if conv2d.bias is not None: c3.bias.copy_(conv2d.bias)
    return c3

conv2d = nn.Conv2d(3, 8, 3, padding=1)
conv3d = inflate(conv2d, kt=3)
img = torch.randn(1, 3, 32, 32)
boring = img.unsqueeze(2).repeat(1, 1, 5, 1, 1)                     # (B, C, T, H, W): same frame 5 times
y2, y3 = conv2d(img), conv3d(boring)
print("boring-video check:", (y3[:, :, 2] - y2).abs().max().item())  # middle frame: about 1e-6

# 2) Tubelet embedding (ViViT / VideoMAE): a Conv3d with kernel = stride = (2, 16, 16)
d = 96
embed = nn.Conv3d(3, d, kernel_size=(2, 16, 16), stride=(2, 16, 16))
video = torch.randn(1, 3, 16, 224, 224)
tok = embed(video)                                                  # (1, d, 8, 14, 14)
B, D, Tp, Hp, Wp = tok.shape
x = tok.flatten(2).transpose(1, 2)                                  # (1, 1568, d), formula (202.6)
print("tokens:", x.shape)

# 3) Divided space-time attention block (TimeSformer), formula (202.8)
class DividedBlock(nn.Module):
    def __init__(self, d, heads):
        super().__init__()
        self.n1, self.n2, self.n3 = nn.LayerNorm(d), nn.LayerNorm(d), nn.LayerNorm(d)
        self.t_attn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.s_attn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
    def forward(self, x, T, N):                                     # x: (B, T*N, d), frame-major
        B, _, d = x.shape
        h = self.n1(x).reshape(B, T, N, d).transpose(1, 2).reshape(B * N, T, d)
        h = self.t_attn(h, h, h, need_weights=False)[0]              # attend over T frames, same patch
        x = x + h.reshape(B, N, T, d).transpose(1, 2).reshape(B, T * N, d)
        h = self.n2(x).reshape(B * T, N, d)
        h = self.s_attn(h, h, h, need_weights=False)[0]              # attend over N patches, same frame
        x = x + h.reshape(B, T * N, d)
        return x + self.mlp(self.n3(x))

blk = DividedBlock(d, heads=4)
out = blk(x, Tp, Hp * Wp)
print("divided block out:", out.shape)
N = Hp * Wp
print("keys per query: joint", Tp * N, " divided", Tp + N)

# 4) Tube masking (VideoMAE), formula (202.10): one spatial mask, shared by all frames
def tube_mask(T, N, ratio, g):
    m = int(ratio * N)                                              # masked positions per frame
    keep = torch.randperm(N, generator=g)[: N - m]                  # visible positions
    mask = torch.ones(T, N, dtype=torch.bool); mask[:, keep] = False
    return mask.flatten()                                           # True = masked
g = torch.Generator().manual_seed(0)
mask = tube_mask(Tp, N, 0.9, g)
visible = x[:, ~mask]                                               # only these go to the encoder
print("visible tokens:", visible.shape[1], "of", mask.numel())

# 5) Symmetric video-text contrastive loss, formula (202.12)
v = F.normalize(torch.randn(4, 64), dim=-1); t = F.normalize(v + 0.3 * torch.randn(4, 64), dim=-1)
logits = v @ t.T / 0.07
labels = torch.arange(4)
loss = (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) / 2
print("contrastive loss:", round(loss.item(), 4))

Output (PyTorch 2.8, CPU):

boring-video check: 1.430511474609375e-06
tokens: torch.Size([1, 1568, 96])
divided block out: torch.Size([1, 1568, 96])
keys per query: joint 1568  divided 204
visible tokens: 160 of 1568
contrastive loss: 0.0315

The boring-video check compares the middle frame. The first and last frames differ from the 2D output, because zero padding in time puts empty frames next to them: inflation is exact only where the filter sees real frames.

Where this is used

ApplicationEncoder roleTypical choice
Action recognition (Kinetics, sports moves)Clip vector → classifier; average over several clips for a whole videoSlowFast, VideoMAE fine-tuned, InternVideo2 (session 203)
Text-to-video retrievalClip vector in a shared space with text, searched with a vector indexVideo–text encoder, CLIP4Clip as a baseline (session 208)
Key-moment detection and temporal groundingA feature per short clip along the timeline; a head predicts start and endFrozen clip features (I3D and SlowFast features were the standard; now video–text features) (session 207)
Video fingerprinting and near-duplicatesCompact per-segment vectors that survive re-encoding and cropsFrame-level image features plus temporal alignment; video encoders when motion matters (session 209)
Video segmentation and trackingDense per-patch features over timeImage backbones with memory (SAM 2) more than clip encoders (session 204)
Video LLMs and Video-RAGVisual tokens for the LLM, or clip embeddings for retrievalVideo–text encoder or per-frame image encoder with token merging (session 205, session 210)

Common mistakes

  • Judging a video model only on Kinetics. Appearance solves much of Kinetics; space-only attention is 0.5 points behind joint there but 22 points behind on SSv2. Test on a motion-heavy set.
  • Inflating without dividing by N. The responses become N times too large and the pretrained batch-norm statistics no longer fit. Use repeat ÷ N or central frame initialization.
  • Mixing up frame-major and patch-major order when reshaping. In a divided block, the reshape to \((B \cdot N, F, d)\) must group tokens of the same position. A wrong transpose still runs, and silently attends over the wrong tokens.
  • Random masking in masked video pretraining. Hidden patches leak from neighboring frames, and the model learns to copy. Use tube masking (or motion-aware masks) and a high ratio.
  • Forgetting the sampling stride. A model trained on 16 frames with stride 4 (2.1 s at 30 fps) sees a different speed of motion if you feed stride 1 at test time. Keep the training sampling, or train with several strides.
  • Using pooled CLIP frame features for motion questions. They are excellent for “what is in the video” and weak for “which way did it move”.
  • Counting only parameters. TimeSformer has 121.4M parameters but its paper reports 0.59 TFLOPs per inference against 1.97 for SlowFast 8×8 R50 with 34.6M parameters. [Certain] Compare FLOPs, tokens and latency at your clip length.

Interview questions

Why can a per-frame image model with average pooling not tell “opening a door” from “closing a door”?basic
Average pooling (formula 202.1) is invariant to the order of frames: reversing or shuffling the clip gives the same sum, so the output is identical. Any model that fuses frames only through a symmetric pooling at the end cannot use order. Fixes: optical flow as an extra input (two-stream), 3D convolutions, temporal attention, or a temporal model (transformer or RNN) on top of the frame features.
How does I3D initialize 3D filters from ImageNet weights, and why does it work?core
Each \(k \times k\) filter is repeated \(N\) times along time and divided by \(N\). On a “boring video” that repeats one image, every time slice gives the same 2D response, and the average equals it, so every conv layer output equals the 2D output; ReLU and pooling keep this, and the whole network reproduces the ImageNet prediction (the boring-video fixed point). Training on video then teaches the time slices to differ. Without the ÷N the responses are N times too large and break the following layers’ statistics.
Explain SlowFast. Why is the Fast pathway thin?core
Two pathways read the same clip: Slow at a low frame rate (4 frames, stride 16) with many channels, for appearance; Fast at \(\alpha = 8\) times the frame rate (32 frames) with \(\beta = 1/8\) of the channels, for motion. Lateral connections fuse Fast into Slow (default: time-strided convolution, concatenation). Motion needs fine time resolution but not many semantic channels, so Fast can be thin: its cost scales roughly with \(\alpha\beta^2 = 1/8\) of Slow, and measured it is about 20% of the total (6.4 of 33.7 GFLOPs for R-50 4×16). Fast alone is weak (51.7%), but it adds 3.0 points to Slow (72.6 → 75.6 on K400).
What is divided space–time attention and how much does it save?core
In each block, temporal attention first (each patch attends to the same position in all \(F\) frames), then spatial attention (each patch attends to the \(N\) patches of its frame), with separate weights, then the MLP. Keys per query drop from \(NF\) to \(N + F\), so attention cost drops by \(NF/(N+F)\): 7.7× for 8 frames of 196 patches, about 64× for 96 frames. Reach is not lost: two hops connect any pair of tokens. In TimeSformer it was also more accurate than joint attention (78.0 vs 77.4 K400, 59.5 vs 58.5 SSv2), and joint attention ran out of memory at 448 px or 32 frames.
Tubelet embedding vs uniform frame sampling: what is the difference?basic
Uniform frame sampling patchifies each frame like ViT and leaves all time mixing to the transformer. Tubelet embedding projects \(t \times h \times w\) blocks (a 3D convolution with stride equal to the kernel), so each token already contains \(t\) frames of motion and the token count drops by \(t\). 16 frames with \(2 \times 16 \times 16\) tubelets give 1,568 tokens instead of 3,136. ViViT found central frame initialization of the tubelet filter best (79.2% vs 77.6% for inflation).
Why does VideoMAE use 90% masking when image MAE uses 75%, and why tube masking?core
Video is temporally redundant: neighboring frames carry almost the same content, so each token holds less information and 75% masking leaves the task too easy. With random masking, a hidden patch is often visible at the same place in a nearby frame and can be copied (information leakage). Tube masking uses the same spatial mask in all frames, so hidden positions are hidden for the whole clip and must be inferred from context and motion. On SSv2: tube 90% 69.6%, random 90% 68.3%, tube 75% 68.0%, frame masking 61.5%.
How much cheaper is VideoMAE pretraining than a full forward pass, and where does the saving come from?advanced
The encoder sees only the visible tokens: 160 of 1,568 at 90% (the code masks \(\lfloor 0.9 \cdot 196 \rfloor = 176\) positions per time step). Linear layers scale with the token count (about 10% of full cost), attention with its square (about 1%). The decoder is small and sees all tokens, so it becomes a large part of what remains; VideoMAE V2’s dual masking also masks decoder tokens to cut this. In wall-clock terms the paper reports 3.2× faster pretraining than MoCo v3 on SSv2.
Why are VideoMAE features not enough for a video LLM or retrieval, and how does InternVideo2 fix it?advanced
Pixel reconstruction learns low-level structure; frozen features are weak (38.9% linear probe vs 69.6% fine-tuned on SSv2) and not aligned with language. InternVideo2 first distills two teachers on unmasked tokens (80% masked): InternViT-6B for semantics, VideoMAE V2-g for motion. Then it aligns video, audio, speech and text with contrastive, matching and masked-language losses, so embeddings live in a text space. Finally it trains with an LLM by next-token prediction, so the features suit open-ended answering.
You must build text-to-video search over a sports archive. Which encoder, and what are the trade-offs?advanced
Use a video–text encoder (InternVideo2 stage-2 style or similar), encode short clips (for example 8 frames over 2–4 s) into normalized vectors, and index them with approximate nearest-neighbor search. CLIP with mean-pooled frames is a strong cheap baseline for “what is shown” queries but weak for motion queries (“backhand slice”, “goal from a corner”). Trade-offs: a 6B encoder is expensive per clip, so use it offline; clip length and stride decide what motion is visible; and domain fine-tuning with contrastive loss on in-domain captions often matters more than encoder size. [Likely]
Joint attention on 96 frames: estimate the cost and what you would do instead.advanced
At 224 px and patch 16 that is \(96 \cdot 196 = 18{,}816\) tokens. Joint attention per layer: \(4 \cdot 18816^2 \cdot 768 \approx 1{,}088\) GFLOPs, about 80% of the layer, and one FP16 attention matrix per head is 0.71 GB (FlashAttention avoids storing it but not the FLOPs). Options: divided attention (292 keys per query), tubelets of \(t = 2\) (half the tokens), local windows (Video Swin), token pooling (MViT), or encode short clips separately and model time on top.
SlowFast uses temporal filters only in res4 and res5 of the Slow pathway. Why?core
The Slow pathway samples one frame every 16. Between two such frames, objects can move far. In early layers the spatial receptive field is small, so a temporal filter sees unrelated content in neighboring frames and only adds noise; the paper measured lower accuracy. In later layers the receptive field is large enough to connect the same object across frames. The Fast pathway, with frames close in time, uses temporal filters everywhere.

Exercises

  1. A clip has 32 frames at 224×224. How many tokens does a ViT with \(16 \times 16\) patches produce with uniform frame sampling? With tubelets of \(2 \times 16 \times 16\)? With \(4 \times 16 \times 16\)?
  2. For the 16-tubelet case of exercise 1 (\(t = 2\)), compute the joint/divided attention ratio of formula (202.9).
  3. Compute the parameters of a \(3 \times 3 \times 3\) convolution with 128 input and 256 output channels, and of its \(1 \times 3 \times 3\) + \(3 \times 1 \times 1\) factorization with a middle width of 256.
  4. A 2D filter has weights summing to 2.0. After I3D inflation with \(N = 5\), what is the sum of the weights in each time slice and in total?
  5. SlowFast with \(\alpha = 4\), \(\beta = 1/4\): estimate the Fast/Slow compute ratio. How many frames does each pathway read from a 64-frame clip with \(\tau = 16\)?
  6. VideoMAE with masking ratio 0.95 on 16 frames at 224 px: how many tokens does the encoder see? (Use the per-frame rounding of the reference code.)
  7. In the toy clip of Figure 6 (8 frames), with random masking at ratio 0.9, estimate the chance that a hidden static token is visible at the same place in at least one of the other 7 frames.
Answerscheck
1. \(32 \cdot 196 = 6{,}272\); \(16 \cdot 196 = 3{,}136\); \(8 \cdot 196 = 1{,}568\). 2. \(NF/(N + F) = 3136 / (196 + 16) = 14.8\times\). 3. Full: \(27 \cdot 128 \cdot 256 = 884{,}736\). Factorized: \(9 \cdot 128 \cdot 256 + 3 \cdot 256 \cdot 256 = 294{,}912 + 196{,}608 = 491{,}520\) (56% of full). 4. Each slice is the 2D filter divided by 5, so each sums to 0.4, and the total is 2.0 again: the response on a boring video is unchanged. 5. \(\alpha\beta^2 = 4/16 = 0.25\). Slow reads \(64/16 = 4\) frames; Fast reads \(4 \cdot 4 = 16\) frames (one every 4). 6. \(\lfloor 0.95 \cdot 196 \rfloor = 186\) hidden per step, 10 visible, times 8 steps = 80 tokens. 7. Each other frame hides the same spot with probability about 0.9, so the chance at least one shows it is \(1 - 0.9^7 \approx 0.52\). The figure shows a bit less (43% for its first seeded mask), because tokens touched by the moving object cannot be copied and the sample is small.

Key takeaways

  • Averaging per-frame features (202.1) ignores order; video encoders add time with 3D filters (202.2), temporal attention or optical flow.
  • I3D inflates 2D filters by repeating them and dividing by \(N\) (202.4), so ImageNet weights give the same output on a still video; SlowFast reads appearance slowly with many channels and motion fast with few, at about \(\alpha\beta^2\) extra cost (202.5).
  • Video transformers turn clips into \(n = \lfloor T/t \rfloor \lfloor H/h \rfloor \lfloor W/w \rfloor\) tokens (202.6). Joint attention costs \((NF)^2\) pairs (202.7); divided attention (202.8) costs \(NF(N + F)\), 7.7× less at 8 frames (202.9), without losing reach.
  • VideoMAE pretrains on unlabeled video by hiding 90% of the tokens with a tube mask (202.10) and rebuilding normalized pixels (202.11): data-efficient and cheap, but features need fine-tuning.
  • Video–text encoders add contrastive alignment (202.12); InternVideo2 distills semantic and motion teachers (202.13), aligns with audio, speech and text, then trains with an LLM.

Key formulas

\[ W^{3D}(\tau) = \tfrac{1}{N} W^{2D}, \quad \frac{\text{FLOPs}_{\text{Fast}}}{\text{FLOPs}_{\text{Slow}}} \approx \alpha\beta^2, \quad n = \Big\lfloor \tfrac{T}{t} \Big\rfloor \Big\lfloor \tfrac{H}{h} \Big\rfloor \Big\lfloor \tfrac{W}{w} \Big\rfloor \]
\[ \frac{\text{pairs}_{\text{joint}}}{\text{pairs}_{\text{divided}}} = \frac{NF}{N + F}, \quad M_{t,p} = m_p, \quad \mathcal{L}_{\text{MAE}} = \frac{1}{|\Omega|}\sum_{i \in \Omega} \lVert \hat{\mathbf{x}}_i - \mathrm{norm}(\mathbf{x}_i) \rVert^2 \]

References for this session

  1. Carreira & Zisserman (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
  2. Feichtenhofer et al. (2018). SlowFast Networks for Video Recognition
  3. Arnab et al. (2021). ViViT: A Video Vision Transformer
  4. Bertasius et al. (2021). Is Space-Time Attention All You Need for Video Understanding?
  5. Tong et al. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
  6. Wang et al. (2024). InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
  7. Kay et al. (2017). The Kinetics Human Action Video Dataset
  8. Simonyan & Zisserman (2014). Two-Stream Convolutional Networks for Action Recognition in Videos
  9. Tran et al. (2015). Learning Spatiotemporal Features with 3D Convolutional Networks
  10. Tran et al. (2018). A Closer Look at Spatiotemporal Convolutions for Action Recognition
  11. Feichtenhofer (2020). X3D: Expanding Architectures for Efficient Video Recognition
  12. Fan et al. (2021). Multiscale Vision Transformers
  13. Liu et al. (2021). Video Swin Transformer
  14. Wang et al. (2023). VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
  15. Li et al. (2023). Unmasked Teacher: Towards Training-Efficient Video Foundation Models
  16. Bardes et al. (2024). Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA)
  17. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  18. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  19. He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
  20. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision