LLM & VLM courseBlog
203Track 8: Video and multimodal systems · session 3 of 21

Action recognition

Action recognition answers the question “what is the person doing?” for a clip, and “what happens, from when to when?” for a long video. This session covers clip-level and video-level classification on datasets like Kinetics, temporal action localization with ActionFormer, zero-shot recognition with video–text models, and why short, fast sports actions are still hard.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions on CLIP (166) and video encoders (202); basic PyTorch

What you will learn

  • The difference between a clip-level and a video-level prediction, how clips are sampled (T frames, stride τ), and how multi-view testing averages them
  • The main datasets (Kinetics, UCF-101, HMDB-51, Something-Something) and why some can be solved from one frame while others need motion
  • Temporal action localization: the output (start, end, class, score), temporal IoU, and mean average precision at several tIoU thresholds
  • How ActionFormer works: a feature pyramid with local self-attention, one score and two boundary distances per moment, and NMS
  • Zero-shot action recognition with CLIP-style models: class names as prompts, pooled frame embeddings, temperature, and what mean pooling cannot see
  • Why sports actions are short, fast and fine-grained, what this means for frame rate and sampling, and how action spotting is scored

Why this matters

Most videos are about people doing things: running, cooking, shooting a basket, falling. Action recognition is the task of naming what is being done. It is the oldest video task in computer vision and still the base of many products: sports analytics, video search, content moderation, fitness apps and safety cameras.

There are three versions of the question, and this session covers each one:

  1. Classification: “This 10-second clip shows high jump.” The input is short and already cut around one action (a trimmed clip).
  2. Temporal action localization: “In this 2-hour match, there are dunks at 12:04–12:07 and 51:30–51:33.” The input is long and untrimmed; the output is a list of (start, end, class, score).
  3. Zero-shot recognition: “Is this a pole vault?”, for a class the model never saw labeled. Video–text models such as CLIP make this possible.

At the end we look at sports. Sports actions are short, fast and look alike (a forward salto and a backward salto share almost every frame), and many lessons of the earlier sections break there.

This session builds on the encoders of session 202 (3D CNNs, SlowFast, video transformers) and on CLIP from session 166. Here we care less about the inside of the encoder and more about the task: what goes in, what comes out, how it is trained and how it is measured.

Clip-level and video-level classification

A video model cannot read all frames at once. A 10-second clip at 30 frames per second (fps) has 300 frames; at 224×224 pixels that is far too much for one forward pass. So the model reads a clip: a small set of \(T\) frames, taken every \(\tau\) frames (the stride):

\[ t_i = t_0 + i\,\tau, \quad i = 0, 1, \dots, T - 1, \qquad \text{span} = \frac{T\,\tau}{f} \ \text{seconds} \tag{203.1} \]
Read the symbols
\(t_0\)
the first frame of the clip
\(T\)
how many frames the model sees (often 8, 16 or 32)
\(\tau\)
the stride: we keep one frame every \(\tau\) frames
\(f\)
the frame rate of the video, in frames per second

Papers write this as “\(T \times \tau\)”. An 8×8 clip at 30 fps covers \(64/30 = 2.13\) seconds. So one clip sees only about a fifth of a 10-second video.

That gives two levels of prediction:

  • Clip-level: the model’s output for one clip, \(\softmax(f_\theta(X_k))\), where \(X_k\) is the \(k\)-th clip.
  • Video-level: one answer for the whole video, made by combining \(K\) clips. The simplest way is to average the clip probabilities:
\[ p(y \mid V) = \frac{1}{K} \sum_{k=1}^{K} \softmax\!\big(f_\theta(X_k)\big)_y, \qquad \hat{y} = \argmax_y \, p(y \mid V) \tag{203.2} \]
Read the symbols
\(V\)
the whole video
\(X_k\)
clip number \(k\), a tensor of shape \(3 \times T \times H \times W\)
\(f_\theta\)
the video network; it returns one score (logit) per class
\(K\)
how many clips we take from the video

This is called multi-view testing. A common protocol on Kinetics is 10 clips spread in time and 3 spatial crops, \(10 \times 3 = 30\) views per video. [Likely] Reported “video-level top-1 accuracy” almost always means this protocol; a single-clip number is lower and is the fairer measure of speed.

Figure 1 shows the problem on a 10-second high-jump video. Only 1.5 seconds (from 6.0 s to 7.5 s) show the jump itself; the rest is the run-up, which looks like plain running. The toy clip model gives a high score to high jump only for the part of the jump it sees.

high jumplong jumprunning
Figure 1. Clip sampling on a 10-second video at 30 fps. The orange band is the jump (6.0–7.5 s). Each gray box is one clip; the bar at the right of the same row shows that clip’s probabilities for the three classes. The bottom bar is the video-level result. Clip logits come from a toy model: if the clip sees a fraction \(c\) of the jump, the logits are \(0.5 + 3.5c\) (high jump), \(0.5 + 2c\) (long jump) and \(2 - c\) (running).
How to read Figure 1

Start with the default: one 8×8 clip in the middle of the video. It covers 3.9–6.1 s and sees only a sliver of the jump, so the video is called running (0.655). Now raise \(K\) to 5: one clip lands on the jump and says high jump with 0.786, but averaging with four run-up clips still gives running. Switch to “Max over clips”: now the jump wins. Or keep one clip and make it longer (\(T = 32\)): it sees the whole jump. The lesson: averaging works when the action fills the video (as in most Kinetics clips) but dilutes short actions in long videos.

Worked example: the numbers in Figure 1

One 8×8 clip spans \(8 \cdot 8 / 30 = 2.13\) s. Centered at 5 s, it covers 3.93–6.07 s, so it sees 0.07 s of the 1.5-s jump: \(c = 0.044\). Logits: high jump \(0.5 + 3.5 \cdot 0.044 = 0.66\), long jump \(0.59\), running \(1.96\). After softmax: 0.178, 0.167, 0.655. Wrong answer.

With \(K = 5\), the clip centers are at 1, 3, 5, 7 and 9 s. The clip at 7 s covers 5.93–8.07 s and sees the full jump (\(c = 1\)): probabilities 0.786, 0.175, 0.039. The average over all five clips is 0.285 (high jump) against 0.554 (running): still wrong. The maximum over clips is 0.786 against 0.691: right.

Training: the clip inherits the video label

In training we usually sample one random clip per video per step and give it the video’s label:

\[ \mathcal{L}(\theta) = -\log \softmax\!\big(f_\theta(X)\big)_{y}, \qquad X \sim \text{random clip of } V \tag{203.3} \]

This is ordinary cross-entropy. It is cheap, and random clip positions act like data augmentation. It also assumes that every clip shows the action. That is roughly true for trimmed datasets and false for long videos, which is one reason localization (next sections) is a separate task.

A second way to sample, from Temporal Segment Networks (TSN), is sparse segment sampling: split the whole video into \(T\) equal segments and take one random frame from each. The clip then covers the entire video with only \(T\) frames, at the cost of large gaps between frames. Dense \(T \times \tau\) sampling sees motion well in a short window; segment sampling sees the whole story coarsely.

Datasets, and the single-frame trap

The Kinetics datasets from DeepMind are the standard for training and measuring video classifiers. Kinetics-400 has 400 human action classes, at least 400 clips per class, each about 10 seconds long and each from a different YouTube video. [Certain] Later versions grew to 600 and 700 classes.

DatasetSizeWhat it tests
Kinetics-400 / 600 / 700400, 600, 700 classes; ~10-s YouTube clips [Certain]Broad human actions; the main pretraining and benchmark set
UCF-101101 classes, over 13k clips, 27 hours [Certain]Older and small; now used for transfer and zero-shot tests
HMDB-5151 classes, about 7k clips [Likely]Same use as UCF-101
Something-Something v2174 classes written as templates, such as “pushing something from left to right” [Certain]Motion and direction; the object does not tell you the class
Charades, AVAMulti-label home videos; per-person actions in movies [Likely]Several actions at once; who does what

A known weakness of Kinetics-style data is scene bias: many classes can be guessed from one frame. A swimming pool means swimming; a guitar means playing guitar. A model can score well without understanding motion at all. [Likely] Something-Something was built to remove this shortcut: “pushing something left” and “pushing something right” have the same objects and the same scene, so only the direction of motion separates them. When you compare video models, always check both kinds of benchmarks. A model that wins on Kinetics but not on Something-Something may be a good image model in disguise.

The models, briefly

The encoders were the topic of session 202. For action recognition, three families matter:

  • 2D CNN + temporal pooling (TSN): run an image network on each frame and average. Fast, strong on scene-biased data, blind to order.
  • 3D CNNs (I3D, SlowFast): convolutions that also slide over time. SlowFast uses two pathways. The Slow pathway reads few frames with many channels (what is there). The Fast pathway reads \(\alpha\) times more frames with a fraction \(\beta\) of the channels (how it moves):
\[ T_{\text{fast}} = \alpha \, T_{\text{slow}}, \qquad C_{\text{fast}} = \beta \, C_{\text{slow}}, \qquad \alpha = 8,\ \beta = \tfrac18 \tag{203.4} \]

The Fast pathway has no temporal downsampling and takes about 20% of the total compute. [Certain] In the paper’s Kinetics-400 ablation, a 4×16 Slow-only network reached 72.6% top-1 and the matching SlowFast network 75.6%; the best SlowFast model reached 79.8% without ImageNet pretraining. [Certain]

  • Video transformers (TimeSformer, ViViT, VideoMAE): patch tokens over space and time with attention. With large-scale pretraining they are the strongest classifiers today; their cost grows quickly with \(T\).
Algorithm 1: train a clip classifier and test it at video level
  1. input: labeled videos \((V, y)\), clip shape \(T \times \tau\), number of test views \(K\)
  2. for each training step do
  3.   pick a batch of videos; for each, choose a random start \(t_0\) and take frames \(t_0 + i\tau\) // formula (203.1)
  4.   random resized crop, horizontal flip (careful: flip changes “left” and “right” classes)
  5.   loss \(\leftarrow -\log \softmax(f_\theta(X))_y\); AdamW step // formula (203.3)
  6. at test time, for each video do
  7.   take \(K\) clips with evenly spaced starts (and optionally 3 spatial crops each)
  8.   \(p(y \mid V) \leftarrow\) mean of the clip softmax outputs // formula (203.2)
  9.   report top-1 (\(\hat{y} = y\)) and top-5 (\(y\) among the 5 largest)

Temporal action localization: start, end and class

Real videos are untrimmed. A basketball broadcast is two hours long; the dunks fill maybe one minute of it. Temporal action localization (TAL), also called temporal action detection, asks for every action instance in the video:

\((s_j, e_j, c_j, \text{score}_j)\) = (start time, end time, class, confidence)

It is object detection, but in one dimension (time) instead of two. Many ideas move over directly: anchors or anchor-free heads, a feature pyramid, IoU, non-maximum suppression and mean average precision.

Temporal IoU

To say whether a predicted segment \(\hat{S} = [\hat{s}, \hat{e}]\) matches a true segment \(S = [s, e]\), we use temporal intersection over union:

\[ \operatorname{tIoU}(\hat{S}, S) = \frac{\max\!\big(0,\ \min(\hat{e}, e) - \max(\hat{s}, s)\big)}{\max(\hat{e}, e) - \min(\hat{s}, s)} \tag{203.5} \]

The top is the length of the overlap; the bottom is the length of the union (for overlapping segments). tIoU is 1 for a perfect match and 0 for no overlap. Example: the true dunk is at [8, 14] s and the model says [9, 13] s. Overlap 4 s, union 6 s, \(\operatorname{tIoU} = 4/6 = 0.667\).

Mean average precision at several thresholds

For one class and one threshold \(\theta\): sort all predictions by score. Walk down the list. A prediction is a true positive if it has tIoU \(\ge \theta\) with a true segment that has not been matched yet; otherwise it is a false positive. A second prediction on the same true segment is a false positive: duplicates are punished. From the running counts we get a precision–recall curve, and the area under it is the average precision \(\text{AP}_c(\theta)\). Then:

\[ \text{mAP}@\theta = \frac{1}{C}\sum_{c=1}^{C} \text{AP}_c(\theta), \qquad \text{avg-mAP} = \frac{1}{|\Theta|}\sum_{\theta \in \Theta} \text{mAP}@\theta \tag{203.6} \]
Read the symbols
\(C\)
the number of action classes
\(\theta\)
the tIoU threshold for a match
\(\Theta\)
the set of thresholds: \(\{0.3, 0.4, \dots, 0.7\}\) on THUMOS14, \(\{0.5, 0.55, \dots, 0.95\}\) on ActivityNet-1.3 [Certain]

Averaging over thresholds rewards models whose boundaries are precise, not only roughly right. Figure 2 lets you move \(\theta\) and watch predictions turn from hits into misses.

ground truthtrue positivefalse positive
Figure 2. One class (“dunk”) in a 60-second video: three true segments (top row) and six predictions sorted by score. Each prediction shows its best tIoU. The readout gives the precision after each prediction and the AP at the chosen threshold.
How to read Figure 2

At \(\theta = 0.5\) the top three predictions each match a different true segment, so \(\text{AP} = 1.0\), even though three false positives follow (they come after all true segments are found, so they do not lower the area). Notice the 0.60 prediction: tIoU 0.667 is good, but its true segment was already taken by the 0.95 prediction, so it is a duplicate and counts as false. Raise \(\theta\) to 0.55: the 0.90 prediction (tIoU 0.545) and the 0.80 one (exactly 0.5) fail and AP drops to 0.444. At \(\theta = 0.9\) nothing matches.

Worked example: AP at \(\theta = 0.6\) in Figure 2

tIoU of each prediction with its best true segment: 0.846, 0.545, 0.5, 0.667 (duplicate), 0, 0.733. At \(\theta = 0.6\) the hit pattern in score order is TP, FP, FP, FP, FP, TP. Precision after each: 1, 0.5, 0.333, 0.25, 0.2, 0.333. Recall reaches 1/3 at the first prediction and 2/3 at the sixth. Using the precision envelope (the best precision at this recall or later): AP \(= \tfrac13 \cdot 1 + \tfrac13 \cdot 0.333 = 0.444\). The third true segment is never found, so recall stops at 2/3. Over \(\Theta = \{0.3, \dots, 0.7\}\) the APs are 1, 1, 1, 0.444, 0.444, so avg-mAP (with one class) is 0.778.

ActionFormer: one prediction per moment

Early methods worked in two stages: propose candidate segments, then classify each one. ActionFormer (Zhang, Wu and Li, 2022) is a simple single-stage, anchor-free detector. [Certain] It works on pre-extracted features: a frozen video encoder (for THUMOS14, a two-stream I3D on 16-frame windows with stride 4, giving 2048 numbers per time step) turns the video into a sequence of feature vectors. [Certain] Then:

  1. A transformer encoder with local self-attention (each time step attends only to a window of neighbors, 19 on THUMOS14) mixes context. [Certain]
  2. Between levels the sequence is downsampled by 2, giving a feature pyramid of 6 levels. Short actions are found on fine levels, long actions on coarse levels. [Certain]
  3. At every time step \(t\) of every level, a light head predicts class probabilities \(p(c \mid t)\) and two distances: \(d^s_t\) back to the start and \(d^e_t\) forward to the end.
\[ \hat{S}_t = \big[\, (t - d^s_t)\, \sigma_l ,\ (t + d^e_t)\, \sigma_l \,\big], \qquad \text{score}_t = \max_c\, p(c \mid t) \tag{203.7} \]
Read the symbols
\(t\)
a time step (a “moment”) on pyramid level \(l\)
\(d^s_t, d^e_t \ge 0\)
predicted distance from \(t\) back to the action start and forward to its end
\(\sigma_l\)
the stride of level \(l\) in seconds (it doubles at each level)

Training uses a focal loss for classification (it down-weights the many easy background moments) and a distance-IoU loss for the boundaries, which is only applied at moments inside an action (positives); only moments near an action’s center are chosen as positives (center sampling). [Certain]

\[ \mathcal{L} = \frac{1}{N_{+}} \sum_{l, t} \Big( \mathcal{L}_{\text{focal}}\big(p_{t}, c_{t}\big) + \lambda\, \mathbb{1}[t \text{ positive}]\; \mathcal{L}_{\text{DIoU}}\big(\hat{S}_t, S_t\big) \Big) \tag{203.8} \]

Here \(N_+\) is the number of positive moments and \(\lambda = 1\) by default. [Certain] At test time, many neighboring moments predict nearly the same segment, so duplicates are removed by non-maximum suppression (NMS). ActionFormer uses Soft-NMS, which lowers the scores of overlapping segments instead of deleting them. [Certain] On THUMOS14 (20 classes, 413 untrimmed videos) it reported 71.0% mAP at tIoU 0.5 and 66.8% avg-mAP over 0.3–0.7; on ActivityNet-1.3 (200 classes, about 20k videos) 36.6% avg-mAP. [Certain] Step through the decoding in Figure 3.

ground truthmoment scorecandidate segmentkept after NMS
Figure 3. Decoding a single-level ActionFormer-style head on a 30-second video with one moment per second. Two true actions: 9–15 s and 21–24 s. The scores and distances are made-up but realistic numbers; the steps (threshold, regress, NMS) are the real procedure.
How to read Figure 3

Step 2 shows the classification head: high scores near the centers of the actions. Step 3 keeps moments with \(\text{score} \ge 0.3\) and draws their two distances as arrows. Step 4 turns each into a segment: seven candidates, all overlapping one of two actions. Step 5 runs greedy NMS (\(\operatorname{tIoU} \ge 0.5\) suppresses): two segments remain, with tIoU 0.972 and 0.783 to the truth. Without NMS the five extra segments would all be false positives in Figure 2’s sense.

Algorithm 2: ActionFormer-style inference on an untrimmed video
  1. input: video, frozen clip encoder, trained pyramid transformer and heads, thresholds \(\text{score}_{\min}\), \(\theta_{\text{nms}}\)
  2. slide the clip encoder over the video: one feature vector per time step // e.g. 16 frames, stride 4
  3. run the local-attention transformer; build the pyramid by downsampling ×2 per level
  4. for each level \(l\) and each step \(t\) do
  5.   compute \(p(c \mid t)\), \(d^s_t\), \(d^e_t\); segment \(\hat{S}_t\) from formula (203.7)
  6. keep the top candidates with score \(\ge \text{score}_{\min}\), over all levels
  7. for each class: sort by score; (Soft-)NMS with tIoU \(\theta_{\text{nms}}\) // remove duplicates
  8. return the list of \((\hat{s}, \hat{e}, c, \text{score})\)

Zero-shot action recognition with video–text models

A classifier trained on Kinetics-400 knows exactly 400 actions. A sports app may need “pick and roll”, a factory may need “opening the safety gate”. Collecting labeled clips for every new class is slow. Zero-shot recognition uses a model trained on video–text or image–text pairs so that the class is given as text at test time.

The recipe follows CLIP’s zero-shot image classifier (session 166):

  1. Turn each class name into a sentence, such as “a video of a person doing pole vault”, and embed it with the text encoder: \(\mathbf{e}_c\).
  2. Embed \(T\) frames with the image encoder, average them (or use a temporal module), and normalize: \(\mathbf{v}\).
  3. Score every class by cosine similarity and take a softmax with a temperature \(\tau_{\text{zs}}\):
\[ \mathbf{v} = \frac{\sum_{i=1}^{T} \mathbf{z}_i}{\big\lVert \sum_{i=1}^{T} \mathbf{z}_i \big\rVert}, \qquad p(c \mid V) = \frac{\exp\!\big(\mathbf{v}^\top \mathbf{e}_c / \tau_{\text{zs}}\big)}{\sum_{c'} \exp\!\big(\mathbf{v}^\top \mathbf{e}_{c'} / \tau_{\text{zs}}\big)} \tag{203.9} \]
Read the symbols
\(\mathbf{z}_i\)
the normalized image embedding of frame \(i\)
\(\mathbf{e}_c\)
the normalized text embedding of class \(c\)
\(\tau_{\text{zs}}\)
temperature; CLIP learns a logit scale of 100, so \(\tau_{\text{zs}} = 0.01\)

Single prompts are fragile. Prompt ensembling averages the text embeddings of several templates (“a video of {c}”, “a person {c}”, “a clip of someone doing {c}”):

\[ \mathbf{e}_c = \operatorname{normalize}\Big( \frac{1}{J} \sum_{j=1}^{J} g\big(\text{template}_j(c)\big) \Big) \tag{203.10} \]

where \(g\) is the text encoder and \(J\) the number of templates. It is free at test time, because the class embeddings are computed once.

Figure 4. Zero-shot classification with a toy 6-dimensional embedding space whose axes mean “run”, “take-off”, “bar”, “pole”, “sand pit” and “hop”. Five class prompts and eight frames per video are fixed vectors in this space. Left: cosine similarity of the video vector to each prompt. Right: softmax probabilities, formula (203.9).
How to read Figure 4

With the mean of all frames, the pole-vault video is closest to pole vault (cosine 0.962 against 0.729 for high jump). Now use single frames: frames 1–3 are the run-up and give running; frame 8 (landing on the mat) gives high jump. For the triple-jump video, frame 7 (landing in sand) gives long jump. Pooling over the whole action fixes this. Then move the temperature: a small \(\tau\) makes the model very sure, a large one spreads the probability. Note one thing the figure cannot show: the mean is the same for any frame order, so mean pooling cannot tell “take-off then landing” from “landing then take-off”.

From image–text to video–text models

Plain CLIP with mean pooling is a strong baseline, but it ignores time. Several models adapt it to video while keeping zero-shot ability:

  • ActionCLIP (2021) fine-tunes CLIP on video–label pairs, treating the label as text, with prompts and a temporal fusion module. [Certain]
  • X-CLIP (2022) adds cross-frame attention between frame tokens and makes video-specific text prompts from the video content. Trained on Kinetics-400 with a ViT-B/16 backbone and 32 frames, it reported zero-shot top-1 of 72.0% on UCF-101, 44.6% on HMDB-51 and 65.2% on 160 sampled Kinetics-600 classes that are not in Kinetics-400; ActionCLIP had 58.3% and 40.8% on the first two. [Certain]
  • Video foundation models such as InternVideo2 are pretrained on large video–text data and are used the same way. [Likely]

Two cautions. First, the zero-shot protocol matters: “unseen” classes must truly be absent from training, which is why X-CLIP tests on Kinetics-600 classes outside Kinetics-400. Second, class names are often ambiguous (“dribbling” in football vs basketball). Add context to the prompt (“dribbling a basketball”) or describe the class in a full sentence.

A third route is a Video-LLM: show the frames and ask “Which of these actions is shown: …?” It is the most flexible (any class list, any explanation) and the slowest. It is covered in session 205.

Sports actions: short, fast and fine-grained

Sports look like the perfect use case: clear rules, known classes, huge archives. In practice they are among the hardest action data. Four things work against you.

  1. Short. A tennis serve contact, a basketball release, a kick: the decisive part often lasts a fraction of a second. In gymnastics, FineGym reports that most sub-action instances are short, and a whole vault averages about 8 seconds. [Certain]
  2. Fast. The ball and limbs move many pixels per frame; at 25–30 fps they blur. A stride that is fine for “cooking” skips the action.
  3. Fine-grained. Classes differ in small details: a double vs a triple twist, a forward vs a backward salto, a jump shot vs a fadeaway. The background, the clothes and the scene are the same for all of them, so scene bias gives no help.
  4. Small and crowded. In broadcast footage the player can be 50 pixels tall among 21 others, and the camera pans and cuts to replays.

The first two points are about sampling. If an action lasts \(d\) seconds and we keep every \(\tau\)-th frame of an \(f\)-fps video, the number of sampled frames inside the action is about

\[ n_{\text{in}} \approx \frac{d \cdot f}{\tau} \tag{203.11} \]

A 0.3-second contact at 30 fps with stride 8 gives \(0.3 \cdot 30 / 8 = 1.125\) frames: sometimes one, sometimes two, and never enough to see motion. With stride 2 we get 4.5 frames; at 60 fps with stride 2, 9 frames. With sparse segment sampling over a whole 10-second clip and \(T = 8\), the expected count is \(T \cdot d / L = 8 \cdot 0.3 / 10 = 0.24\) frames: usually zero.

Figure 5. Five seconds of video. Gray ticks are all frames; dark ticks are the frames we sample (every \(\tau\)-th, starting at frame 0). The orange band is a short action. The readout counts the sampled frames that fall inside it and compares with formula (203.11).
How to read Figure 5

With the default (0.3 s, 30 fps, stride 8) only one sampled frame lands in the action. Slide the action start: the count jumps between 1 and 2, because it depends on where the action falls relative to the sampling grid. Set the stride to 16: some positions give zero frames, and the model cannot see the action at all. Then try 60 fps with stride 2. This is why sports models use dense sampling or high frame rates, and why the action spotting models below read every frame or every second frame.

FineGym measured this directly. On its Gym99 set, a TSN model went from 35.46% (1 frame) to 61.4% (3), 70.8% (5), 74.4% (7) and 78.82% (12 frames), and saturated at around 12 frames. [Certain] Models that use temporal order also mattered: with RGB input, TSM (temporal shift) reached 70.6% mean accuracy against 61.4% for TSN, and shuffling frames at test time hurt the order-aware TRN model a lot. [Certain] Kinetics pretraining was not always helpful there. [Certain] The message: for fine-grained sports, more frames and order-aware models beat a bigger image backbone.

Action spotting: one timestamp per event

For many sports events the start and end are not well defined. When does a goal start? So the broadcast world uses action spotting: predict one timestamp \(\hat{t}\) per event (goal, card, corner, substitution). SoccerNet-v2 provides about 300k annotations on 500 broadcast football games, for spotting and other tasks [Certain], with 17 action classes for spotting [Likely]. A prediction counts as a hit if it is close enough to the true timestamp \(t^*\):

\[ \text{hit} \iff |\hat{t} - t^*| \le \frac{\delta}{2} \tag{203.12} \]

Here \(\delta\) is the tolerance. AP is computed as in Figure 2, with this rule instead of tIoU, and averaged over several tolerances (a “loose” range of tens of seconds and a “tight” range of a few seconds). [Likely] Spotting with tight tolerance is close to key-moment detection, which is the topic of session 207.

Code: sampling, aggregation, localization metrics, decoding and zero-shot

The code below runs on CPU in a second and prints the same numbers as the figures. It uses random weights for the clip model, because the point is the shapes and the logic, not accuracy.

import math, torch
import torch.nn.functional as F
torch.manual_seed(0)

# 1) clip sampling, formula (203.1): K clips of T frames with stride tau, spread over the video
def clip_indices(num_frames, T, tau, K):
    span = T * tau
    starts = torch.linspace(0, max(0, num_frames - span), K).round().long()
    return starts[:, None] + tau * torch.arange(T)[None, :]          # (K, T) frame ids

idx = clip_indices(num_frames=300, T=8, tau=8, K=5)                  # a 10 s video at 30 fps
print(idx[:, [0, -1]].tolist())                                       # first and last frame of each clip

# 2) a tiny clip classifier and video-level aggregation, formula (203.2)
class TinyVideoNet(torch.nn.Module):
    def __init__(self, num_classes=400):
        super().__init__()
        self.conv = torch.nn.Conv3d(3, 16, kernel_size=(3, 7, 7), stride=(1, 4, 4), padding=(1, 3, 3))
        self.head = torch.nn.Linear(16, num_classes)
    def forward(self, x):                      # x: (B, 3, T, H, W)
        h = F.relu(self.conv(x)).mean(dim=(2, 3, 4))   # global space-time average pool
        return self.head(h)                    # (B, num_classes) logits

model = TinyVideoNet().eval()
video = torch.rand(3, 300, 64, 64)                                     # (C, frames, H, W)
clips = torch.stack([video[:, i] for i in idx])                       # (K, 3, T, H, W)
with torch.no_grad():
    probs = model(clips).softmax(-1)                                  # (K, 400) per-clip probabilities
p_video = probs.mean(0)                                               # average over clips
print(clips.shape, p_video.shape, round(p_video.sum().item(), 4))

# 3) temporal IoU (203.5) and average precision for one class (203.6)
def tiou(a, b):
    inter = max(0.0, min(a[1], b[1]) - max(a[0], b[0]))
    return inter / (max(a[1], b[1]) - min(a[0], b[0]))

def average_precision(preds, gts, thr):
    preds = sorted(preds, key=lambda p: -p[0])                       # (score, start, end)
    used, tp = set(), []
    for s, a, b in preds:
        cand = [(tiou((a, b), g), j) for j, g in enumerate(gts) if j not in used]
        best = max(cand, default=(0.0, None))
        if best[0] >= thr: used.add(best[1]); tp.append(1)
        else: tp.append(0)
    tp = torch.tensor(tp, dtype=torch.float)
    prec = tp.cumsum(0) / torch.arange(1, len(tp) + 1)
    rec = tp.cumsum(0) / len(gts)
    prec = torch.cat([torch.zeros(1), prec, torch.zeros(1)])
    rec = torch.cat([torch.zeros(1), rec, torch.ones(1)])
    for i in range(len(prec) - 2, -1, -1):                           # precision envelope
        prec[i] = torch.maximum(prec[i], prec[i + 1])
    return ((rec[1:] - rec[:-1]) * prec[1:]).sum().item()

gts = [(8, 14), (30, 33), (45, 52)]
preds = [(0.95, 8.5, 14.5), (0.90, 29, 34.5), (0.80, 44, 49), (0.60, 9, 13), (0.40, 20, 24), (0.30, 46.5, 52.5)]
aps = [average_precision(preds, gts, t) for t in (0.3, 0.4, 0.5, 0.6, 0.7)]
print([round(a, 4) for a in aps], round(sum(aps) / 5, 4))           # AP per threshold, average mAP

# 4) ActionFormer-style decoding (203.7): each moment t predicts a score and distances to start and end
def decode(scores, d_start, d_end, stride=1.0, score_thr=0.3, nms_thr=0.5):
    t = (torch.arange(len(scores)) + 0.5) * stride                    # time of each moment, seconds
    keep = scores >= score_thr
    segs = torch.stack([t - d_start * stride, t + d_end * stride], 1)[keep]
    sc = scores[keep]
    out = []
    for i in sc.argsort(descending=True, stable=True).tolist():       # greedy 1-D NMS
        s = segs[i].tolist()
        if all(tiou(s, o[1:]) < nms_thr for o in out):
            out.append((round(sc[i].item(), 3), round(s[0], 2), round(s[1], 2)))
    return out

scores = torch.tensor([0.05, 0.2, 0.7, 0.9, 0.8, 0.3, 0.05])
d_s = torch.tensor([0.0, 1.4, 2.1, 3.0, 4.2, 4.9, 0.0])
d_e = torch.tensor([0.0, 3.8, 2.9, 2.0, 0.9, 0.3, 0.0])
print(decode(scores, d_s, d_e))

# 5) zero-shot classification (203.9): cosine similarity between a pooled video embedding and class texts
names = ["high jump", "long jump", "pole vault", "triple jump", "running"]
text = F.normalize(torch.tensor([[.5, 1, 1, 0, 0, 0], [.6, .8, 0, 0, 1, 0], [.5, .6, 1, 1, 0, 0],
                                 [.6, .6, 0, 0, 1, 1], [1., 0, 0, 0, 0, 0]]), dim=-1)
frames = F.normalize(torch.tensor([[1, 0, 0, .5, 0, 0]] * 3 + [[.6, .5, 0, 1, 0, 0], [.1, 1, .3, 1, 0, 0],
                                   [0, .6, 1, .4, 0, 0], [0, .2, .6, .2, 0, 0], [0, 0, .3, 0, 0, 0]]), dim=-1)
video_emb = F.normalize(frames.mean(0), dim=-1)                       # mean pooling over 8 frames
p = (video_emb @ text.T / 0.01).softmax(-1)                           # temperature 0.01 (CLIP's logit scale 100)
print(names[p.argmax()], round(p.max().item(), 3), names[(frames[0] @ text.T).argmax()])

Output (PyTorch 2.8, CPU):

[[0, 56], [59, 115], [118, 174], [177, 233], [236, 292]]
torch.Size([5, 3, 8, 64, 64]) torch.Size([400]) 1.0
[1.0, 1.0, 1.0, 0.4444, 0.4444] 0.7778
[(0.9, 0.5, 5.5)]
pole vault 1.0 running

Read it line by line: five clips of 64 frames each cover the 300 frames; the video-level vector is a proper distribution; the AP values match Figure 2; the four overlapping candidates in part 4 collapse into one segment after NMS; and mean pooling picks pole vault while the first frame alone says running, as in Figure 4.

Where this is used

ApplicationTask formTypical approach
Video platform tagging and searchVideo-level multi-label classificationFrozen video or CLIP embeddings + linear heads; embeddings reused for session 208
Sports highlights (goals, dunks, aces)Action spotting / localization in long broadcastsDense per-frame features, a temporal model (transformer or 1D CNN), tolerance-based AP; key-moment detection
Coaching and judging (gymnastics, diving, tennis technique)Fine-grained classification and segmentation into phasesHigh fps, dense sampling, pose features, order-aware models
Content moderation (violence, dangerous acts)Clip classification on uploads; zero-shot for new policiesVideo–text models with prompts, then a small labeled set to calibrate thresholds
Safety cameras (falls, intrusions, PPE)Online detection on a live streamShort sliding windows, low latency, per-person tracking (session 204)
Video-RAG and Video-LLMsFind the moment that answers a questionLocalization or spotting to pick clips, then a VLM reads them (session 210)
Pretraining video encodersKinetics classification as a benchmark or as supervised pretrainingThe encoder is then used for detection, retrieval and Video-LLMs

Common mistakes

  • Comparing single-clip and multi-view numbers. A 30-view result can be several points higher than one clip and costs 30× more. Always state the views and the FLOPs per view.
  • Trusting Kinetics accuracy as “motion understanding”. Scene bias lets image models do well. Test on a motion-centric set (Something-Something) or use the frame-shuffle test.
  • Horizontal flip on direction classes. Flipping “moving something left to right” makes it the other class. Turn off flip, or swap the labels.
  • Averaging clips for short actions in long videos. The background dominates (Figure 1). Use max pooling, attention pooling or a localization model.
  • Too large a stride for fast sports. Formula (203.11): at 30 fps with stride 8, a 0.3-s action gets about one frame. Check \(d \cdot f / \tau\) before training.
  • Forgetting NMS or applying it across classes. Duplicates count as false positives and wreck AP. Run NMS per class, and consider Soft-NMS when true actions can overlap.
  • Reporting mAP at one easy threshold. mAP@0.3 hides bad boundaries. Report avg-mAP over the standard thresholds.
  • Leaky zero-shot evaluation. If “unseen” classes are in the pretraining labels or captions, the result is not zero-shot. Use the standard splits and say which data the model saw.

Interview questions

What is the difference between clip-level and video-level accuracy?basic
Clip-level accuracy scores the model on one sampled clip (\(T\) frames with stride \(\tau\)). Video-level accuracy combines several clips (often 10 temporal × 3 spatial crops) by averaging their softmax outputs (203.2) and then checks the top class. Video-level is higher but costs \(K\) forward passes; when you compare models you must use the same protocol and also report the cost per view.
What does “8×8” mean for a video model, and how much time does it cover?basic
\(T = 8\) frames taken every \(\tau = 8\) frames. The clip spans \(T\tau = 64\) raw frames, which is \(64/30 = 2.13\) s at 30 fps. A 16×4 clip covers the same time with twice as many frames: more detail on motion, twice the compute.
Why can many Kinetics classes be recognized from a single frame, and how do you check whether your model uses motion?core
Kinetics classes are often tied to objects and scenes (pool for swimming, guitar for playing guitar), so appearance alone is a strong cue: this is scene or representation bias. To check for motion use: (1) shuffle or reverse frames at test time and measure the drop; (2) evaluate on a motion-centric dataset such as Something-Something v2, whose classes share objects and differ in direction; (3) compare with a single-frame baseline trained on the same data.
How is temporal action localization evaluated?core
Per class, sort predictions by score; a prediction is a true positive if its tIoU (203.5) with a still-unmatched true segment is at least \(\theta\); otherwise (including duplicates) it is a false positive. AP is the area under the precision–recall curve; \(\text{mAP}@\theta\) averages over classes; avg-mAP averages over thresholds (0.3–0.7 on THUMOS14, 0.5–0.95 on ActivityNet). Higher thresholds reward precise boundaries.
Explain ActionFormer in a few sentences.core
It is a single-stage, anchor-free detector over a sequence of pre-extracted clip features. A transformer with local self-attention builds a 6-level temporal feature pyramid (×2 downsampling per level). At every moment of every level, a head predicts class probabilities and distances to the start and end, which define a segment (203.7). It trains with focal loss plus a DIoU regression loss on positive moments near action centers, and removes duplicates with Soft-NMS. It reported 71.0% mAP@0.5 on THUMOS14.
Why does the localization head predict distances to the boundaries instead of fixed anchor windows?advanced
Action lengths vary from under a second to minutes. Anchors need a hand-chosen set of lengths per position and a matching rule, and any length not covered is hard to fit. Predicting \((d^s, d^e)\) from each moment is anchor-free: any length can be produced, and the pyramid handles scale (short actions on fine levels, long ones on coarse levels). It also makes the head simple: one classifier and one 2-number regressor per moment.
How does zero-shot action recognition with CLIP work, and what is its main weakness?core
Embed each class name inside prompt templates with the text encoder (ensemble several templates, 203.10); embed sampled frames with the image encoder, average and normalize; take cosine similarities divided by a temperature and a softmax (203.9). The main weakness is that mean pooling is order-blind and frame-level: it recognizes scenes and objects well but not direction, order or fine motion. Video adaptations (ActionCLIP, X-CLIP) add temporal modules and video-aware prompts.
Your client wants to recognize 20 new actions and has no labels. What do you do?core
Start with zero-shot video–text classification and well-written prompts (full descriptions, sport context), and evaluate it on a small hand-labeled set of maybe 50 clips per class. If it is not good enough, use it to pre-label data, have humans correct a few hundred clips, and train a linear probe or small head on frozen embeddings (few-shot). Fine-tune the encoder only if needed. Watch for classes that differ only in motion, where frame-based zero-shot will fail.
Why are sports actions hard for standard action recognition models?core
They are short (a fraction of a second for the key part), fast (motion blur, large displacement between frames), fine-grained (classes share scene, clothes and most of the motion), and the players are small in broadcast frames with camera motion and replays. Standard sparse sampling may not put any frame on the action (203.11), scene bias does not help, and order matters. FineGym showed TSN accuracy rising from 35% with 1 frame to about 79% with 12 frames, and order-aware models beating plain averaging.
Design a system that finds goals in full football broadcasts.advanced
Frame the task as action spotting (one timestamp per goal, tolerance-based AP, 203.12). Decode at 2–5 fps, extract per-frame features with a frozen pretrained encoder (batch on GPU), and run a temporal model (transformer or 1D CNN) over a window of tens of seconds that outputs per-frame class scores; spot with peak picking plus NMS in time. Add audio (crowd noise) and on-screen graphics (score change) as strong cues. Handle replays by shot-boundary and replay detection. Measure on held-out matches, tune the threshold for the product’s precision target, and keep humans in the loop for publishing.
What is the role of SlowFast’s two pathways, and what do \(\alpha\) and \(\beta\) mean?advanced
The Slow pathway reads few frames (large stride) with full channel width and learns appearance. The Fast pathway reads \(\alpha\) times more frames (\(\alpha = 8\)) with only \(\beta = 1/8\) of the channels and no temporal downsampling, so it captures motion cheaply (about 20% of the compute). Lateral connections (time-strided convolutions) feed Fast into Slow. On Kinetics-400 this beat a Slow-only model by 3 points top-1 in the paper’s 4×16 ablation (75.6 vs 72.6).

Exercises

  1. A model uses 16×4 clips on 25-fps video. How many seconds does one clip cover? How many clips with no overlap cover a 10-s video?
  2. In Figure 1, use one clip with \(T = 16\), \(\tau = 8\). Compute the clip’s position, \(c\), and the predicted class by hand.
  3. Compute the tIoU of \([10, 20]\) and \([15, 30]\). Is it a match at \(\theta = 0.5\)?
  4. In Figure 2, what is the AP at \(\theta = 0.8\)? Check with the slider.
  5. An ActionFormer moment at \(t = 12\) on a level with stride \(\sigma_l = 0.5\) s predicts \(d^s = 4\), \(d^e = 6\). What segment does it give, in seconds?
  6. A 0.4-s action, 60-fps video. What is the largest stride that still gives at least 4 sampled frames inside the action on average?
  7. Why does prompt ensembling not slow down inference?
Answerscheck
1. \(16 \cdot 4 / 25 = 2.56\) s; \(10 / 2.56 = 3.9\), so 4 clips (the last one slightly overlaps or is padded). 2. Span \(16 \cdot 8 / 30 = 4.27\) s, centered at 5 s: 2.87–7.13 s. It sees 6.0–7.13 s, \(c = 1.13/1.5 = 0.756\). Logits 3.14, 2.01, 1.24; softmax 0.68, 0.22, 0.10: high jump, correct. 3. Overlap \(20 - 15 = 5\), union \(30 - 10 = 20\), tIoU 0.25: not a match. 4. Only the 0.95 prediction (tIoU 0.846) matches; recall 1/3 with precision 1, so \(\text{AP} = 0.333\). 5. \([(12 - 4) \cdot 0.5,\ (12 + 6) \cdot 0.5] = [4, 9]\) s. 6. \(0.4 \cdot 60 / \tau \ge 4\) gives \(\tau \le 6\). 7. The class text embeddings are averaged once and cached; at test time each video is still compared with one vector per class.

Key takeaways

  • Video models read clips of \(T\) frames with stride \(\tau\) (203.1); a video-level answer averages \(K\) clips (203.2). Averaging works when the action fills the video and fails for short actions in long videos.
  • Kinetics is the standard benchmark but has scene bias; Something-Something tests motion. Always ask which one a number comes from.
  • Temporal action localization outputs (start, end, class, score); it is measured with tIoU (203.5) and mAP averaged over thresholds (203.6). Duplicates are false positives, so NMS is part of the model.
  • ActionFormer predicts, at every moment of a 6-level pyramid, a class score and two boundary distances (203.7), trained with focal + DIoU loss (203.8).
  • Zero-shot recognition compares a pooled video embedding with prompt embeddings of class names (203.9, 203.10). It is order-blind with mean pooling; video adaptations such as X-CLIP improve it.
  • Sports actions are short, fast and fine-grained: count \(d f / \tau\) (203.11), sample densely, use order-aware models, and use spotting (203.12) when boundaries are unclear.

Key formulas

\[ \text{span} = \frac{T\tau}{f}, \quad p(y \mid V) = \frac1K \sum_k \softmax(f_\theta(X_k))_y, \quad \operatorname{tIoU} = \frac{|\hat{S} \cap S|}{|\hat{S} \cup S|} \]
\[ \hat{S}_t = [(t - d^s_t)\sigma_l,\ (t + d^e_t)\sigma_l], \quad p(c \mid V) = \softmax_c\!\big(\mathbf{v}^\top \mathbf{e}_c / \tau_{\text{zs}}\big), \quad n_{\text{in}} \approx \frac{d f}{\tau} \]

References for this session

  1. Kay et al. (2017). The Kinetics Human Action Video Dataset
  2. Zhang et al. (2022). ActionFormer: Localizing Moments of Actions with Transformers
  3. Feichtenhofer et al. (2018). SlowFast Networks for Video Recognition
  4. Kay et al. (2017). The Kinetics Human Action Video Dataset
  5. Feichtenhofer et al. (2018). SlowFast Networks for Video Recognition
  6. Zhang et al. (2022). ActionFormer: Localizing Moments of Actions with Transformers
  7. Carreira & Zisserman (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
  8. Wang, Xiong, Wang, Qiao, Lin, Tang & Van Gool (2016). Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
  9. Soomro, Zamir & Shah (2012). UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
  10. Goyal et al. (2017). The "something something" video database for learning and evaluating visual common sense
  11. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  12. Wang, Xing & Liu (2021). ActionCLIP: A New Paradigm for Video Action Recognition
  13. Ni, Peng, Chen, Zhang, Meng, Fu, Xiang & Ling (2022). Expanding Language-Image Pretrained Models for General Video Recognition
  14. Bodla et al. (2017). Soft-NMS: Improving Object Detection With One Line of Code
  15. Shao, Zhao, Dai & Lin (2020). FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding
  16. Deliège et al. (2021). SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos
  17. Tong et al. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
  18. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models