LLM & VLM courseBlog
175Track 7: Vision and VLMs · session 13 of 38

Early video–language pretraining: VideoBERT, VideoCLIP, MERLOT

Videos on YouTube come with free text: the words people say, turned into a transcript by speech recognition. Three models from 2019 to 2021 show three ways to learn from that text. VideoBERT turns video into discrete “visual words” and trains BERT on them, VideoCLIP learns a contrastive video–text space with better positives and harder negatives, and MERLOT learns what happens in a video and in what order.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions on BERT-style masked language modeling (33), CLIP (166) and two-stream VLMs (172)

What you will learn

  • Why narrated videos give free but noisy labels, and why speech and action are often not at the same time
  • How VideoBERT turns 1.5-second clips into 20,736 visual words with hierarchical k-means, and trains BERT-large on words plus visual words
  • How VideoCLIP builds positive pairs from overlapping (not exactly aligned) clips and hard negatives from retrieved clusters of similar videos
  • The symmetric InfoNCE loss, why hard negatives give a larger gradient, and how VideoCLIP does zero-shot retrieval, VideoQA and action segmentation
  • How MERLOT trains an image encoder from scratch on 180M frame–transcript segments with three losses: contrastive matching, attention-masked language modeling and temporal reordering
  • What the ablations measured: data size, overlap and retrieval, data diversity, and which loss mattered most
  • Which lessons modern Video-LLMs kept: speech as free supervision, explicit time alignment, frozen or cheap frame features, and two-stage retrieval

Why this matters

Labeled video is very expensive. To build a dataset like YouCook II, people watched 2,000 cooking videos and wrote a caption for every step by hand [Certain]. But YouTube has millions of videos where a person talks while doing something: “now crack two eggs into the bowl”. Speech recognition (ASR, automatic speech recognition) turns that talk into text with time stamps, for free. This session is about the first models that learned from that free text: VideoBERT (2019), VideoCLIP (2021) and MERLOT (2021).

The three papers are three different answers to one question: how do we connect a stream of frames with a stream of spoken words?

  • VideoBERT makes video look like text. It cuts the video into 1.5-second clips, turns each clip into one of 20,736 discrete “visual words”, and trains a BERT model on sentences made of real words and visual words [Certain].
  • VideoCLIP keeps video and text in two separate encoders, like CLIP (session 166), and trains them with a contrastive loss. Its new ideas are about which pairs to use: positives that overlap in time instead of being exactly aligned, and negatives taken from similar videos [Certain].
  • MERLOT learns from frames and transcripts of 6 million videos with a joint encoder. Besides matching frames with words, it must put shuffled frames back in the right order, so it learns what happens and in what order [Certain].

Why study them now, when Video-LLMs (session 205) read frames with a ViT and answer in free text? Because the problems they found are still the problems. Speech is still the cheapest supervision for video. Speech and action are still not at the same time. Contrastive video–text encoders are still the first stage of text-to-video search and Video-RAG (session 208, session 210). And “in what order did things happen?” is still a weak point of many video models. This session sits between image–text pretraining (sessions 166 to 174) and the video track (Track 8).

Speech as free labels, and why they are noisy

A narrated video gives us two streams on one time axis: frames, and ASR words with start and end times. The simplest way to make training pairs is exact alignment: take one ASR sentence and the video between its start and end times, and call them a positive pair. HowTo100M (Miech et al., 2019) did this at scale: 136 million clips from 1.22 million narrated instructional videos, covering more than 23,000 tasks [Certain].

The problem: people do not speak and act at the same time. VideoCLIP’s paper gives the example “I am going to show you how to cook fried rice”: the frames under that sentence show a person speaking, and the rice in the wok appears much later [Certain]. Often the speaker first says what they will do and then does it, or does it and then explains. Some sentences (“don’t forget to subscribe”) have no visual content at all. So exactly aligned pairs are often low relevance: the text does not describe the frames under it.

Figure 1 makes this concrete with a toy video of 120 seconds and 8 spoken sentences. For each sentence we know when the matching action is really visible. We measure two things for a candidate video clip \([x_1, x_2]\) and the action interval \([a, b]\):

\[ \text{coverage} = \frac{\lvert [x_1, x_2] \cap [a, b] \rvert}{b - a}, \qquad \text{purity} = \frac{\lvert [x_1, x_2] \cap [a, b] \rvert}{x_2 - x_1} \tag{175.1} \]
Read the symbols
\([x_1, x_2]\)
the video clip used as the positive for one sentence, in seconds
\([a, b]\)
the time when the action the sentence talks about is really on screen
\(\lvert \cdot \rvert\)
the length of an interval; the intersection \(\cap\) is the part the two intervals share
coverage
how much of the action the clip shows (1 means all of it)
purity
how much of the clip is the action (1 means nothing else is in the clip)
ASR sentence (speech)action really on screenvideo clip used as the positive
Figure 1. Exact versus overlapping positives on a toy 120-second video. Each row is one ASR sentence (upper thin bar), the time when its action is on screen (lower thin bar), and the video clip that would be paired with it (box; see the legend for colors). In overlapping mode, the box shows the clip centered in the sentence, the dashed box every place the clip can reach, the clip center is drawn uniformly inside the sentence and the clip has length \(L\); the readout averages over 40 evenly spaced centers. Click a row to see its numbers, formula (175.1).
How to read Figure 1

With exact alignment, the clip is the sentence itself. Mean coverage is only about 2.7%: the actions mostly happen just after the words. Switch to overlapping clips with \(L = 16\) s: mean coverage jumps to about 68%, and 5 of the 8 sentences now see at least half of their action. Purity also goes up at first (from about 10% to about 27%), but if you push \(L\) to 32 s, purity falls again (about 20%): long clips contain other steps too. VideoCLIP draws \(L\) at random between 3 and 32 s, so it trains on both short, precise pairs and long, forgiving pairs. Row 1 (“Today we make fried rice”) is the paper’s example: its action happens 30 seconds later, and no clip length here reaches it.

There are three families of fixes, and the three papers use all of them. (1) Make the pairing looser in time: concatenate neighboring sentences (VideoBERT), use overlapping clips of random length (VideoCLIP), or give the model a whole video context (MERLOT). (2) Make the loss tolerant: MIL-NCE (Miech et al., 2020) treats several nearby sentences as candidate positives for one short clip of 3.2 seconds [Certain], so only one of them needs to fit. (3) Pick better data: MERLOT filters out videos whose content is not visual (for example, video game commentaries) and adds punctuation to the raw ASR [Certain].

VideoBERT: video as a second language

VideoBERT (Sun et al., 2019) asked: what is the smallest change to BERT that lets it read video? BERT reads discrete tokens from a vocabulary. So VideoBERT turns video into discrete tokens too, and then trains a normal BERT on mixed sequences [Certain].

Step 1: clips become feature vectors

Frames are sampled at 20 frames per second and cut into non-overlapping windows of 30 frames, so one window is 1.5 seconds [Certain]. Each window goes through S3D (a 3D convolutional network that adds separable temporal convolutions to an Inception backbone, pretrained on the Kinetics action dataset). The activations before its final classifier are average-pooled into one 1024-dimensional vector per clip [Certain]. The S3D network is frozen: it is only a feature extractor.

Step 2: feature vectors become visual words

A 1024-d vector is continuous. To get a vocabulary, VideoBERT runs hierarchical k-means: k-means with \(k\) clusters on all clip vectors, then k-means with \(k\) clusters again inside each cluster, and so on for \(d\) levels. The authors chose \(d = 4\) and \(k = 12\) by looking at the clusters, which gives \(12^4 = 20{,}736\) visual words [Certain]. A clip is assigned by walking down the tree:

\[ c_\ell = \argmin_{j \in \{0, \dots, k-1\}} \big\lVert \mathbf{f} - \mu_{c_1 \dots c_{\ell-1}\, j} \big\rVert_2, \qquad \text{token}(\mathbf{f}) = \sum_{\ell=1}^{d} c_\ell \, k^{\,d-\ell} \tag{175.2} \]
Read the symbols
\(\mathbf{f} \in \R^{1024}\)
the S3D feature of one 1.5-second clip
\(k, d\)
clusters per level and number of levels; the vocabulary has \(k^d\) words
\(\mu_{c_1 \dots c_{\ell-1} j}\)
the centroid of child \(j\) inside the cell chosen at the previous levels
\(c_\ell\)
the child chosen at level \(\ell\); the token id writes \(c_1, \dots, c_d\) as a number in base \(k\)

Why hierarchical and not one flat k-means with 20,736 centers? A flat assignment compares each vector with all 20,736 centroids; the tree compares it with only \(k \cdot d = 48\). Training is cheaper too, since each k-means run has only 12 centers. The price: a point near a cell border at level 1 can end up in a worse leaf than flat k-means would choose. Figure 2 lets you build the tree on 2D points.

Figure 2. Hierarchical k-means on 480 seeded 2D points (a stand-in for 1024-d S3D features). Colors show the level-1 cluster; small crosses are the leaf centroids, which are the visual words. Below the plot, a toy “video” of 12 consecutive clips that drift across the plane is turned into a visual sentence. Click any point to see its path down the tree, formula (175.2).
How to read Figure 2

The vocabulary size is \(k^d\): with \(k = 3, d = 2\) it is 9; with \(k = 6, d = 3\) it is 216. The readout shows the mean squared quantization error (the average squared distance from a point to its leaf centroid): it falls as the vocabulary grows, but each word then covers fewer points, so it is seen less often in training. VideoBERT’s \(12^4 = 20{,}736\) is a trade-off of the same kind, chosen by eye. Watch the visual sentence: nearby clips often get the same word, so real video sentences contain many repeats.

Step 3: mixed sentences and three objectives

Text comes from YouTube ASR, split into sentences by an LSTM language model that adds punctuation, and tokenized into BERT’s 30,000 WordPieces [Certain]. Video tokens that fall inside the time stamps of an ASR sentence form one “video sentence”; when there is no ASR, every 16 video tokens form a segment [Certain]. A new token [>] joins a text sentence and a video sentence [Certain]:

[CLS] orange chicken with [MASK] sauce [>] v01 [MASK] v08 v72 [SEP]

The model is BERT-large (24 layers, width 1024, 16 heads), initialized from the text checkpoint. The 20,736 visual words are appended to the word embedding table and initialized with the S3D cluster centroids, and the input embeddings are frozen during pretraining [Certain]. Training uses three regimes with a weighted sum of losses [Certain]: masked token prediction on text only, masked token prediction on video only, and on video–text pairs a linguistic–visual alignment classifier on [CLS] that predicts whether the sentence and the video sentence are aligned in time:

\[ \mathcal{L} = \lambda_{\text{t}}\, \mathcal{L}_{\text{MLM}}^{\text{text}} + \lambda_{\text{v}}\, \mathcal{L}_{\text{MLM}}^{\text{video}} + \lambda_{\text{a}}\, \mathcal{L}_{\text{align}}, \qquad \mathcal{L}_{\text{MLM}} = -\sum_{l \in \mathcal{M}} \log p_\theta\big(x_l \mid \mathbf{x}_{\setminus \mathcal{M}}\big) \tag{175.3} \]
\[ \mathcal{L}_{\text{align}} = -\big[\, c \log \sigma(\mathbf{u}^\top \mathbf{h}_{\text{CLS}}) + (1 - c) \log\big(1 - \sigma(\mathbf{u}^\top \mathbf{h}_{\text{CLS}})\big) \big] \tag{175.4} \]
Read the symbols
\(\mathcal{M}\)
the masked positions; for video-only data they are visual words, so the model must predict one of 20,736 classes
\(\mathbf{x}_{\setminus \mathcal{M}}\)
the sequence with those positions replaced by [MASK]
\(\lambda_{\text{t}}, \lambda_{\text{v}}, \lambda_{\text{a}}\)
loss weights (the paper says “weighted sum” but does not give the values)
\(c\)
1 if the text and video sentences come from the same time, 0 if not
\(\mathbf{h}_{\text{CLS}}, \mathbf{u}, \sigma\)
the [CLS] output, a learned vector, and the sigmoid

The masked video loss is a “language model for video”: to fill a masked clip, the model must know what usually comes before and after it in a recipe. The alignment loss is the cross-modal part. The authors note it is noisy, for the reason of Figure 1, and add two tricks [Certain]: they randomly concatenate neighboring sentences into one long sentence, so the right words are more likely to be inside the window; and they subsample video tokens at a random rate of 1 to 5 steps, so the model sees actions at different speeds and over longer time spans.

Worked example: how big is VideoBERT’s data in tokens?

The pretraining set has 312K cooking videos with a total of 23,186 hours, about 966 days [Certain]. At one visual word per 1.5 s, that is \(23{,}186 \times 3{,}600 / 1.5 \approx 55.6\) million visual words. A 5-minute video becomes \(300 / 1.5 = 200\) visual words. Only 180K videos had ASR and about 120K had English ASR; all videos are used for the video-only loss, only English ones for the text losses [Certain].

The new embedding rows: \(20{,}736 \times 1024 \approx 21.2\) million numbers, frozen. Training: 4 Cloud TPUs, batch 128, 0.5 million iterations (about 8 epochs), Adam with learning rate \(10^{-5}\), about 2 days [Certain]. That is \(0.5 \times 10^{6} \times 128 = 64\) million training sequences.

Zero-shot classification by filling a template

Once trained, VideoBERT can be used without any fine-tuning. Give it the visual words of a clip and the fixed sentence “now let me show you how to [MASK] the [MASK]”; read the verb from the first slot and the noun from the second [Certain]:

\[ \hat{\text{verb}} = \argmax_{w}\ p_\theta\big(x_{m_1} = w \mid \text{template},\ v_{1:T}\big), \qquad \hat{\text{noun}} = \argmax_{w}\ p_\theta\big(x_{m_2} = w \mid \text{template},\ v_{1:T}\big) \tag{175.5} \]

On YouCook II validation (labels made from the 45 most common verbs and 100 most common nouns in the captions), VideoBERT reached 43.3% top-5 verb accuracy and 33.7% top-5 object accuracy with no YouCook II training, close to a supervised linear classifier on the same S3D features (46.9% and 30.9%) [Certain]. Its top-1 verb accuracy was only 3.2% (supervised: 16.1%), partly because the vocabulary is open: the model may say “mix” when the label is “stir” [Certain]. Without the video, its language prior gets 6.9% top-5 verb accuracy [Certain], so the video really carries the signal. And accuracy grew with the number of pretraining videos with no sign of saturation: top-5 verb accuracy 15.5%, 15.7%, 24.5%, 43.3% for 10K, 50K, 100K and 300K videos [Certain] (Figure 5). For captioning, VideoBERT features plus S3D features beat the previous best on YouCook II on every metric, for example BLEU-4 4.33 versus 3.84 [Certain].

What did not survive: the discrete tokens. Quantizing a 1024-d feature into one of 20,736 words throws away detail (two different “stir” clips can share a word), and the k-means vocabulary is fixed before training, so the visual side cannot improve. Later models keep continuous features.

VideoCLIP: contrastive learning with better pairs

VideoCLIP (Xu et al., 2021) goes the other way: no fusion encoder, no discrete tokens. It is a dual encoder, like CLIP: one transformer for video, one for text, and similarity by dot product. Its contribution is in the pairs.

The two towers

The video side is a frozen S3D (this time pretrained on HowTo100M by MIL-NCE) that gives one 512-d token per second of video, at 30 fps [Certain]. A trainable one-layer MLP with GELU maps each token to 768-d, then a video transformer runs; the text goes through its own transformer. Both are initialized from BERT-base-uncased: the video side uses only the first 6 layers, the text side all 12 [Certain]. Each side is average-pooled into one vector:

\[ \mathbf{x}_v = f_{\text{MLP}}\big(\operatorname{stopgrad}(f_{\text{S3D}}(c_v))\big), \quad \mathbf{z}_v = \operatorname{AvgPool}\big(f_{\theta_v}(\mathbf{x}_v)\big), \quad \mathbf{z}_t = \operatorname{AvgPool}\big(f_{\theta_t}(\mathbf{x}_t)\big) \tag{175.6} \]
Read the symbols
\(c_v\)
a video clip of 3 to 32 seconds; up to 32 video tokens (plus [CLS] and [SEP], 34 in total) [Certain]
\(\operatorname{stopgrad}\)
no gradient flows into S3D, so its features can be computed once and stored
\(\mathbf{x}_t\)
BERT word embeddings of a text clip of 8 to 61 tokens (63 with [CLS] and [SEP]) [Certain]
\(\mathbf{z}_v, \mathbf{z}_t\)
one vector per clip, the mean over all output tokens

Why average pooling instead of [CLS]? So that every output token must carry meaning, which helps token-level tasks like action segmentation later. In the ablation, using [CLS] lowered zero-shot YouCook2 R@1 from 22.7 to 22.1 [Certain]. Why up to 61 text tokens for 32 video tokens? HowTo100M speech runs at about 2.4 tokens per second, while video tokens come at 1 per second [Certain].

The loss: symmetric InfoNCE

For a batch \(B\) of pairs, the loss pulls each video toward its own text and away from the other texts in the batch, and the same in the other direction:

\[ \mathcal{L} = -\sum_{(v,t) \in B} \Big( \log \operatorname{NCE}(\mathbf{z}_v, \mathbf{z}_t) + \log \operatorname{NCE}(\mathbf{z}_t, \mathbf{z}_v) \Big), \qquad \operatorname{NCE}(\mathbf{z}_v, \mathbf{z}_t) = \frac{\exp(\mathbf{z}_v \cdot \mathbf{z}_t^{+} / \tau)}{\sum_{\mathbf{z} \in \{\mathbf{z}_t^{+}\} \cup \{\mathbf{z}_t^{-}\}} \exp(\mathbf{z}_v \cdot \mathbf{z} / \tau)} \tag{175.7} \]
Read the symbols
\(\mathbf{z}_t^{+}\)
the positive: a text clip that overlaps in time with video clip \(v\)
\(\{\mathbf{z}_t^{-}\}\)
the negatives: all other text clips in the batch
\(\tau\)
the temperature; VideoCLIP sets \(\tau = 1.0\) [Certain]
\(\operatorname{NCE}(\mathbf{z}_t, \mathbf{z}_v)\)
the same with roles swapped: one text against all videos in the batch (used for text-to-video retrieval)

This is the same loss as CLIP’s; read session 166 for the batch-matrix view. Two things are new: how the positive is chosen (overlap), and how the batch is built (retrieval), so that the negatives are hard.

Positives: overlapping clips

To build one positive pair, VideoCLIP (i) samples a text clip first (a random video clip might have no speech near it), (ii) samples a time stamp inside that text clip as the center, and (iii) grows a video clip of random duration around it, up to about 32 seconds [Certain]:

\[ t = [s, e], \quad c \sim \mathcal{U}(s, e), \quad L \sim \mathcal{U}(3, 32)\ \text{s}, \quad v = \big[\, c - \tfrac{L}{2},\ c + \tfrac{L}{2} \,\big] \tag{175.8} \]

The video and text clips share some time but not the same start and end. This is the overlapping mode of Figure 1: the positive usually contains the action even when it happens a few seconds after the words. 16 pairs are sampled per video and 32 videos form a batch of 512 [Certain].

Negatives: retrieval-augmented batches

With random videos in a batch, most negatives are easy: a cooking sentence against a car-repair clip. The loss quickly becomes small and teaches little. VideoCLIP builds batches from clusters of similar videos instead, so that negatives come from videos about the same kind of task. Once per epoch [Certain]:

\[ \mathbf{z}_V = \frac{1}{2 \lvert B_V \rvert} \sum_{(v,t) \in B_V} (\mathbf{z}_v + \mathbf{z}_t), \qquad \text{cluster}(V) = \text{sample } k \text{ videos from } \operatorname{kNN}(\mathbf{z}_V,\ 2k) \tag{175.9} \]
Read the symbols
\(B_V\)
all clip pairs of video \(V\); \(\mathbf{z}_V\) is the mean of their video and text vectors, one vector per video
\(\operatorname{kNN}(\mathbf{z}_V, 2k)\)
the \(2k\) nearest videos in a FAISS index over all \(\mathbf{z}_V\)
\(k\)
videos per batch, \(k = 32\)

Why sample \(k\) from the \(2k\) nearest and not take the \(k\) nearest? The goal is videos that are close to each other, not all close to one seed video; sampling from a slightly larger neighborhood gave better clusters in the ablation (R@1 22.7 versus 22.5 for plain kNN) [Certain].

Why do hard negatives help? Look at the gradient of the InfoNCE loss with respect to the positive score \(s^{+} = \mathbf{z}_v \cdot \mathbf{z}_t^{+} / \tau\): it is \(-(1 - p^{+})\), where \(p^{+}\) is the softmax probability of the positive. If all negatives are far, \(p^{+} \approx 1\) and the gradient is almost zero. Hard negatives keep \(p^{+}\) away from 1, so every pair still pushes.

Worked example: an easy and a hard batch

Three video–text pairs, cosine similarities (rows are videos, columns texts), \(\tau = 0.1\). Easy batch: row 1 is \((0.9, 0.3, 0.1)\). The softmax of \((9, 3, 1)\) is \((0.9972, 0.0025, 0.0003)\), so the loss for this row is \(-\ln 0.9972 = 0.0028\) and the gradient on the positive score is \(-(1 - 0.9972) = -0.0028\).

Hard batch, same positive: row 1 is \((0.9, 0.8, 0.7)\). The softmax of \((9, 8, 7)\) is \((0.6652, 0.2447, 0.0900)\); loss \(-\ln 0.6652 = 0.408\), gradient \(-0.335\), about 120 times larger. Over the whole easy batch \(\left(\begin{smallmatrix} 0.9 & 0.3 & 0.1 \\ 0.2 & 0.8 & 0.4 \\ 0.1 & 0.5 & 0.7 \end{smallmatrix}\right)\) the mean video-to-text loss is 0.051; over the hard batch \(\left(\begin{smallmatrix} 0.9 & 0.8 & 0.7 \\ 0.8 & 0.85 & 0.75 \\ 0.7 & 0.75 & 0.8 \end{smallmatrix}\right)\) it is 0.589. The hard batch carries much more learning signal per pair.

video embeddingtext embeddingpositive pair (diagonal)
Figure 3. Easy and hard negatives for the loss (175.7). 36 toy videos from 6 topics live on a unit circle (topics are 60° apart; videos in a topic are 6° apart; each text is 1° to 2.5° from its video). A random batch takes one video per topic; a retrieved cluster takes all 6 videos of one topic, as formula (175.9) would. Right: the similarity matrix; darker means more similar. Click a row to see its softmax.
How to read Figure 3

At \(\tau = 0.1\), a random batch has a mean negative similarity of about −0.19 and a loss of about 0.07: the model already “knows” the answer. The share of gradient that still flows, the mean of \(1 - p^{+}\), is only about 0.065. Switch to the retrieved cluster: negatives have similarity about 0.96, the loss rises to about 1.54, and the gradient share to about 0.78, roughly 12 times more. Top-1 accuracy inside the batch is still 6 of 6 in both cases: the hard batch is harder, not wrong. Raise \(\tau\) to 1.0 (VideoCLIP’s value): on cosine similarities, which lie in \([-1, 1]\), the logits are squeezed into a narrow range, so the random batch’s loss rises to about 1.04 and the cluster’s to about 1.76, close to \(\ln 6 = 1.79\) (uniform guessing). That is why models with normalized vectors usually use a small or learned \(\tau\).

What the ablations measured

On zero-shot text-to-video retrieval on YouCook2, VideoCLIP reached R@1 22.7, against 15.1 for MIL-NCE and 6.1 for HowTo100M’s model [Certain]. Removing retrieval augmentation lowered R@1 to 18.5; removing overlap as well (exactly aligned pairs) lowered it to 12.4 [Certain]. Using MIL-NCE’s clips and loss inside the same architecture gave 16.1 [Certain]. Sharing one transformer for video and text cost little (21.9) [Certain]. Figure 5 shows the whole table. The results also show the limit of instructional data: zero-shot on MSR-VTT, whose captions describe generic YouTube scenes, R@1 was only 10.4 (MIL-NCE: 9.9); after fine-tuning it was 30.9 [Certain]. The authors name domain shift from HowTo100M as the likely reason [Certain].

Zero-shot on more than retrieval

Because the text tower can embed any label, many video tasks become similarity searches [Certain]. Multiple-choice VideoQA: rank the candidate answers by \(\mathbf{z}_v \cdot \mathbf{z}_{\text{answer}}\) (73.9% zero-shot on MSR-VTT QA). Action step localization on CrossTask: softmax over \(\mathbf{h}_u \cdot \mathbf{z}_{\text{step}}\) for each video token \(u\). Action segmentation on COIN, with an “Outside” label for frames that match nothing:

\[ \hat{y}_u = \begin{cases} \argmax_{t \in \mathcal{L}} \mathbf{h}_u \cdot \mathbf{z}_t & \text{if } \max_{t \in \mathcal{L}} \mathbf{h}_u \cdot \mathbf{z}_t \gt \gamma \\ \text{Outside} & \text{otherwise} \end{cases}, \qquad \gamma = \max_{t \ne t'} \mathbf{z}_t \cdot \mathbf{z}_{t'} \tag{175.10} \]

Here \(\mathbf{h}_u\) is the output of video token \(u\) (one per second), \(\mathcal{L}\) the 778 COIN step labels, and \(\gamma\) a threshold taken from the labels themselves, since zero-shot means no validation data to tune it [Certain]. A token is assigned a step only if it matches that step better than any two labels match each other. Zero-shot frame accuracy was 58.9%, above every supervised method in the paper’s table except its own fine-tuned model (68.7%) and MIL-NCE (61.0%) [Certain]. This is exactly the setup of key-moment detection: score every second of video against a text query.

MERLOT: what happens, and in what order

MERLOT (Zellers et al., 2021) aims at something different: script knowledge, the common sense of how events follow each other (people sit down, then order, then eat). It is a joint encoder trained on a large and diverse video set, and it is tested on reasoning tasks (VCR, TVQA) instead of retrieval.

Data: YT-Temporal-180M

Starting from 27 million candidate videos (HowTo100M, VLOG lifestyle videos and YouTube suggestions for topics like “science”), the authors keep 6 million videos that have English ASR, are at most 20 minutes long, are not in “ungrounded” categories like video game commentary, and whose thumbnails are likely to show objects [Certain]. Each video is cut into segments of \(L = 32\) BPE tokens of transcript; each segment gets the frame from its middle time step. That gives 180 million segments [Certain].

Architecture

Unlike VideoBERT and VideoCLIP, MERLOT trains its image encoder from scratch with the rest of the model, with no supervised features [Certain]. It is a grid model, not a detector: a ResNet-50 (with its last block removed) followed by a 12-layer, 768-d ViT, on widescreen frames of 192 × 352 pixels, with 2 × 2 average pooling at the end. It costs 40 GFLOPs per frame, 2% of the 2 TFLOPs of a Faster R-CNN [Certain]. The output is a \(W/32 \times H/32\) grid plus CLS states. The joint encoder is a 12-layer, 768-d transformer initialized from RoBERTa-base; position embeddings differ per segment, so the model knows which frame and which words belong to which time step [Certain].

Worked example: MERLOT’s sequence length

A frame of 352 × 192 pixels gives a grid of \(352/32 \times 192/32 = 11 \times 6 = 66\) cells. One segment in the joint encoder is then \(66\) grid cells \(+ 1\) image CLS \(+ 32\) word tokens \(= 99\) tokens, and 4 segments give \(4 \times 99 = 396\). The paper reports a joint sequence length of 396 [Certain]; this split of 396 into parts is our reconstruction [Likely].

Training examples have \(N = 16\) segments. The language-only pass for the contrastive loss reads all \(16 \times 32 = 512\) transcript tokens; the joint encoder gets 4 groups of 4 segments to save memory [Certain]. With 1,024 sequences per batch, that is \(1{,}024 \times 16 = 16{,}384\) segments per batch (“16k”), on a TPU v3-1024 pod, 40 epochs, 30 hours [Certain].

Three objectives

1. Contrastive frame–transcript matching. The image encoder must learn to see, and nothing else teaches it from scratch. So each frame’s CLS vector is matched with a text vector of its segment. The text vector comes from a language-only pass over the whole transcript (the segment CLS states), because a segment’s 32 words alone often do not describe its frame. Both are projected to 768-d and L2-normalized; negatives are all other frame–caption pairs in the batch, from the same video or not; \(\tau = 0.05\) [Certain]:

\[ \mathcal{L}_{\text{con}} = -\frac{1}{2M} \sum_{i=1}^{M} \Big( \log \frac{e^{\mathbf{a}_i \cdot \mathbf{b}_i / \tau}}{\sum_{j} e^{\mathbf{a}_i \cdot \mathbf{b}_j / \tau}} + \log \frac{e^{\mathbf{a}_i \cdot \mathbf{b}_i / \tau}}{\sum_{j} e^{\mathbf{a}_j \cdot \mathbf{b}_i / \tau}} \Big) \tag{175.11} \]
Read the symbols
\(\mathbf{a}_i\)
the normalized image vector of frame \(i\) (from the image encoder’s CLS)
\(\mathbf{b}_i\)
the normalized text vector of segment \(i\), computed with the whole transcript as context
\(M\)
the number of segments in the batch (16,384)
\(\tau = 0.05\)
temperature: dot products in \([-1, 1]\) become logits in \([-20, 20]\)

2. Attention-masked language modeling. 20% of the words given to the joint encoder are replaced by [MASK], a random word or kept, and predicted with cross-entropy [Certain]. But speech is not a caption: people say “umm” and “yeah”, and repeat the names of key objects. Random masking often hides fillers, and when it hides “saw” the word may appear again two sentences later, so the model can copy it from the text without looking. MERLOT’s fix, attention masking: half of the time mask a random token, the other half mask one of the 20% of tokens that the language-only encoder attends to most, since those tend to be the visual words; then extend the mask by SpanBERT-style spans [Certain].

3. Temporal reordering. In 40% of the examples, pick \(i\) between 2 and \(N\) and scramble \(i\) frames: their segment position embeddings are replaced by special learned embeddings [image_unk_0], [image_unk_1], … that say nothing about time [Certain]. For every pair of frames, a two-layer MLP reads the two CLS states and predicts which came first [Certain]:

\[ \mathcal{L}_{\text{order}} = -\sum_{i \lt j} \log p_\theta\big(\mathbb{1}[t_i \lt t_j] \;\big|\; [\mathbf{h}_{t_i}; \mathbf{h}_{t_j}]\big) \tag{175.12} \]
Read the symbols
\(\mathbf{h}_{t_i}\)
the joint encoder’s CLS output for frame \(i\)
\([\cdot\,;\cdot]\)
concatenation of the two vectors; the MLP outputs two logits: “\(i\) before \(j\)” or “\(i\) after \(j\)”
\(\mathbb{1}[t_i \lt t_j]\)
the label: 1 if frame \(i\) really comes before frame \(j\)

The three are added, with the contrastive loss multiplied by 0.25 so that its gradients are about as large as the masked LM loss [Certain]; we assume weight 1 for the other two [Likely]:

\[ \mathcal{L}_{\text{MERLOT}} = 0.25\, \mathcal{L}_{\text{con}} + \mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{order}} \tag{175.13} \]

How do we measure ordering? Zero-shot, on visual stories (SIND): 5 images with captions in the right order; the model must put the images in order. MERLOT uses its reordering head to find the most probable order; CLIP and UNITER, which only score image–caption pairs, use a best bipartite matching [Certain]. Two of the metrics, for a predicted order with ranks \(\hat r_i\) and true ranks \(r_i\):

\[ \rho = 1 - \frac{6 \sum_{i=1}^{n} (\hat r_i - r_i)^2}{n (n^2 - 1)}, \qquad \text{pairwise acc} = \frac{\#\{(i, j): i \lt j,\ \text{order of } i, j \text{ correct}\}}{n(n-1)/2} \tag{175.14} \]

MERLOT scored Spearman \(\rho = 0.733\) and pairwise accuracy 84.5%, against 0.609 and 78.7% for CLIP and 0.545 and 75.2% for UNITER [Certain]. Try the metrics in Figure 4.

Training input: what the joint encoder is told about time
Your prediction: click two frames to swap them
Figure 4. Temporal reordering. Top: one training sample with \(N = 5\) segments; \(i\) frames chosen at random (seeded) get [image_unk_k] position embeddings instead of their time step, the rest keep [image_t]. Bottom: a 5-frame story in a scrambled order; swap frames to fix it. The readout gives Spearman \(\rho\) and pairwise accuracy, formula (175.14), and the number of pairwise labels the loss (175.12) uses.
How to read Figure 4

The start order “put tea bag, fill kettle, drink, boil water, pour” has \(\rho = 0.3\) and pairwise accuracy 6/10 = 0.6. Swap two neighbors and watch both metrics change; reach the true order and both are 1. Note the difference: Spearman punishes a frame that is far from its place much more (squared distance), pairwise accuracy only counts pairs. In the top row, frames with a dashed border lost their time stamp; the model must place them using the frames that kept theirs, and their content. With \(N = 5\), the loss always has \(5 \cdot 4 / 2 = 10\) pairwise labels.

What the ablations measured

The MERLOT ablations use VCR question answering (validation accuracy) and TVQA+ [Certain]. Figure 5 shows them. The most important points:

  • The contrastive loss is essential. Without it, VCR fell from 75.2 to 57.5 and TVQA+ from 75.8 to 67.6. Without the ordering loss, VCR was even slightly higher (75.5) and TVQA+ slightly lower (75.6) [Certain]. The authors conclude that the ordering head is mainly an interface to ask the model about time [Certain].
  • Diverse video beats instructional video. Same number of steps: Conceptual Captions + COCO (images) 58.9, HowTo100M 66.3, a HowTo100M-sized subset of YT-Temporal-180M 72.8, the full YT-Temporal-180M 75.2 [Certain].
  • Clean ASR helps. Raw ASR instead of punctuated ASR: 72.8 instead of 75.2 [Certain].
  • Attention masking helps only with longer context. With 1 segment it did not help (73.5 versus 73.8 on VCR); with 4 segments it did (75.2 versus 74.1) [Certain]. More context makes text-only shortcuts easier, and attention masking removes them.
  • Longer training kept helping: VCR 75.2 after 5 epochs, 79.4 after 40, with no plateau [Certain].

After fine-tuning, MERLOT (base size) reached 80.6% on VCR Q→A, above ViLBERT (73.3), UNITER (75.0) and ERNIE-ViL (77.0), all of which use detector features from images [Certain]. It also set a new state of the art on 12 video QA tasks, for example TVQA 78.7 and TGIF-Action 94.0 [Certain], with 5 to 7 frames per clip [Certain]. Its successor MERLOT Reserve (2022) added audio and trained on 20 million videos [Certain].

Figure 5. Ablations from the three papers. VideoBERT Table 2: zero-shot YouCook II action classification versus number of pretraining videos. VideoCLIP Table 7: zero-shot YouCook2 text-to-video retrieval. MERLOT Table 4: VCR validation Q→A (and TVQA+ for losses) for different data, losses and training lengths. All bars start at zero. Click a row to compare it with the first row.
How to read Figure 5

VideoBERT: going from 100K to 300K videos almost doubles top-5 verb accuracy (24.5 to 43.3). VideoCLIP: the first three rows are the story of this section, 22.7 with everything, 18.5 without retrieval, 12.4 without retrieval and overlap. MERLOT data: each step toward more diverse video adds points; the image-caption row is far below. MERLOT losses: removing the contrastive loss is the only large drop. Epochs: the curve still rises at 40.

Three designs side by side

text pathvideo pathwhere the modalities meetfrozen
Figure 6. The three designs, bottom to top. VideoBERT: frozen S3D, then k-means into discrete words, then one BERT-large over words and visual words. VideoCLIP: frozen S3D, an MLP, and two separate transformers that meet only in a dot product. MERLOT: an image encoder trained from scratch and a joint transformer, with a contrastive loss on the image encoder output. Sizes are from the papers; the drawing is not to scale.
VideoBERT (2019)VideoCLIP (2021)MERLOT (2021)
Visual inputS3D (Kinetics), 1 token per 1.5 s, quantized to 20,736 wordsS3D (HowTo100M, MIL-NCE), 1 token per s, 512-d, frozenone frame per segment, ResNet-50 + ViT trained from scratch
Fusionone stream, BERT-largenone: dual encoder (6-layer video, 12-layer text)joint encoder, RoBERTa-base
Lossesmasked text, masked video words, alignment classificationsymmetric InfoNCE, overlapped positives, retrieved hard negativescontrastive frame–transcript, attention-masked LM, temporal reordering
Pretraining data312K cooking videos, 23,186 hHowTo100M, 1.1M videosYT-Temporal-180M, 6M videos, 180M segments
Compute4 Cloud TPUs, about 2 days8 V100 GPUs, 25 epochs, about 1 dayTPU v3-1024, 40 epochs, 30 hours
Typical usezero-shot verb/noun by template, features for captioningzero-shot retrieval, VideoQA, segmentation, localizationfine-tuned video QA and VCR, zero-shot ordering

All entries are from the three papers [Certain]. Note how the training cost moved: VideoBERT and VideoCLIP pay for vision only once, by freezing a pretrained 3D CNN and caching its features; MERLOT pays much more compute (a TPU v3-1024 pod) to train vision end to end, and wins on reasoning tasks. The question “frozen features or end-to-end?” returns in every Video-LLM.

The algorithms and the code

Algorithm 1: VideoBERT data and one training step
  1. once: sample frames at 20 fps; cut 30-frame (1.5 s) windows; S3D \(\rightarrow \mathbf{f} \in \R^{1024}\) per window // frozen, cached
  2. once: hierarchical k-means with \(k = 12\), \(d = 4\) on all \(\mathbf{f}\); token \(\leftarrow\) path id, formula (175.2) // 20,736 visual words
  3. group video tokens by ASR sentence time stamps (or 16 per segment if there is no ASR)
  4. with some probability, concatenate neighboring ASR sentences; subsample video tokens with a random step in \(\{1, \dots, 5\}\)
  5. build [CLS] text [>] video [SEP]; for alignment, replace the video sentence with one from another time with probability 0.5 // label c
  6. mask tokens; loss \(\leftarrow\) weighted sum of text MLM, video MLM and alignment BCE, formulas (175.3) and (175.4)
  7. Adam step, learning rate \(10^{-5}\), linear decay
Algorithm 2: one VideoCLIP epoch with retrieval-augmented batches
  1. input: videos \(\mathcal{V}\) with cached S3D tokens and ASR; model \(M\); \(k = 32\) videos per batch, 16 pairs per video
  2. for each video \(V\): \(\mathbf{z}_V \leftarrow\) mean of \(\mathbf{z}_v\) and \(\mathbf{z}_t\) over its clip pairs, formula (175.9)
  3. build a dense index (FAISS) over all \(\mathbf{z}_V\)
  4. for each batch: pick a random seed video; sample \(k\) videos from its \(2k\) nearest neighbors
  5. for each video in the cluster, 16 times: sample a text clip \([s, e]\); center \(c \sim \mathcal{U}(s, e)\); length \(L \sim \mathcal{U}(3, 32)\) s; video clip \([c - L/2, c + L/2]\), formula (175.8)
  6. encode, average-pool, loss \(\leftarrow\) symmetric InfoNCE over the 512 pairs, formula (175.7) // negatives from similar videos
  7. Adam step (betas 0.9, 0.98, learning rate \(5 \times 10^{-5}\), warm-up 1,000 steps, gradient clip 2.0) [Certain]
Algorithm 3: one MERLOT training step
  1. input: a batch of sequences, each \(N = 16\) segments of (middle frame, 32 BPE tokens)
  2. image encoder on every frame \(\rightarrow\) grid features and CLS vectors \(\mathbf{a}_i\)
  3. language-only pass over the whole 512-token transcript \(\rightarrow\) segment CLS vectors \(\mathbf{b}_i\); \(\mathcal{L}_{\text{con}}\) with \(\tau = 0.05\), formula (175.11)
  4. mask 20% of words, half at random and half among the most-attended tokens of the language-only pass, then extend into spans
  5. with probability 0.4: pick \(i \in \{2, \dots, N\}\); give \(i\) random frames [image_unk] position embeddings
  6. joint encoder on 4 groups of 4 segments; \(\mathcal{L}_{\text{MLM}}\) at masked words; \(\mathcal{L}_{\text{order}}\) over all frame pairs, formula (175.12)
  7. loss \(\leftarrow 0.25\, \mathcal{L}_{\text{con}} + \mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{order}}\), formula (175.13); backward; optimizer step

The PyTorch version of the core pieces, with toy sizes: hierarchical k-means tokenization (175.2), a VideoBERT input with one masked word and one masked visual word (175.3), VideoCLIP’s overlapped clip sampling (175.8) and symmetric InfoNCE over average-pooled towers (175.6, 175.7), and MERLOT’s pairwise ordering loss (175.12).

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# ---------- 1. VideoBERT: hierarchical k-means turns clip features into visual words ----------
def kmeans(x, k, iters=10):
    c = x[torch.randperm(len(x))[:k]].clone()
    for _ in range(iters):
        a = torch.cdist(x, c).argmin(1)
        for j in range(k):
            if (a == j).any(): c[j] = x[a == j].mean(0)
    return c

def build_tree(x, k, d):                       # formula (175.2): k clusters per level, d levels
    if d == 0 or len(x) < k: return None
    c = kmeans(x, k); a = torch.cdist(x, c).argmin(1)
    return (c, [build_tree(x[a == j], k, d - 1) for j in range(k)])

def tokenize(f, tree, k, d):                   # walk down the tree; id = sum_l c_l * k^(d-l)
    tok, node = 0, tree
    for level in range(d):
        j = 0 if node is None else torch.cdist(f[None], node[0]).argmin().item()
        tok = tok * k + j
        node = None if node is None else node[1][j]
    return tok

FEAT, K, D = 16, 3, 2                          # VideoBERT: 1024-d S3D features, k = 12, d = 4
clips = torch.randn(600, FEAT)                 # stand-in for one S3D vector per 1.5 s clip
tree = build_tree(clips, K, D)
video_words = [tokenize(f, tree, K, D) for f in clips[:8]]
print("vocab", K ** D, "first visual words", video_words)

# ---------- 2. VideoBERT input: [CLS] text [>] video [SEP], masked prediction ----------
TXT_V = 50                                     # text vocab; visual words come after it
CLS, SEP, GT, MASK = 0, 1, 2, 3
text = torch.tensor([CLS, 10, 11, 12, 13, GT])
video = torch.tensor(video_words) + TXT_V
seq = torch.cat([text, video, torch.tensor([SEP])])
enc = nn.TransformerEncoder(nn.TransformerEncoderLayer(32, 4, 64, 0.0, batch_first=True), 2)
emb, head = nn.Embedding(TXT_V + K ** D, 32), nn.Linear(32, TXT_V + K ** D)
mask = torch.zeros_like(seq, dtype=torch.bool); mask[[2, 8]] = True     # one word, one visual word
h = enc(emb(seq.masked_fill(mask, MASK))[None])[0]
print("seq len", len(seq), "masked loss", round(F.cross_entropy(head(h[mask]), seq[mask]).item(), 3))

# ---------- 3. VideoCLIP: overlapped clips + symmetric InfoNCE over average-pooled tokens ----------
def overlapped_clip(s, e, g):                  # formula (175.8)
    c = s + (e - s) * torch.rand(1, generator=g).item()
    L = 3 + 29 * torch.rand(1, generator=g).item()
    return max(0.0, c - L / 2), c + L / 2

g = torch.Generator().manual_seed(1)
print("clip for text span [10, 14] s:", tuple(round(t, 1) for t in overlapped_clip(10, 14, g)))

class Tower(nn.Module):
    def __init__(s, din):
        super().__init__()
        s.proj = nn.Sequential(nn.Linear(din, 32), nn.GELU())           # the MLP after frozen S3D
        s.enc = nn.TransformerEncoder(nn.TransformerEncoderLayer(32, 4, 64, 0.0, batch_first=True), 2)
    def forward(s, x): return s.enc(s.proj(x)).mean(1)                  # AvgPool, not [CLS]

def info_nce(zv, zt, tau=1.0):                 # formulas (175.6) and (175.7)
    S = zv @ zt.T / tau
    y = torch.arange(len(S))
    return F.cross_entropy(S, y) + F.cross_entropy(S.T, y)

vt, tt = Tower(FEAT), Tower(8)
B = 6
v_tok = torch.randn(B, 10, FEAT)               # 10 one-second S3D tokens per clip (frozen)
t_tok = torch.randn(B, 20, 8)                  # 20 text-token embeddings per clip
print("VideoCLIP loss", round(info_nce(vt(v_tok), tt(t_tok)).item(), 3))

# ---------- 4. MERLOT: temporal reordering as pairwise "is i before j?" ----------
N, H = 4, 32
frame_h = torch.randn(N, H)                    # CLS state of each frame from the joint encoder
order_mlp = nn.Sequential(nn.Linear(2 * H, H), nn.ReLU(), nn.Linear(H, 2))
true_t = torch.tensor([2, 0, 3, 1])            # real time step of each (scrambled) frame
i, j = torch.triu_indices(N, N, 1)             # all N(N-1)/2 pairs
logits = order_mlp(torch.cat([frame_h[i], frame_h[j]], -1))
labels = (true_t[i] > true_t[j]).long()        # 0: t_i before t_j, 1: t_i after t_j
print("pairs", len(labels), "order loss", round(F.cross_entropy(logits, labels).item(), 3))

Output with PyTorch 2.8 on a CPU (the script is deterministic; it gave the same output on every run):

vocab 9 first visual words [0, 0, 2, 5, 1, 7, 4, 0]
seq len 15 masked loss 3.968
clip for text span [10, 14] s: (7.5, 18.6)
VideoCLIP loss 4.909
pairs 6 order loss 0.698

Things to notice. The tokenizer gives ids in \([0, k^d)\), and two of the first eight clips share word 0: discrete tokens repeat. The masked loss is near \(\ln 59 \approx 4.08\) (59 tokens in the vocabulary), as expected for an untrained model. The sampled clip (7.5 s to 18.6 s) contains the whole text span [10, 14] plus several seconds on each side. The InfoNCE loss is above \(2 \ln 6 = 3.58\) (the value for uniform guessing): with \(\tau = 1\) and unnormalized, untrained vectors, the dot products are large and random, so the model is confidently wrong. Two fixes are standard: normalize the vectors, or learn \(\tau\). The ordering loss is near \(\ln 2 = 0.693\): a coin flip, as it should be before training.

Lessons kept by modern Video-LLMs

Video-LLMs (session 205) look very different: a ViT reads a few frames, a connector turns them into tokens, and an LLM writes the answer. But much of what these three papers found is still inside them.

  • Speech is free supervision. HowTo100M, YT-Temporal-180M and their successors made ASR the standard source of video–text pairs. MERLOT Reserve (2022) went further and learned from the audio itself, not only the transcript [Certain]. Today, ASR transcripts are also given to Video-LLMs as extra input text, and used to write or check synthetic captions [Likely]. The noise found here (low relevance, fillers, raw ASR without punctuation) is why those pipelines clean and re-caption the text [Likely].
  • Time alignment must be explicit. VideoBERT grouped tokens by ASR time stamps; VideoCLIP sampled overlapping windows; MERLOT gave each segment its own position embedding and trained ordering. Qwen2.5-VL describes “absolute time encoding” so the model can localize events to the second [Certain]. Many Video-LLMs also write time stamps as text next to frame tokens [Likely]. See session 206.
  • Cheap visual features first. VideoBERT and VideoCLIP froze a 3D CNN and cached its features; MERLOT used one frame per segment and 5 to 7 frames at fine-tuning time. ClipBERT (2021) made the same point: a few sparsely sampled frames are often enough [Certain]. Video-LLMs sample a few frames and compress tokens for the same reason (session 183).
  • Two-stage retrieval. A dual encoder like VideoCLIP is the fast first stage of text-to-video search and Video-RAG; a heavier fusion model or a Video-LLM reranks or answers on the top few (session 208). Hard negatives, from retrieval or clustering, are still how these retrievers are trained (session 60).
  • Data diversity matters more than size alone. MERLOT’s HowTo100M-sized subset of diverse video beat HowTo100M itself by 6.5 VCR points [Certain]. Instructional videos are a narrow world: VideoCLIP’s MSR-VTT zero-shot gap shows the same thing.

And what faded: discrete k-means visual words (continuous features won for understanding), and image encoders trained from scratch on video (most later models start from CLIP or SigLIP image encoders, as CLIP4Clip did [Likely]).

Where this is used

TaskWhich idea from this sessionHow it looks today
Text-to-video retrievalVideoCLIP dual encoder, symmetric InfoNCE, hard negatives from clustersCLIP-based video encoders, a vector index over clips, then reranking (session 208)
Zero-shot video classification and action recognitionVideoBERT template filling; VideoCLIP label embeddingsText prompts of class names against video embeddings, or a Video-LLM asked to name the action
Key-moment detection, action step localizationPer-second token scores against step texts (175.10), with an “Outside” thresholdMoment retrieval with time stamps; Video-LLMs that output start and end times
Action segmentationVideoCLIP token-level similarity to 778 COIN labelsSame, with stronger frame encoders and temporal smoothing
Video QA, multiple choiceVideoCLIP answer ranking; MERLOT fine-tuned [CLS] headVideo-LLMs generate the answer (session 210)
Video-RAGASR text as a searchable stream aligned with clips; dual-encoder clip retrievalIndex both transcripts and clip embeddings; retrieve, then give the clips and text to a Video-LLM
Video fingerprinting and near-duplicatesDiscrete codes per clip (VideoBERT-style quantization) make sequences easy to hash and compare [Likely]Learned compact descriptors per segment and sequence matching (session 209)
Procedure understanding, “what comes next”VideoBERT video MLM (forecasting future tokens); MERLOT orderingTemporal reasoning benchmarks and instruction-following assistants for how-to videos

Common mistakes

  • Trusting exact ASR alignment. The words often come before or after the action. In the toy of Figure 1, exact clips cover only 2.7% of the actions; overlapping clips of 16 s cover about 68%. Use overlap, neighboring sentences or a multiple-positive loss like MIL-NCE.
  • Only easy negatives. With random batches, the InfoNCE gradient \(1 - p^{+}\) is near 0 for most pairs (0.065 in Figure 3, versus 0.78 with a retrieved cluster). Build batches from similar videos, but sample from a slightly larger neighborhood (\(k\) from \(2k\)).
  • Hard negatives that are really positives. Two clips from the same video, or two videos of the same recipe, may both match a sentence. Clustering makes this more likely. Watch for false negatives, for example by excluding clips that overlap in time with the positive.
  • Random masking on speech. Speech has fillers and repeated words. Random masking hides “umm” and lets the model copy a repeated object name from the text. Mask the words that are likely to be visual (MERLOT’s attention masking), or mask whole spans.
  • Using \(\tau = 1\) with normalized vectors. Cosine similarities lie in \([-1, 1]\); at \(\tau = 1\) the softmax is almost flat (loss near \(\ln B\)). VideoCLIP’s \(\tau = 1.0\) is for its own setup; MERLOT, with normalized vectors, used \(\tau = 0.05\).
  • Too small a visual vocabulary, or a frozen one. One k-means word per 1.5 s loses detail, and it cannot improve during training. If you need discrete video tokens (for example for generation), learn the codebook.
  • Instructional video only. HowTo100M-only models drop sharply on generic video (MSR-VTT zero-shot R@1 10.4 versus 22.7 on YouCook2). Mix domains, as YT-Temporal-180M did.
  • Test videos inside pretraining data. YouCook II and MSR-VTT come from YouTube too. VideoBERT removed YouCook II videos from its pretraining set and VideoCLIP followed HowTo100M’s splits to avoid overlap [Certain]. Do the same.

Interview questions

How does VideoBERT turn a video into tokens?basic
Frames at 20 fps are cut into 30-frame (1.5 s) windows. A frozen S3D network pretrained on Kinetics gives one 1024-d vector per window. Hierarchical k-means with 12 clusters per level and 4 levels assigns each vector to one of \(12^4 = 20{,}736\) leaves; the leaf id is the visual word. These words are appended to BERT’s WordPiece vocabulary (embeddings initialized from the centroids and frozen), and the model reads sequences like [CLS] words [>] visual words [SEP].
Why are ASR transcripts called free labels, and why are they noisy?basic
They come with the video at no labeling cost: YouTube’s speech recognition gives words with time stamps for millions of videos (HowTo100M: 136M clips from 1.22M videos). They are noisy because (1) people do not talk and act at the same time, so the frames under a sentence often do not show what it says; (2) much speech is not visual (“subscribe”, “umm”); (3) ASR makes recognition errors and has no punctuation. Fixes: overlapping or longer windows, multiple-instance losses (MIL-NCE), filtering by visual content, and cleaning the text.
What is VideoBERT’s linguistic–visual alignment task, and what two tricks make it work?core
It replaces BERT’s next-sentence prediction: a binary classifier on [CLS] predicts whether a text sentence and a video sentence come from the same time. Because the time alignment is a weak signal of meaning, VideoBERT (1) randomly concatenates neighboring ASR sentences into one long sentence, so the right words are more likely inside the window, and (2) subsamples the video tokens with a random step from 1 to 5, so the model sees actions at different speeds and over longer spans.
How does VideoBERT do zero-shot action classification, and what are the limits?core
It appends the clip’s visual words to the template “now let me show you how to [MASK] the [MASK]” and reads the most likely verb and noun from the two masked slots. On YouCook II it reached 43.3% top-5 verb and 33.7% top-5 object accuracy without any YouCook II training, close to a supervised linear classifier on the same S3D features (46.9% and 30.9%). Limits: top-1 verb accuracy was only 3.2%, because the open vocabulary produces synonyms that do not match the fixed label set, and the result depends on the template wording.
Explain VideoCLIP’s overlapping positives. Why not exact alignment, and how does it differ from MIL-NCE?core
VideoCLIP samples a text clip first, picks a random center inside it, and grows a video clip of random length (3 to 32 s) around that center. The pair overlaps in time but has different boundaries, so the video often contains the action even when it happens after the words. Exact alignment (same start and end) gives many low-relevance positives and can even put the truly relevant clip among the negatives. MIL-NCE keeps a fixed short clip (3.2 s) and lets several nearby sentences act as candidate positives inside the loss. In VideoCLIP’s ablation, exact alignment without retrieval gave YouCook2 R@1 12.4, overlap without retrieval 18.5, MIL-NCE clips and loss 16.1, and the full model 22.7.
What is retrieval-augmented training in VideoCLIP, and why sample k videos from the 2k nearest?core
Once per epoch, each video gets one vector (the mean of its clip video and text vectors), all videos go into a FAISS index, and each batch is built from a cluster of similar videos: pick a random seed video and sample \(k = 32\) videos from its \(2k\) nearest neighbors. Then the in-batch negatives are about similar tasks, which keeps \(1 - p^{+}\) (the gradient on the positive) large. Sampling from \(2k\) instead of taking the top \(k\) makes the videos close to each other, not just to the seed, and adds some randomness; it was slightly better (R@1 22.7 vs 22.5). Removing retrieval cost 4.2 R@1 points.
Why does a hard negative give a larger gradient than an easy one? Show it with numbers.core
For one anchor, the InfoNCE loss is \(-\log p^{+}\) with \(p^{+} = \softmax(s)_{+}\), and its gradient with respect to the positive logit is \(-(1 - p^{+})\); each negative logit gets \(+p^{-}\). With \(\tau = 0.1\) and similarities (0.9, 0.3, 0.1), \(p^{+} = 0.9972\), so the gradient is 0.0028. With (0.9, 0.8, 0.7), \(p^{+} = 0.665\) and the gradient is 0.335, about 120 times more. Easy negatives produce almost no learning signal.
What are MERLOT’s three objectives, and which one mattered most?core
(1) Contrastive frame–transcript matching: the image encoder’s CLS vector against a text vector of the segment computed with the whole transcript as context, normalized, \(\tau = 0.05\), weight 0.25. (2) Masked language modeling in the joint encoder, 20% of tokens, with attention masking. (3) Temporal reordering: scramble 2 to N frames by giving them [image_unk] position embeddings, and predict the order of every frame pair with an MLP. The contrastive loss mattered most: without it, VCR fell from 75.2 to 57.5. Without the ordering loss VCR was 75.5 and TVQA+ 75.6 (versus 75.8), so ordering mostly serves as an interface for temporal questions.
What is attention masking, and why does random masking fail on speech?advanced
In captions, a masked concrete word usually cannot be guessed from text alone, so the model must look at the image. Speech breaks this: people repeat key object names and use many fillers. Random masking often hides fillers (“umm”), which teach nothing, or hides one copy of a repeated word, which the model can copy from the text. MERLOT masks a random token half of the time and, the other half, one of the 20% most-attended tokens from its language-only encoder (these tend to be visual words), then extends to spans. It helped only with longer context: with 4 segments VCR went from 74.1 to 75.2; with 1 segment it did not help.
MERLOT trains its image encoder from scratch with grid features. Why not use a detector like the image VLMs of 2019?advanced
Cost and scale. A Faster R-CNN needs about 2 TFLOPs per image; MERLOT’s ResNet-50 + ViT encoder at 192 × 352 needs 40 GFLOPs, 2% of that, which matters when training on 180M frames for 40 epochs. A detector is also frozen and limited to its classes; a grid encoder trained end to end can learn whatever the losses need. The price is that the encoder must learn vision from scratch, which is why the contrastive loss is essential. MERLOT still beat detector-based models on VCR (80.6 vs 75.0 for UNITER at base size), using drawn-on boxes for the referred people (79.4 vs 74.8 without boxes in the ablation).
Design a zero-shot key-moment detector for how-to videos using these ideas.advanced
Use a VideoCLIP-style dual encoder: one embedding per second of video (token-level outputs, which is why average pooling during training helps) and one embedding per step description. Score every second against every step, \(\mathbf{h}_u \cdot \mathbf{z}_t\). Assign a step only if the best score is above a threshold; without validation data, set the threshold from the labels themselves, \(\gamma = \max_{t \ne t'} \mathbf{z}_t \cdot \mathbf{z}_{t'}\) (formula 175.10); otherwise output “background”. Smooth the per-second labels over time and merge runs into moments. For long videos use sliding windows (VideoCLIP used 32 s windows with a 16 s step on COIN and averaged overlapping logits). Optionally rerank the top moments with a heavier model or a Video-LLM.
VideoCLIP is a dual encoder, VideoBERT and MERLOT are fusion encoders. When would you choose each for a video product?advanced
A dual encoder embeds each clip and each query once, so search over millions of clips costs one query encoding plus a nearest-neighbor lookup; it is the right first stage for retrieval, Video-RAG and moment search. A fusion encoder lets frames and words attend to each other in every layer, which helps reasoning (MERLOT on VCR and TVQA), but each (video, text) pair needs its own forward pass, so it can only rerank a short list or answer about one video. In practice: dual encoder for recall, fusion model or Video-LLM for the top few.

Exercises

  1. VideoBERT used \(k = 12\) and \(d = 4\). How many visual words would \(k = 8, d = 5\) give? How many centroid comparisons does assigning one clip need with the tree, and with a flat k-means over the same vocabulary?
  2. A 12-minute cooking video is processed by VideoBERT. How many visual words does it produce? With a random subsampling step of 3, how many are left?
  3. VideoCLIP samples a text clip \([20, 24]\) s, center \(c = 23\) and length \(L = 10\) s. What is the video clip? If the action is visible during \([26, 32]\), what are coverage and purity (175.1)?
  4. Compute the InfoNCE video-to-text loss for one row with similarities \((0.6, 0.5, 0.2)\) (the first is the positive), at \(\tau = 0.1\) and at \(\tau = 1\). Which temperature gives the larger gradient \(1 - p^{+}\)?
  5. MERLOT gives the joint encoder 4 segments. How many pairwise ordering labels does one group produce? How many would one sequence of \(N = 16\) segments produce if all pairs were used at once?
  6. A model orders a 5-image story as (true ranks of the images in predicted order) \((1, 0, 2, 4, 3)\). Compute Spearman \(\rho\) and pairwise accuracy.
Answerscheck
1. \(8^5 = 32{,}768\) words. Tree: \(8 \times 5 = 40\) comparisons; flat: 32,768. 2. \(720 / 1.5 = 480\) visual words; with step 3, \(480 / 3 = 160\). 3. Clip \([18, 28]\). Intersection with \([26, 32]\) is \([26, 28]\), 2 s. Coverage \(2 / 6 = 0.333\); purity \(2 / 10 = 0.2\). 4. At \(\tau = 0.1\): logits (6, 5, 2), \(p^{+} = e^{6} / (e^{6} + e^{5} + e^{2}) = 0.7214\), loss 0.327, gradient \(1 - p^{+} = 0.2786\). At \(\tau = 1\): logits (0.6, 0.5, 0.2), \(p^{+} = 0.3883\), loss 0.946, gradient 0.6117. Here \(\tau = 1\) gives the larger gradient on this pair, but it also stays large for easy pairs, so the model cannot tell easy from hard; small \(\tau\) focuses the gradient on the hard ones. 5. \(4 \cdot 3 / 2 = 6\) per group, 24 for the 4 groups; all 16 at once: \(16 \cdot 15 / 2 = 120\). 6. Differences between predicted position and true rank: \((0 - 1, 1 - 0, 2 - 2, 3 - 4, 4 - 3) = (-1, 1, 0, -1, 1)\); \(\sum d^2 = 4\); \(\rho = 1 - 6 \cdot 4 / (5 \cdot 24) = 0.8\). Wrong pairs: (1, 0) and (4, 3), so 8 of 10 are right: pairwise accuracy 0.8.

Key takeaways

  • Narrated videos give free text with time stamps, but speech and action are often not at the same time; exact alignment gives many low-relevance pairs (175.1).
  • VideoBERT quantizes 1.5-second S3D features into \(12^4 = 20{,}736\) visual words with hierarchical k-means (175.2) and trains BERT-large with masked text, masked video and alignment losses (175.3, 175.4); it can classify zero-shot by filling a template (175.5).
  • VideoCLIP is a dual encoder with average pooling (175.6) and symmetric InfoNCE (175.7). Overlapping positives (175.8) and retrieved clusters of similar videos as negatives (175.9) raised zero-shot YouCook2 R@1 from 12.4 to 22.7.
  • Hard negatives matter because the InfoNCE gradient on a positive is \(1 - p^{+}\), which is near zero when negatives are easy.
  • MERLOT trains vision from scratch on 180M frame–transcript segments with contrastive matching (175.11), attention-masked LM and temporal reordering (175.12, 175.13). The contrastive loss was essential; diverse video beat instructional video.
  • Modern Video-LLMs keep the lessons: speech as supervision, explicit time alignment, cheap or cached frame features, and dual encoders as the fast first stage of video search.

Key formulas

\[ \text{token}(\mathbf{f}) = \sum_{\ell=1}^{d} c_\ell\, k^{\,d-\ell}, \quad |\text{vocab}| = k^d, \quad \operatorname{NCE}(\mathbf{z}_v, \mathbf{z}_t) = \frac{e^{\mathbf{z}_v \cdot \mathbf{z}_t^{+} / \tau}}{\sum_{\mathbf{z}} e^{\mathbf{z}_v \cdot \mathbf{z} / \tau}} \]
\[ v = [c - \tfrac{L}{2}, c + \tfrac{L}{2}],\ c \sim \mathcal{U}(s, e), \quad \frac{\partial (-\log p^{+})}{\partial s^{+}} = -(1 - p^{+}), \quad \mathcal{L}_{\text{MERLOT}} = 0.25\,\mathcal{L}_{\text{con}} + \mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{order}} \]

References for this session

  1. Sun et al. (2019). VideoBERT: A Joint Model for Video and Language Representation Learning
  2. Xu et al. (2021). VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
  3. Zellers et al. (2021). MERLOT: Multimodal Neural Script Knowledge Models
  4. Sun et al. (2019). VideoBERT: A Joint Model for Video and Language Representation Learning
  5. Xu et al. (2021). VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
  6. Zellers et al. (2021). MERLOT: Multimodal Neural Script Knowledge Models
  7. Miech, Zhukov, Alayrac, Tapaswi, Laptev & Sivic (2019). HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
  8. Miech, Alayrac, Smaira, Laptev, Sivic & Zisserman (2020). End-to-End Learning of Visual Representations from Uncurated Instructional Videos
  9. Xie, Sun, Huang, Tu & Murphy (2018). Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification
  10. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  11. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  12. van den Oord et al. (2018). Representation Learning with Contrastive Predictive Coding
  13. Zellers et al. (2022). MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound
  14. Lei et al. (2021). Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
  15. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  16. Bai et al. (2025). Qwen2.5-VL Technical Report