Early video–language pretraining: VideoBERT, VideoCLIP, MERLOT
Videos on YouTube come with free text: the words people say, turned into a transcript by speech recognition. Three models from 2019 to 2021 show three ways to learn from that text. VideoBERT turns video into discrete “visual words” and trains BERT on them, VideoCLIP learns a contrastive video–text space with better positives and harder negatives, and MERLOT learns what happens in a video and in what order.
What you will learn
- Why narrated videos give free but noisy labels, and why speech and action are often not at the same time
- How VideoBERT turns 1.5-second clips into 20,736 visual words with hierarchical k-means, and trains BERT-large on words plus visual words
- How VideoCLIP builds positive pairs from overlapping (not exactly aligned) clips and hard negatives from retrieved clusters of similar videos
- The symmetric InfoNCE loss, why hard negatives give a larger gradient, and how VideoCLIP does zero-shot retrieval, VideoQA and action segmentation
- How MERLOT trains an image encoder from scratch on 180M frame–transcript segments with three losses: contrastive matching, attention-masked language modeling and temporal reordering
- What the ablations measured: data size, overlap and retrieval, data diversity, and which loss mattered most
- Which lessons modern Video-LLMs kept: speech as free supervision, explicit time alignment, frozen or cheap frame features, and two-stage retrieval
Why this matters
Labeled video is very expensive. To build a dataset like YouCook II, people watched 2,000 cooking videos and wrote a caption for every step by hand [Certain]. But YouTube has millions of videos where a person talks while doing something: “now crack two eggs into the bowl”. Speech recognition (ASR, automatic speech recognition) turns that talk into text with time stamps, for free. This session is about the first models that learned from that free text: VideoBERT (2019), VideoCLIP (2021) and MERLOT (2021).
The three papers are three different answers to one question: how do we connect a stream of frames with a stream of spoken words?
- VideoBERT makes video look like text. It cuts the video into 1.5-second clips, turns each clip into one of 20,736 discrete “visual words”, and trains a BERT model on sentences made of real words and visual words [Certain].
- VideoCLIP keeps video and text in two separate encoders, like CLIP (session 166), and trains them with a contrastive loss. Its new ideas are about which pairs to use: positives that overlap in time instead of being exactly aligned, and negatives taken from similar videos [Certain].
- MERLOT learns from frames and transcripts of 6 million videos with a joint encoder. Besides matching frames with words, it must put shuffled frames back in the right order, so it learns what happens and in what order [Certain].
Why study them now, when Video-LLMs (session 205) read frames with a ViT and answer in free text? Because the problems they found are still the problems. Speech is still the cheapest supervision for video. Speech and action are still not at the same time. Contrastive video–text encoders are still the first stage of text-to-video search and Video-RAG (session 208, session 210). And “in what order did things happen?” is still a weak point of many video models. This session sits between image–text pretraining (sessions 166 to 174) and the video track (Track 8).
Speech as free labels, and why they are noisy
A narrated video gives us two streams on one time axis: frames, and ASR words with start and end times. The simplest way to make training pairs is exact alignment: take one ASR sentence and the video between its start and end times, and call them a positive pair. HowTo100M (Miech et al., 2019) did this at scale: 136 million clips from 1.22 million narrated instructional videos, covering more than 23,000 tasks [Certain].
The problem: people do not speak and act at the same time. VideoCLIP’s paper gives the example “I am going to show you how to cook fried rice”: the frames under that sentence show a person speaking, and the rice in the wok appears much later [Certain]. Often the speaker first says what they will do and then does it, or does it and then explains. Some sentences (“don’t forget to subscribe”) have no visual content at all. So exactly aligned pairs are often low relevance: the text does not describe the frames under it.
Figure 1 makes this concrete with a toy video of 120 seconds and 8 spoken sentences. For each sentence we know when the matching action is really visible. We measure two things for a candidate video clip \([x_1, x_2]\) and the action interval \([a, b]\):
- Read the symbols
- \([x_1, x_2]\)
- the video clip used as the positive for one sentence, in seconds
- \([a, b]\)
- the time when the action the sentence talks about is really on screen
- \(\lvert \cdot \rvert\)
- the length of an interval; the intersection \(\cap\) is the part the two intervals share
- coverage
- how much of the action the clip shows (1 means all of it)
- purity
- how much of the clip is the action (1 means nothing else is in the clip)
How to read Figure 1
With exact alignment, the clip is the sentence itself. Mean coverage is only about 2.7%: the actions mostly happen just after the words. Switch to overlapping clips with \(L = 16\) s: mean coverage jumps to about 68%, and 5 of the 8 sentences now see at least half of their action. Purity also goes up at first (from about 10% to about 27%), but if you push \(L\) to 32 s, purity falls again (about 20%): long clips contain other steps too. VideoCLIP draws \(L\) at random between 3 and 32 s, so it trains on both short, precise pairs and long, forgiving pairs. Row 1 (“Today we make fried rice”) is the paper’s example: its action happens 30 seconds later, and no clip length here reaches it.
There are three families of fixes, and the three papers use all of them. (1) Make the pairing looser in time: concatenate neighboring sentences (VideoBERT), use overlapping clips of random length (VideoCLIP), or give the model a whole video context (MERLOT). (2) Make the loss tolerant: MIL-NCE (Miech et al., 2020) treats several nearby sentences as candidate positives for one short clip of 3.2 seconds [Certain], so only one of them needs to fit. (3) Pick better data: MERLOT filters out videos whose content is not visual (for example, video game commentaries) and adds punctuation to the raw ASR [Certain].
VideoBERT: video as a second language
VideoBERT (Sun et al., 2019) asked: what is the smallest change to BERT that lets it read video? BERT reads discrete tokens from a vocabulary. So VideoBERT turns video into discrete tokens too, and then trains a normal BERT on mixed sequences [Certain].
Step 1: clips become feature vectors
Frames are sampled at 20 frames per second and cut into non-overlapping windows of 30 frames, so one window is 1.5 seconds [Certain]. Each window goes through S3D (a 3D convolutional network that adds separable temporal convolutions to an Inception backbone, pretrained on the Kinetics action dataset). The activations before its final classifier are average-pooled into one 1024-dimensional vector per clip [Certain]. The S3D network is frozen: it is only a feature extractor.
Step 2: feature vectors become visual words
A 1024-d vector is continuous. To get a vocabulary, VideoBERT runs hierarchical k-means: k-means with \(k\) clusters on all clip vectors, then k-means with \(k\) clusters again inside each cluster, and so on for \(d\) levels. The authors chose \(d = 4\) and \(k = 12\) by looking at the clusters, which gives \(12^4 = 20{,}736\) visual words [Certain]. A clip is assigned by walking down the tree:
- Read the symbols
- \(\mathbf{f} \in \R^{1024}\)
- the S3D feature of one 1.5-second clip
- \(k, d\)
- clusters per level and number of levels; the vocabulary has \(k^d\) words
- \(\mu_{c_1 \dots c_{\ell-1} j}\)
- the centroid of child \(j\) inside the cell chosen at the previous levels
- \(c_\ell\)
- the child chosen at level \(\ell\); the token id writes \(c_1, \dots, c_d\) as a number in base \(k\)
Why hierarchical and not one flat k-means with 20,736 centers? A flat assignment compares each vector with all 20,736 centroids; the tree compares it with only \(k \cdot d = 48\). Training is cheaper too, since each k-means run has only 12 centers. The price: a point near a cell border at level 1 can end up in a worse leaf than flat k-means would choose. Figure 2 lets you build the tree on 2D points.
How to read Figure 2
The vocabulary size is \(k^d\): with \(k = 3, d = 2\) it is 9; with \(k = 6, d = 3\) it is 216. The readout shows the mean squared quantization error (the average squared distance from a point to its leaf centroid): it falls as the vocabulary grows, but each word then covers fewer points, so it is seen less often in training. VideoBERT’s \(12^4 = 20{,}736\) is a trade-off of the same kind, chosen by eye. Watch the visual sentence: nearby clips often get the same word, so real video sentences contain many repeats.
Step 3: mixed sentences and three objectives
Text comes from YouTube ASR, split into sentences by an LSTM language model that adds punctuation, and tokenized into BERT’s 30,000 WordPieces [Certain]. Video tokens that fall inside the time stamps of an ASR sentence form one “video sentence”; when there is no ASR, every 16 video tokens form a segment [Certain]. A new token [>] joins a text sentence and a video sentence [Certain]:
[CLS] orange chicken with [MASK] sauce [>] v01 [MASK] v08 v72 [SEP]
The model is BERT-large (24 layers, width 1024, 16 heads), initialized from the text checkpoint. The 20,736 visual words are appended to the word embedding table and initialized with the S3D cluster centroids, and the input embeddings are frozen during pretraining [Certain]. Training uses three regimes with a weighted sum of losses [Certain]: masked token prediction on text only, masked token prediction on video only, and on video–text pairs a linguistic–visual alignment classifier on [CLS] that predicts whether the sentence and the video sentence are aligned in time:
- Read the symbols
- \(\mathcal{M}\)
- the masked positions; for video-only data they are visual words, so the model must predict one of 20,736 classes
- \(\mathbf{x}_{\setminus \mathcal{M}}\)
- the sequence with those positions replaced by [MASK]
- \(\lambda_{\text{t}}, \lambda_{\text{v}}, \lambda_{\text{a}}\)
- loss weights (the paper says “weighted sum” but does not give the values)
- \(c\)
- 1 if the text and video sentences come from the same time, 0 if not
- \(\mathbf{h}_{\text{CLS}}, \mathbf{u}, \sigma\)
- the [CLS] output, a learned vector, and the sigmoid
The masked video loss is a “language model for video”: to fill a masked clip, the model must know what usually comes before and after it in a recipe. The alignment loss is the cross-modal part. The authors note it is noisy, for the reason of Figure 1, and add two tricks [Certain]: they randomly concatenate neighboring sentences into one long sentence, so the right words are more likely to be inside the window; and they subsample video tokens at a random rate of 1 to 5 steps, so the model sees actions at different speeds and over longer time spans.
The pretraining set has 312K cooking videos with a total of 23,186 hours, about 966 days [Certain]. At one visual word per 1.5 s, that is \(23{,}186 \times 3{,}600 / 1.5 \approx 55.6\) million visual words. A 5-minute video becomes \(300 / 1.5 = 200\) visual words. Only 180K videos had ASR and about 120K had English ASR; all videos are used for the video-only loss, only English ones for the text losses [Certain].
The new embedding rows: \(20{,}736 \times 1024 \approx 21.2\) million numbers, frozen. Training: 4 Cloud TPUs, batch 128, 0.5 million iterations (about 8 epochs), Adam with learning rate \(10^{-5}\), about 2 days [Certain]. That is \(0.5 \times 10^{6} \times 128 = 64\) million training sequences.
Zero-shot classification by filling a template
Once trained, VideoBERT can be used without any fine-tuning. Give it the visual words of a clip and the fixed sentence “now let me show you how to [MASK] the [MASK]”; read the verb from the first slot and the noun from the second [Certain]:
On YouCook II validation (labels made from the 45 most common verbs and 100 most common nouns in the captions), VideoBERT reached 43.3% top-5 verb accuracy and 33.7% top-5 object accuracy with no YouCook II training, close to a supervised linear classifier on the same S3D features (46.9% and 30.9%) [Certain]. Its top-1 verb accuracy was only 3.2% (supervised: 16.1%), partly because the vocabulary is open: the model may say “mix” when the label is “stir” [Certain]. Without the video, its language prior gets 6.9% top-5 verb accuracy [Certain], so the video really carries the signal. And accuracy grew with the number of pretraining videos with no sign of saturation: top-5 verb accuracy 15.5%, 15.7%, 24.5%, 43.3% for 10K, 50K, 100K and 300K videos [Certain] (Figure 5). For captioning, VideoBERT features plus S3D features beat the previous best on YouCook II on every metric, for example BLEU-4 4.33 versus 3.84 [Certain].
What did not survive: the discrete tokens. Quantizing a 1024-d feature into one of 20,736 words throws away detail (two different “stir” clips can share a word), and the k-means vocabulary is fixed before training, so the visual side cannot improve. Later models keep continuous features.
VideoCLIP: contrastive learning with better pairs
VideoCLIP (Xu et al., 2021) goes the other way: no fusion encoder, no discrete tokens. It is a dual encoder, like CLIP: one transformer for video, one for text, and similarity by dot product. Its contribution is in the pairs.
The two towers
The video side is a frozen S3D (this time pretrained on HowTo100M by MIL-NCE) that gives one 512-d token per second of video, at 30 fps [Certain]. A trainable one-layer MLP with GELU maps each token to 768-d, then a video transformer runs; the text goes through its own transformer. Both are initialized from BERT-base-uncased: the video side uses only the first 6 layers, the text side all 12 [Certain]. Each side is average-pooled into one vector:
- Read the symbols
- \(c_v\)
- a video clip of 3 to 32 seconds; up to 32 video tokens (plus [CLS] and [SEP], 34 in total) [Certain]
- \(\operatorname{stopgrad}\)
- no gradient flows into S3D, so its features can be computed once and stored
- \(\mathbf{x}_t\)
- BERT word embeddings of a text clip of 8 to 61 tokens (63 with [CLS] and [SEP]) [Certain]
- \(\mathbf{z}_v, \mathbf{z}_t\)
- one vector per clip, the mean over all output tokens
Why average pooling instead of [CLS]? So that every output token must carry meaning, which helps token-level tasks like action segmentation later. In the ablation, using [CLS] lowered zero-shot YouCook2 R@1 from 22.7 to 22.1 [Certain]. Why up to 61 text tokens for 32 video tokens? HowTo100M speech runs at about 2.4 tokens per second, while video tokens come at 1 per second [Certain].
The loss: symmetric InfoNCE
For a batch \(B\) of pairs, the loss pulls each video toward its own text and away from the other texts in the batch, and the same in the other direction:
- Read the symbols
- \(\mathbf{z}_t^{+}\)
- the positive: a text clip that overlaps in time with video clip \(v\)
- \(\{\mathbf{z}_t^{-}\}\)
- the negatives: all other text clips in the batch
- \(\tau\)
- the temperature; VideoCLIP sets \(\tau = 1.0\) [Certain]
- \(\operatorname{NCE}(\mathbf{z}_t, \mathbf{z}_v)\)
- the same with roles swapped: one text against all videos in the batch (used for text-to-video retrieval)
This is the same loss as CLIP’s; read session 166 for the batch-matrix view. Two things are new: how the positive is chosen (overlap), and how the batch is built (retrieval), so that the negatives are hard.
Positives: overlapping clips
To build one positive pair, VideoCLIP (i) samples a text clip first (a random video clip might have no speech near it), (ii) samples a time stamp inside that text clip as the center, and (iii) grows a video clip of random duration around it, up to about 32 seconds [Certain]:
The video and text clips share some time but not the same start and end. This is the overlapping mode of Figure 1: the positive usually contains the action even when it happens a few seconds after the words. 16 pairs are sampled per video and 32 videos form a batch of 512 [Certain].
Negatives: retrieval-augmented batches
With random videos in a batch, most negatives are easy: a cooking sentence against a car-repair clip. The loss quickly becomes small and teaches little. VideoCLIP builds batches from clusters of similar videos instead, so that negatives come from videos about the same kind of task. Once per epoch [Certain]:
- Read the symbols
- \(B_V\)
- all clip pairs of video \(V\); \(\mathbf{z}_V\) is the mean of their video and text vectors, one vector per video
- \(\operatorname{kNN}(\mathbf{z}_V, 2k)\)
- the \(2k\) nearest videos in a FAISS index over all \(\mathbf{z}_V\)
- \(k\)
- videos per batch, \(k = 32\)
Why sample \(k\) from the \(2k\) nearest and not take the \(k\) nearest? The goal is videos that are close to each other, not all close to one seed video; sampling from a slightly larger neighborhood gave better clusters in the ablation (R@1 22.7 versus 22.5 for plain kNN) [Certain].
Why do hard negatives help? Look at the gradient of the InfoNCE loss with respect to the positive score \(s^{+} = \mathbf{z}_v \cdot \mathbf{z}_t^{+} / \tau\): it is \(-(1 - p^{+})\), where \(p^{+}\) is the softmax probability of the positive. If all negatives are far, \(p^{+} \approx 1\) and the gradient is almost zero. Hard negatives keep \(p^{+}\) away from 1, so every pair still pushes.
Three video–text pairs, cosine similarities (rows are videos, columns texts), \(\tau = 0.1\). Easy batch: row 1 is \((0.9, 0.3, 0.1)\). The softmax of \((9, 3, 1)\) is \((0.9972, 0.0025, 0.0003)\), so the loss for this row is \(-\ln 0.9972 = 0.0028\) and the gradient on the positive score is \(-(1 - 0.9972) = -0.0028\).
Hard batch, same positive: row 1 is \((0.9, 0.8, 0.7)\). The softmax of \((9, 8, 7)\) is \((0.6652, 0.2447, 0.0900)\); loss \(-\ln 0.6652 = 0.408\), gradient \(-0.335\), about 120 times larger. Over the whole easy batch \(\left(\begin{smallmatrix} 0.9 & 0.3 & 0.1 \\ 0.2 & 0.8 & 0.4 \\ 0.1 & 0.5 & 0.7 \end{smallmatrix}\right)\) the mean video-to-text loss is 0.051; over the hard batch \(\left(\begin{smallmatrix} 0.9 & 0.8 & 0.7 \\ 0.8 & 0.85 & 0.75 \\ 0.7 & 0.75 & 0.8 \end{smallmatrix}\right)\) it is 0.589. The hard batch carries much more learning signal per pair.
How to read Figure 3
At \(\tau = 0.1\), a random batch has a mean negative similarity of about −0.19 and a loss of about 0.07: the model already “knows” the answer. The share of gradient that still flows, the mean of \(1 - p^{+}\), is only about 0.065. Switch to the retrieved cluster: negatives have similarity about 0.96, the loss rises to about 1.54, and the gradient share to about 0.78, roughly 12 times more. Top-1 accuracy inside the batch is still 6 of 6 in both cases: the hard batch is harder, not wrong. Raise \(\tau\) to 1.0 (VideoCLIP’s value): on cosine similarities, which lie in \([-1, 1]\), the logits are squeezed into a narrow range, so the random batch’s loss rises to about 1.04 and the cluster’s to about 1.76, close to \(\ln 6 = 1.79\) (uniform guessing). That is why models with normalized vectors usually use a small or learned \(\tau\).
What the ablations measured
On zero-shot text-to-video retrieval on YouCook2, VideoCLIP reached R@1 22.7, against 15.1 for MIL-NCE and 6.1 for HowTo100M’s model [Certain]. Removing retrieval augmentation lowered R@1 to 18.5; removing overlap as well (exactly aligned pairs) lowered it to 12.4 [Certain]. Using MIL-NCE’s clips and loss inside the same architecture gave 16.1 [Certain]. Sharing one transformer for video and text cost little (21.9) [Certain]. Figure 5 shows the whole table. The results also show the limit of instructional data: zero-shot on MSR-VTT, whose captions describe generic YouTube scenes, R@1 was only 10.4 (MIL-NCE: 9.9); after fine-tuning it was 30.9 [Certain]. The authors name domain shift from HowTo100M as the likely reason [Certain].
Zero-shot on more than retrieval
Because the text tower can embed any label, many video tasks become similarity searches [Certain]. Multiple-choice VideoQA: rank the candidate answers by \(\mathbf{z}_v \cdot \mathbf{z}_{\text{answer}}\) (73.9% zero-shot on MSR-VTT QA). Action step localization on CrossTask: softmax over \(\mathbf{h}_u \cdot \mathbf{z}_{\text{step}}\) for each video token \(u\). Action segmentation on COIN, with an “Outside” label for frames that match nothing:
Here \(\mathbf{h}_u\) is the output of video token \(u\) (one per second), \(\mathcal{L}\) the 778 COIN step labels, and \(\gamma\) a threshold taken from the labels themselves, since zero-shot means no validation data to tune it [Certain]. A token is assigned a step only if it matches that step better than any two labels match each other. Zero-shot frame accuracy was 58.9%, above every supervised method in the paper’s table except its own fine-tuned model (68.7%) and MIL-NCE (61.0%) [Certain]. This is exactly the setup of key-moment detection: score every second of video against a text query.
MERLOT: what happens, and in what order
MERLOT (Zellers et al., 2021) aims at something different: script knowledge, the common sense of how events follow each other (people sit down, then order, then eat). It is a joint encoder trained on a large and diverse video set, and it is tested on reasoning tasks (VCR, TVQA) instead of retrieval.
Data: YT-Temporal-180M
Starting from 27 million candidate videos (HowTo100M, VLOG lifestyle videos and YouTube suggestions for topics like “science”), the authors keep 6 million videos that have English ASR, are at most 20 minutes long, are not in “ungrounded” categories like video game commentary, and whose thumbnails are likely to show objects [Certain]. Each video is cut into segments of \(L = 32\) BPE tokens of transcript; each segment gets the frame from its middle time step. That gives 180 million segments [Certain].
Architecture
Unlike VideoBERT and VideoCLIP, MERLOT trains its image encoder from scratch with the rest of the model, with no supervised features [Certain]. It is a grid model, not a detector: a ResNet-50 (with its last block removed) followed by a 12-layer, 768-d ViT, on widescreen frames of 192 × 352 pixels, with 2 × 2 average pooling at the end. It costs 40 GFLOPs per frame, 2% of the 2 TFLOPs of a Faster R-CNN [Certain]. The output is a \(W/32 \times H/32\) grid plus CLS states. The joint encoder is a 12-layer, 768-d transformer initialized from RoBERTa-base; position embeddings differ per segment, so the model knows which frame and which words belong to which time step [Certain].
A frame of 352 × 192 pixels gives a grid of \(352/32 \times 192/32 = 11 \times 6 = 66\) cells. One segment in the joint encoder is then \(66\) grid cells \(+ 1\) image CLS \(+ 32\) word tokens \(= 99\) tokens, and 4 segments give \(4 \times 99 = 396\). The paper reports a joint sequence length of 396 [Certain]; this split of 396 into parts is our reconstruction [Likely].
Training examples have \(N = 16\) segments. The language-only pass for the contrastive loss reads all \(16 \times 32 = 512\) transcript tokens; the joint encoder gets 4 groups of 4 segments to save memory [Certain]. With 1,024 sequences per batch, that is \(1{,}024 \times 16 = 16{,}384\) segments per batch (“16k”), on a TPU v3-1024 pod, 40 epochs, 30 hours [Certain].
Three objectives
1. Contrastive frame–transcript matching. The image encoder must learn to see, and nothing else teaches it from scratch. So each frame’s CLS vector is matched with a text vector of its segment. The text vector comes from a language-only pass over the whole transcript (the segment CLS states), because a segment’s 32 words alone often do not describe its frame. Both are projected to 768-d and L2-normalized; negatives are all other frame–caption pairs in the batch, from the same video or not; \(\tau = 0.05\) [Certain]:
- Read the symbols
- \(\mathbf{a}_i\)
- the normalized image vector of frame \(i\) (from the image encoder’s CLS)
- \(\mathbf{b}_i\)
- the normalized text vector of segment \(i\), computed with the whole transcript as context
- \(M\)
- the number of segments in the batch (16,384)
- \(\tau = 0.05\)
- temperature: dot products in \([-1, 1]\) become logits in \([-20, 20]\)
2. Attention-masked language modeling. 20% of the words given to the joint encoder are replaced by [MASK], a random word or kept, and predicted with cross-entropy [Certain]. But speech is not a caption: people say “umm” and “yeah”, and repeat the names of key objects. Random masking often hides fillers, and when it hides “saw” the word may appear again two sentences later, so the model can copy it from the text without looking. MERLOT’s fix, attention masking: half of the time mask a random token, the other half mask one of the 20% of tokens that the language-only encoder attends to most, since those tend to be the visual words; then extend the mask by SpanBERT-style spans [Certain].
3. Temporal reordering. In 40% of the examples, pick \(i\) between 2 and \(N\) and scramble \(i\) frames: their segment position embeddings are replaced by special learned embeddings [image_unk_0], [image_unk_1], … that say nothing about time [Certain]. For every pair of frames, a two-layer MLP reads the two CLS states and predicts which came first [Certain]:
- Read the symbols
- \(\mathbf{h}_{t_i}\)
- the joint encoder’s CLS output for frame \(i\)
- \([\cdot\,;\cdot]\)
- concatenation of the two vectors; the MLP outputs two logits: “\(i\) before \(j\)” or “\(i\) after \(j\)”
- \(\mathbb{1}[t_i \lt t_j]\)
- the label: 1 if frame \(i\) really comes before frame \(j\)
The three are added, with the contrastive loss multiplied by 0.25 so that its gradients are about as large as the masked LM loss [Certain]; we assume weight 1 for the other two [Likely]:
How do we measure ordering? Zero-shot, on visual stories (SIND): 5 images with captions in the right order; the model must put the images in order. MERLOT uses its reordering head to find the most probable order; CLIP and UNITER, which only score image–caption pairs, use a best bipartite matching [Certain]. Two of the metrics, for a predicted order with ranks \(\hat r_i\) and true ranks \(r_i\):
MERLOT scored Spearman \(\rho = 0.733\) and pairwise accuracy 84.5%, against 0.609 and 78.7% for CLIP and 0.545 and 75.2% for UNITER [Certain]. Try the metrics in Figure 4.
How to read Figure 4
The start order “put tea bag, fill kettle, drink, boil water, pour” has \(\rho = 0.3\) and pairwise accuracy 6/10 = 0.6. Swap two neighbors and watch both metrics change; reach the true order and both are 1. Note the difference: Spearman punishes a frame that is far from its place much more (squared distance), pairwise accuracy only counts pairs. In the top row, frames with a dashed border lost their time stamp; the model must place them using the frames that kept theirs, and their content. With \(N = 5\), the loss always has \(5 \cdot 4 / 2 = 10\) pairwise labels.
What the ablations measured
The MERLOT ablations use VCR question answering (validation accuracy) and TVQA+ [Certain]. Figure 5 shows them. The most important points:
- The contrastive loss is essential. Without it, VCR fell from 75.2 to 57.5 and TVQA+ from 75.8 to 67.6. Without the ordering loss, VCR was even slightly higher (75.5) and TVQA+ slightly lower (75.6) [Certain]. The authors conclude that the ordering head is mainly an interface to ask the model about time [Certain].
- Diverse video beats instructional video. Same number of steps: Conceptual Captions + COCO (images) 58.9, HowTo100M 66.3, a HowTo100M-sized subset of YT-Temporal-180M 72.8, the full YT-Temporal-180M 75.2 [Certain].
- Clean ASR helps. Raw ASR instead of punctuated ASR: 72.8 instead of 75.2 [Certain].
- Attention masking helps only with longer context. With 1 segment it did not help (73.5 versus 73.8 on VCR); with 4 segments it did (75.2 versus 74.1) [Certain]. More context makes text-only shortcuts easier, and attention masking removes them.
- Longer training kept helping: VCR 75.2 after 5 epochs, 79.4 after 40, with no plateau [Certain].
After fine-tuning, MERLOT (base size) reached 80.6% on VCR Q→A, above ViLBERT (73.3), UNITER (75.0) and ERNIE-ViL (77.0), all of which use detector features from images [Certain]. It also set a new state of the art on 12 video QA tasks, for example TVQA 78.7 and TGIF-Action 94.0 [Certain], with 5 to 7 frames per clip [Certain]. Its successor MERLOT Reserve (2022) added audio and trained on 20 million videos [Certain].
How to read Figure 5
VideoBERT: going from 100K to 300K videos almost doubles top-5 verb accuracy (24.5 to 43.3). VideoCLIP: the first three rows are the story of this section, 22.7 with everything, 18.5 without retrieval, 12.4 without retrieval and overlap. MERLOT data: each step toward more diverse video adds points; the image-caption row is far below. MERLOT losses: removing the contrastive loss is the only large drop. Epochs: the curve still rises at 40.
Three designs side by side
| VideoBERT (2019) | VideoCLIP (2021) | MERLOT (2021) | |
|---|---|---|---|
| Visual input | S3D (Kinetics), 1 token per 1.5 s, quantized to 20,736 words | S3D (HowTo100M, MIL-NCE), 1 token per s, 512-d, frozen | one frame per segment, ResNet-50 + ViT trained from scratch |
| Fusion | one stream, BERT-large | none: dual encoder (6-layer video, 12-layer text) | joint encoder, RoBERTa-base |
| Losses | masked text, masked video words, alignment classification | symmetric InfoNCE, overlapped positives, retrieved hard negatives | contrastive frame–transcript, attention-masked LM, temporal reordering |
| Pretraining data | 312K cooking videos, 23,186 h | HowTo100M, 1.1M videos | YT-Temporal-180M, 6M videos, 180M segments |
| Compute | 4 Cloud TPUs, about 2 days | 8 V100 GPUs, 25 epochs, about 1 day | TPU v3-1024, 40 epochs, 30 hours |
| Typical use | zero-shot verb/noun by template, features for captioning | zero-shot retrieval, VideoQA, segmentation, localization | fine-tuned video QA and VCR, zero-shot ordering |
All entries are from the three papers [Certain]. Note how the training cost moved: VideoBERT and VideoCLIP pay for vision only once, by freezing a pretrained 3D CNN and caching its features; MERLOT pays much more compute (a TPU v3-1024 pod) to train vision end to end, and wins on reasoning tasks. The question “frozen features or end-to-end?” returns in every Video-LLM.
The algorithms and the code
- once: sample frames at 20 fps; cut 30-frame (1.5 s) windows; S3D \(\rightarrow \mathbf{f} \in \R^{1024}\) per window // frozen, cached
- once: hierarchical k-means with \(k = 12\), \(d = 4\) on all \(\mathbf{f}\); token \(\leftarrow\) path id, formula (175.2) // 20,736 visual words
- group video tokens by ASR sentence time stamps (or 16 per segment if there is no ASR)
- with some probability, concatenate neighboring ASR sentences; subsample video tokens with a random step in \(\{1, \dots, 5\}\)
- build [CLS] text [>] video [SEP]; for alignment, replace the video sentence with one from another time with probability 0.5 // label c
- mask tokens; loss \(\leftarrow\) weighted sum of text MLM, video MLM and alignment BCE, formulas (175.3) and (175.4)
- Adam step, learning rate \(10^{-5}\), linear decay
- input: videos \(\mathcal{V}\) with cached S3D tokens and ASR; model \(M\); \(k = 32\) videos per batch, 16 pairs per video
- for each video \(V\): \(\mathbf{z}_V \leftarrow\) mean of \(\mathbf{z}_v\) and \(\mathbf{z}_t\) over its clip pairs, formula (175.9)
- build a dense index (FAISS) over all \(\mathbf{z}_V\)
- for each batch: pick a random seed video; sample \(k\) videos from its \(2k\) nearest neighbors
- for each video in the cluster, 16 times: sample a text clip \([s, e]\); center \(c \sim \mathcal{U}(s, e)\); length \(L \sim \mathcal{U}(3, 32)\) s; video clip \([c - L/2, c + L/2]\), formula (175.8)
- encode, average-pool, loss \(\leftarrow\) symmetric InfoNCE over the 512 pairs, formula (175.7) // negatives from similar videos
- Adam step (betas 0.9, 0.98, learning rate \(5 \times 10^{-5}\), warm-up 1,000 steps, gradient clip 2.0) [Certain]
- input: a batch of sequences, each \(N = 16\) segments of (middle frame, 32 BPE tokens)
- image encoder on every frame \(\rightarrow\) grid features and CLS vectors \(\mathbf{a}_i\)
- language-only pass over the whole 512-token transcript \(\rightarrow\) segment CLS vectors \(\mathbf{b}_i\); \(\mathcal{L}_{\text{con}}\) with \(\tau = 0.05\), formula (175.11)
- mask 20% of words, half at random and half among the most-attended tokens of the language-only pass, then extend into spans
- with probability 0.4: pick \(i \in \{2, \dots, N\}\); give \(i\) random frames [image_unk] position embeddings
- joint encoder on 4 groups of 4 segments; \(\mathcal{L}_{\text{MLM}}\) at masked words; \(\mathcal{L}_{\text{order}}\) over all frame pairs, formula (175.12)
- loss \(\leftarrow 0.25\, \mathcal{L}_{\text{con}} + \mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{order}}\), formula (175.13); backward; optimizer step
The PyTorch version of the core pieces, with toy sizes: hierarchical k-means tokenization (175.2), a VideoBERT input with one masked word and one masked visual word (175.3), VideoCLIP’s overlapped clip sampling (175.8) and symmetric InfoNCE over average-pooled towers (175.6, 175.7), and MERLOT’s pairwise ordering loss (175.12).
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # ---------- 1. VideoBERT: hierarchical k-means turns clip features into visual words ---------- def kmeans(x, k, iters=10): c = x[torch.randperm(len(x))[:k]].clone() for _ in range(iters): a = torch.cdist(x, c).argmin(1) for j in range(k): if (a == j).any(): c[j] = x[a == j].mean(0) return c def build_tree(x, k, d): # formula (175.2): k clusters per level, d levels if d == 0 or len(x) < k: return None c = kmeans(x, k); a = torch.cdist(x, c).argmin(1) return (c, [build_tree(x[a == j], k, d - 1) for j in range(k)]) def tokenize(f, tree, k, d): # walk down the tree; id = sum_l c_l * k^(d-l) tok, node = 0, tree for level in range(d): j = 0 if node is None else torch.cdist(f[None], node[0]).argmin().item() tok = tok * k + j node = None if node is None else node[1][j] return tok FEAT, K, D = 16, 3, 2 # VideoBERT: 1024-d S3D features, k = 12, d = 4 clips = torch.randn(600, FEAT) # stand-in for one S3D vector per 1.5 s clip tree = build_tree(clips, K, D) video_words = [tokenize(f, tree, K, D) for f in clips[:8]] print("vocab", K ** D, "first visual words", video_words) # ---------- 2. VideoBERT input: [CLS] text [>] video [SEP], masked prediction ---------- TXT_V = 50 # text vocab; visual words come after it CLS, SEP, GT, MASK = 0, 1, 2, 3 text = torch.tensor([CLS, 10, 11, 12, 13, GT]) video = torch.tensor(video_words) + TXT_V seq = torch.cat([text, video, torch.tensor([SEP])]) enc = nn.TransformerEncoder(nn.TransformerEncoderLayer(32, 4, 64, 0.0, batch_first=True), 2) emb, head = nn.Embedding(TXT_V + K ** D, 32), nn.Linear(32, TXT_V + K ** D) mask = torch.zeros_like(seq, dtype=torch.bool); mask[[2, 8]] = True # one word, one visual word h = enc(emb(seq.masked_fill(mask, MASK))[None])[0] print("seq len", len(seq), "masked loss", round(F.cross_entropy(head(h[mask]), seq[mask]).item(), 3)) # ---------- 3. VideoCLIP: overlapped clips + symmetric InfoNCE over average-pooled tokens ---------- def overlapped_clip(s, e, g): # formula (175.8) c = s + (e - s) * torch.rand(1, generator=g).item() L = 3 + 29 * torch.rand(1, generator=g).item() return max(0.0, c - L / 2), c + L / 2 g = torch.Generator().manual_seed(1) print("clip for text span [10, 14] s:", tuple(round(t, 1) for t in overlapped_clip(10, 14, g))) class Tower(nn.Module): def __init__(s, din): super().__init__() s.proj = nn.Sequential(nn.Linear(din, 32), nn.GELU()) # the MLP after frozen S3D s.enc = nn.TransformerEncoder(nn.TransformerEncoderLayer(32, 4, 64, 0.0, batch_first=True), 2) def forward(s, x): return s.enc(s.proj(x)).mean(1) # AvgPool, not [CLS] def info_nce(zv, zt, tau=1.0): # formulas (175.6) and (175.7) S = zv @ zt.T / tau y = torch.arange(len(S)) return F.cross_entropy(S, y) + F.cross_entropy(S.T, y) vt, tt = Tower(FEAT), Tower(8) B = 6 v_tok = torch.randn(B, 10, FEAT) # 10 one-second S3D tokens per clip (frozen) t_tok = torch.randn(B, 20, 8) # 20 text-token embeddings per clip print("VideoCLIP loss", round(info_nce(vt(v_tok), tt(t_tok)).item(), 3)) # ---------- 4. MERLOT: temporal reordering as pairwise "is i before j?" ---------- N, H = 4, 32 frame_h = torch.randn(N, H) # CLS state of each frame from the joint encoder order_mlp = nn.Sequential(nn.Linear(2 * H, H), nn.ReLU(), nn.Linear(H, 2)) true_t = torch.tensor([2, 0, 3, 1]) # real time step of each (scrambled) frame i, j = torch.triu_indices(N, N, 1) # all N(N-1)/2 pairs logits = order_mlp(torch.cat([frame_h[i], frame_h[j]], -1)) labels = (true_t[i] > true_t[j]).long() # 0: t_i before t_j, 1: t_i after t_j print("pairs", len(labels), "order loss", round(F.cross_entropy(logits, labels).item(), 3))
Output with PyTorch 2.8 on a CPU (the script is deterministic; it gave the same output on every run):
vocab 9 first visual words [0, 0, 2, 5, 1, 7, 4, 0] seq len 15 masked loss 3.968 clip for text span [10, 14] s: (7.5, 18.6) VideoCLIP loss 4.909 pairs 6 order loss 0.698
Things to notice. The tokenizer gives ids in \([0, k^d)\), and two of the first eight clips share word 0: discrete tokens repeat. The masked loss is near \(\ln 59 \approx 4.08\) (59 tokens in the vocabulary), as expected for an untrained model. The sampled clip (7.5 s to 18.6 s) contains the whole text span [10, 14] plus several seconds on each side. The InfoNCE loss is above \(2 \ln 6 = 3.58\) (the value for uniform guessing): with \(\tau = 1\) and unnormalized, untrained vectors, the dot products are large and random, so the model is confidently wrong. Two fixes are standard: normalize the vectors, or learn \(\tau\). The ordering loss is near \(\ln 2 = 0.693\): a coin flip, as it should be before training.
Lessons kept by modern Video-LLMs
Video-LLMs (session 205) look very different: a ViT reads a few frames, a connector turns them into tokens, and an LLM writes the answer. But much of what these three papers found is still inside them.
- Speech is free supervision. HowTo100M, YT-Temporal-180M and their successors made ASR the standard source of video–text pairs. MERLOT Reserve (2022) went further and learned from the audio itself, not only the transcript [Certain]. Today, ASR transcripts are also given to Video-LLMs as extra input text, and used to write or check synthetic captions [Likely]. The noise found here (low relevance, fillers, raw ASR without punctuation) is why those pipelines clean and re-caption the text [Likely].
- Time alignment must be explicit. VideoBERT grouped tokens by ASR time stamps; VideoCLIP sampled overlapping windows; MERLOT gave each segment its own position embedding and trained ordering. Qwen2.5-VL describes “absolute time encoding” so the model can localize events to the second [Certain]. Many Video-LLMs also write time stamps as text next to frame tokens [Likely]. See session 206.
- Cheap visual features first. VideoBERT and VideoCLIP froze a 3D CNN and cached its features; MERLOT used one frame per segment and 5 to 7 frames at fine-tuning time. ClipBERT (2021) made the same point: a few sparsely sampled frames are often enough [Certain]. Video-LLMs sample a few frames and compress tokens for the same reason (session 183).
- Two-stage retrieval. A dual encoder like VideoCLIP is the fast first stage of text-to-video search and Video-RAG; a heavier fusion model or a Video-LLM reranks or answers on the top few (session 208). Hard negatives, from retrieval or clustering, are still how these retrievers are trained (session 60).
- Data diversity matters more than size alone. MERLOT’s HowTo100M-sized subset of diverse video beat HowTo100M itself by 6.5 VCR points [Certain]. Instructional videos are a narrow world: VideoCLIP’s MSR-VTT zero-shot gap shows the same thing.
And what faded: discrete k-means visual words (continuous features won for understanding), and image encoders trained from scratch on video (most later models start from CLIP or SigLIP image encoders, as CLIP4Clip did [Likely]).
Where this is used
| Task | Which idea from this session | How it looks today |
|---|---|---|
| Text-to-video retrieval | VideoCLIP dual encoder, symmetric InfoNCE, hard negatives from clusters | CLIP-based video encoders, a vector index over clips, then reranking (session 208) |
| Zero-shot video classification and action recognition | VideoBERT template filling; VideoCLIP label embeddings | Text prompts of class names against video embeddings, or a Video-LLM asked to name the action |
| Key-moment detection, action step localization | Per-second token scores against step texts (175.10), with an “Outside” threshold | Moment retrieval with time stamps; Video-LLMs that output start and end times |
| Action segmentation | VideoCLIP token-level similarity to 778 COIN labels | Same, with stronger frame encoders and temporal smoothing |
| Video QA, multiple choice | VideoCLIP answer ranking; MERLOT fine-tuned [CLS] head | Video-LLMs generate the answer (session 210) |
| Video-RAG | ASR text as a searchable stream aligned with clips; dual-encoder clip retrieval | Index both transcripts and clip embeddings; retrieve, then give the clips and text to a Video-LLM |
| Video fingerprinting and near-duplicates | Discrete codes per clip (VideoBERT-style quantization) make sequences easy to hash and compare [Likely] | Learned compact descriptors per segment and sequence matching (session 209) |
| Procedure understanding, “what comes next” | VideoBERT video MLM (forecasting future tokens); MERLOT ordering | Temporal reasoning benchmarks and instruction-following assistants for how-to videos |
Common mistakes
- Trusting exact ASR alignment. The words often come before or after the action. In the toy of Figure 1, exact clips cover only 2.7% of the actions; overlapping clips of 16 s cover about 68%. Use overlap, neighboring sentences or a multiple-positive loss like MIL-NCE.
- Only easy negatives. With random batches, the InfoNCE gradient \(1 - p^{+}\) is near 0 for most pairs (0.065 in Figure 3, versus 0.78 with a retrieved cluster). Build batches from similar videos, but sample from a slightly larger neighborhood (\(k\) from \(2k\)).
- Hard negatives that are really positives. Two clips from the same video, or two videos of the same recipe, may both match a sentence. Clustering makes this more likely. Watch for false negatives, for example by excluding clips that overlap in time with the positive.
- Random masking on speech. Speech has fillers and repeated words. Random masking hides “umm” and lets the model copy a repeated object name from the text. Mask the words that are likely to be visual (MERLOT’s attention masking), or mask whole spans.
- Using \(\tau = 1\) with normalized vectors. Cosine similarities lie in \([-1, 1]\); at \(\tau = 1\) the softmax is almost flat (loss near \(\ln B\)). VideoCLIP’s \(\tau = 1.0\) is for its own setup; MERLOT, with normalized vectors, used \(\tau = 0.05\).
- Too small a visual vocabulary, or a frozen one. One k-means word per 1.5 s loses detail, and it cannot improve during training. If you need discrete video tokens (for example for generation), learn the codebook.
- Instructional video only. HowTo100M-only models drop sharply on generic video (MSR-VTT zero-shot R@1 10.4 versus 22.7 on YouCook2). Mix domains, as YT-Temporal-180M did.
- Test videos inside pretraining data. YouCook II and MSR-VTT come from YouTube too. VideoBERT removed YouCook II videos from its pretraining set and VideoCLIP followed HowTo100M’s splits to avoid overlap [Certain]. Do the same.
Interview questions
How does VideoBERT turn a video into tokens?basic
Why are ASR transcripts called free labels, and why are they noisy?basic
What is VideoBERT’s linguistic–visual alignment task, and what two tricks make it work?core
How does VideoBERT do zero-shot action classification, and what are the limits?core
Explain VideoCLIP’s overlapping positives. Why not exact alignment, and how does it differ from MIL-NCE?core
What is retrieval-augmented training in VideoCLIP, and why sample k videos from the 2k nearest?core
Why does a hard negative give a larger gradient than an easy one? Show it with numbers.core
What are MERLOT’s three objectives, and which one mattered most?core
What is attention masking, and why does random masking fail on speech?advanced
MERLOT trains its image encoder from scratch with grid features. Why not use a detector like the image VLMs of 2019?advanced
Design a zero-shot key-moment detector for how-to videos using these ideas.advanced
VideoCLIP is a dual encoder, VideoBERT and MERLOT are fusion encoders. When would you choose each for a video product?advanced
Exercises
- VideoBERT used \(k = 12\) and \(d = 4\). How many visual words would \(k = 8, d = 5\) give? How many centroid comparisons does assigning one clip need with the tree, and with a flat k-means over the same vocabulary?
- A 12-minute cooking video is processed by VideoBERT. How many visual words does it produce? With a random subsampling step of 3, how many are left?
- VideoCLIP samples a text clip \([20, 24]\) s, center \(c = 23\) and length \(L = 10\) s. What is the video clip? If the action is visible during \([26, 32]\), what are coverage and purity (175.1)?
- Compute the InfoNCE video-to-text loss for one row with similarities \((0.6, 0.5, 0.2)\) (the first is the positive), at \(\tau = 0.1\) and at \(\tau = 1\). Which temperature gives the larger gradient \(1 - p^{+}\)?
- MERLOT gives the joint encoder 4 segments. How many pairwise ordering labels does one group produce? How many would one sequence of \(N = 16\) segments produce if all pairs were used at once?
- A model orders a 5-image story as (true ranks of the images in predicted order) \((1, 0, 2, 4, 3)\). Compute Spearman \(\rho\) and pairwise accuracy.
Answerscheck
Key takeaways
- Narrated videos give free text with time stamps, but speech and action are often not at the same time; exact alignment gives many low-relevance pairs (175.1).
- VideoBERT quantizes 1.5-second S3D features into \(12^4 = 20{,}736\) visual words with hierarchical k-means (175.2) and trains BERT-large with masked text, masked video and alignment losses (175.3, 175.4); it can classify zero-shot by filling a template (175.5).
- VideoCLIP is a dual encoder with average pooling (175.6) and symmetric InfoNCE (175.7). Overlapping positives (175.8) and retrieved clusters of similar videos as negatives (175.9) raised zero-shot YouCook2 R@1 from 12.4 to 22.7.
- Hard negatives matter because the InfoNCE gradient on a positive is \(1 - p^{+}\), which is near zero when negatives are easy.
- MERLOT trains vision from scratch on 180M frame–transcript segments with contrastive matching (175.11), attention-masked LM and temporal reordering (175.12, 175.13). The contrastive loss was essential; diverse video beat instructional video.
- Modern Video-LLMs keep the lessons: speech as supervision, explicit time alignment, cheap or cached frame features, and dual encoders as the fast first stage of video search.
Key formulas
References for this session
- Sun et al. (2019). VideoBERT: A Joint Model for Video and Language Representation Learning
- Xu et al. (2021). VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
- Zellers et al. (2021). MERLOT: Multimodal Neural Script Knowledge Models
- Sun et al. (2019). VideoBERT: A Joint Model for Video and Language Representation Learning
- Xu et al. (2021). VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
- Zellers et al. (2021). MERLOT: Multimodal Neural Script Knowledge Models
- Miech, Zhukov, Alayrac, Tapaswi, Laptev & Sivic (2019). HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Miech, Alayrac, Smaira, Laptev, Sivic & Zisserman (2020). End-to-End Learning of Visual Representations from Uncurated Instructional Videos
- Xie, Sun, Huang, Tu & Murphy (2018). Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- van den Oord et al. (2018). Representation Learning with Contrastive Predictive Coding
- Zellers et al. (2022). MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound
- Lei et al. (2021). Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Bai et al. (2025). Qwen2.5-VL Technical Report