LLM & VLM courseBlog
166Track 7: Vision and VLMs · session 4 of 38

CLIP: learning images and text together

CLIP trains an image encoder and a text encoder together so that a photo and its caption land on the same point of one shared vector space. The only training signal is a symmetric contrastive loss over a batch of image–text pairs, with a learned temperature. After training, you can classify images into any set of labels by writing the labels as sentences, with no labeled training images at all. This session builds the loss by hand, trains a tiny CLIP, and ends with what CLIP cannot do.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 3 (cross-entropy), 60 (contrastive training) and 164 (the Vision Transformer)

What you will learn

  • The dual-encoder design: two towers, two projections, L2 normalization, one shared space
  • The symmetric InfoNCE loss over an N × N similarity matrix, computed by hand, and its gradient (pull the diagonal, push the rest)
  • Why the temperature must be learned, why it starts at 0.07 and why it is capped at 100
  • Why contrastive training needs huge batches (the ln N bound) and how the N × N matrix is split across GPUs
  • Zero-shot classification with prompts such as “a photo of a {label}.”, and prompt ensembling in embedding space
  • The real CLIP recipe: 400 million web pairs, batch 32,768, 32 epochs, and the 76.2% zero-shot ImageNet result
  • The limits: counting, spatial relations and word order, small text, typographic attacks, the modality gap and web-data bias

Why this matters

Before 2021, a vision model could only name the classes it was trained on: train on 1,000 ImageNet classes and you get exactly those 1,000 outputs. CLIP (Contrastive Language–Image Pre-training, Radford et al., 2021) removed that wall. It learns from 400 million images paired with the text that appeared next to them on the web, and it puts images and sentences into one shared vector space. After training, “which class is this photo?” becomes “which sentence is closest to this photo?”, and you can write any sentence you like.

This idea is everywhere in modern multimodal systems. The image encoder of many vision-language models (VLMs) started life as a CLIP encoder: LLaVA uses CLIP ViT-L/14 [Certain] (session 181). Text-to-image search, video search and near-duplicate detection run on CLIP-style embeddings (session 169). Stable Diffusion v1 conditions its image generator on the CLIP ViT-L/14 text encoder [Certain] (session 228). Datasets such as LAION were filtered with CLIP scores [Certain]. And the next session, session 167, is a direct answer to CLIP’s loss.

The previous sessions built strong image encoders: the Vision Transformer (session 164) with labels, and MAE and DINO without labels (session 165). Self-supervised features are good, but they are not connected to words. CLIP connects them. Technically, CLIP is the same in-batch contrastive training that session 60 used for text search, with one change: the two towers read different modalities.

Two encoders, one shared space

CLIP has two separate networks, often called towers:

  • An image encoder \(f_\theta\). CLIP trained five ResNets and three Vision Transformers (ViT-B/32, ViT-B/16, ViT-L/14). For a ViT, the image feature is the output at the class token. [Certain]
  • A text encoder \(g_\phi\): a 12-layer transformer, 512 wide with 8 heads in the base size (63 million parameters), reading lower-cased byte-pair-encoded text of at most 76 tokens. The text feature is the top-layer activation at the end-of-sequence token [EOS]. [Certain]

The two features have different sizes (for ViT-L/14, 1,024 for the image and 768 for the text). Two learned linear projections \(W_I\) and \(W_T\) map both to the same embedding size \(d\), and each result is divided by its length (L2 normalization), so every embedding is a point on the unit sphere:

\[ \mathbf{u}_i = \frac{W_I\, f_\theta(x_i)}{\lVert W_I\, f_\theta(x_i) \rVert}, \qquad \mathbf{v}_j = \frac{W_T\, g_\phi(y_j)}{\lVert W_T\, g_\phi(y_j) \rVert} \tag{166.1} \]
Read the symbols
\(x_i, y_j\)
the \(i\)-th image and the \(j\)-th text of the batch
\(f_\theta, g_\phi\)
image encoder and text encoder, with weights \(\theta\) and \(\phi\)
\(W_I, W_T\)
linear projections into the shared space of size \(d\)
\(\mathbf{u}_i, \mathbf{v}_j\)
unit-length image and text embeddings in \(\R^d\); their dot product is their cosine similarity, a number in \([-1, 1]\)

The model sizes, from the CLIP paper’s Table 20 [Certain]:

ModelImage inputVision transformer (layers, width, heads)Text transformer (layers, width, heads)Shared size \(d\)
ViT-B/3222412, 768, 1212, 512, 8512
ViT-B/1622412, 768, 1212, 512, 8512
ViT-L/1422424, 1024, 1612, 768, 12768
ViT-L/14@336px33624, 1024, 1612, 768, 12768

Notice what is not there: no layer reads the image and the text together. The towers meet only at the very end, through one dot product. This is called a dual encoder (or two-tower, or late-interaction) model. It has a big practical advantage: you can embed a billion images once, store the vectors, and later compare any new text against all of them with fast vector search. A model that reads image and text together (a fusion model, see session 172) must run once per pair.

dog cat car images image encoderViT or ResNet WI, then÷ length u₁u₂u₃ “a dogon grass”“a cat …”“a red car” captions text encodertransformer, [EOS] WT, then÷ length v₁ v₂ v₃ logits ℓij = s · ui · vj v₁v₂v₃ u₁u₂u₃ u₁·v₁u₁·v₂u₁·v₃ u₂·v₁u₂·v₂u₂·v₃ u₃·v₁u₃·v₂u₃·v₃ rows: softmax over texts (image → text)columns: softmax over images (text → image)
Figure 1. The CLIP architecture for a batch of \(N = 3\) pairs. Each tower runs alone; the towers meet only in the \(N \times N\) matrix of dot products. The diagonal (outlined) holds the true pairs. Every other cell is a negative: an image with a caption that belongs to a different image in the same batch.

Why not train the image model to write the caption, like a captioning model? The CLIP authors tried. A transformer that predicts the caption word by word learned zero-shot ImageNet classification 3 times slower than a model that only predicts which words occur (a bag of words), and swapping that prediction for the contrastive objective gave another 4 times better efficiency [Certain, their Figure 2]. Predicting the exact words of a web caption is a very hard task, full of details that do not help recognition. Picking which caption, among the ones in the batch, belongs to this image is easier, and it is enough to learn visual concepts.

The contrastive loss: every batch is a classification problem

Take a batch of \(N\) (image, text) pairs. Embed all images and all texts with (166.1), then build the \(N \times N\) matrix of scaled dot products, the logits:

\[ \ell_{ij} = s\; \mathbf{u}_i^\top \mathbf{v}_j = \frac{\cos(x_i, y_j)}{\tau} \tag{166.2} \]
Read the symbols
\(\ell_{ij}\)
score of image \(i\) with text \(j\); the true pairs are on the diagonal \(i = j\)
\(s\)
the logit scale, a learned positive number
\(\tau = 1/s\)
the temperature; the two names describe the same knob

Now read each row as a classification problem: “image \(i\), which of these \(N\) texts is yours?” The correct class is \(i\). Use a softmax over the row and the usual cross-entropy (session 3). Then read each column the other way: “text \(j\), which of these \(N\) images is yours?”:

\[ \mathcal{L}_{I \to T} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{e^{\ell_{ii}}}{\sum_{j=1}^{N} e^{\ell_{ij}}} \tag{166.3} \]
\[ \mathcal{L}_{T \to I} = -\frac{1}{N} \sum_{j=1}^{N} \log \frac{e^{\ell_{jj}}}{\sum_{i=1}^{N} e^{\ell_{ij}}} \tag{166.4} \]
\[ \mathcal{L}_{\text{CLIP}} = \tfrac{1}{2} \big( \mathcal{L}_{I \to T} + \mathcal{L}_{T \to I} \big) \tag{166.5} \]
Read the symbols
\(\mathcal{L}_{I \to T}\)
image-to-text loss: cross-entropy of every row, averaged
\(\mathcal{L}_{T \to I}\)
text-to-image loss: cross-entropy of every column, averaged
\(\mathcal{L}_{\text{CLIP}}\)
the symmetric loss that CLIP minimizes; the CLIP paper gives exactly this as numpy-like pseudocode [Certain]

This family of losses is called InfoNCE (from “noise-contrastive estimation”, van den Oord et al., 2018). Each positive is compared against \(N - 1\) negatives that cost nothing extra: they are simply the other pairs of the batch. A batch of \(N\) pairs gives \(N\) positives and \(N(N-1)\) negatives.

Why symmetric? The two directions are different problems. A row softmax asks the image to prefer its caption over other captions; a column softmax asks the caption to prefer its image over other images. A model trained only on rows can still place a caption close to many images. CLIP is used in both directions (search images with text, label images with text), so it trains both. Removing one direction is a common interview follow-up; it usually still works but is a little worse [Likely].

Worked example: three pairs by hand, scale \(s = 10\)

Suppose the cosine similarities of a batch of three pairs are (rows = images, columns = texts):

\[ C = \begin{pmatrix} 0.30 & 0.10 & 0.05 \\ 0.12 & 0.28 & 0.15 \\ 0.02 & 0.20 & 0.25 \end{pmatrix}, \qquad \ell = 10\, C \]

Row 1 has logits (3.0, 1.0, 0.5). Softmax: \(e^{3} / (e^{3} + e^{1} + e^{0.5}) = 20.09 / 24.45 = 0.821\). Its loss is \(-\ln 0.821 = 0.197\). Row 3 has logits (0.2, 2.0, 2.5): its own caption is only slightly ahead of caption 2, so the probability is 0.586 and the loss is 0.535. Doing all three rows: \(\mathcal{L}_{I \to T} = (0.197 + 0.388 + 0.535)/3 = 0.373\). The three columns give \(\mathcal{L}_{T \to I} = (0.204 + 0.479 + 0.408)/3 = 0.364\). So \(\mathcal{L}_{\text{CLIP}} = 0.368\).

Now the same cosines with \(s = 1\): every row is almost uniform (row 1 gives 0.385, 0.315, 0.300), and the loss is 0.990, close to the “know nothing” value \(\ln 3 = 1.099\). The geometry has not changed, only the scale. Cosines live in \([-1, 1]\), so without a large scale the softmax cannot become confident. Hold this thought for the temperature section.

Figure 2 runs the whole forward pass of one training step in your browser, one operation at a time, on a batch of 4 pairs. The projected features are 3 numbers each, so you can check every product by hand. The batch has a near-duplicate: a dog photo and a puppy photo, whose captions fit both images. Watch formula (166.1) normalize each vector, the \(4 \times 4\) matrix fill in cell by cell, the temperature turn cosines into logits (166.2), and the row and column softmaxes turn logits into the two losses (166.3) and (166.4).

Projected features → unit vectors (166.1)
The \(4 \times 4\) matrix, the row losses and the column losses
cell being computedprobability on the true pairprobability lost to a wrong pair
  1. input: projected features \(\tilde{\mathbf{x}}_i = W_I f_\theta(x_i)\), \(\tilde{\mathbf{y}}_j = W_T g_\phi(y_j)\); temperature \(\tau\)
  2. for \(i = 1 \dots N\): \(\mathbf{u}_i \leftarrow \tilde{\mathbf{x}}_i / \lVert \tilde{\mathbf{x}}_i \rVert\) // (166.1)
  3. for \(j = 1 \dots N\): \(\mathbf{v}_j \leftarrow \tilde{\mathbf{y}}_j / \lVert \tilde{\mathbf{y}}_j \rVert\)
  4. for each \(i, j\): \(C_{ij} \leftarrow \mathbf{u}_i \cdot \mathbf{v}_j\) // cosine
  5. \(\ell \leftarrow C / \tau\) // (166.2), \(s = 1/\tau\)
  6. for each row \(i\): \(P_{i,:} \leftarrow \softmax(\ell_{i,:})\); \(\ \mathcal{L}^{r}_i \leftarrow -\ln P_{ii}\) // (166.3)
  7. for each column \(j\): \(Q_{:,j} \leftarrow \softmax(\ell_{:,j})\); \(\ \mathcal{L}^{c}_j \leftarrow -\ln Q_{jj}\) // (166.4)
  8. \(\mathcal{L} \leftarrow \tfrac12\big(\text{mean}_i\, \mathcal{L}^{r}_i + \text{mean}_j\, \mathcal{L}^{c}_j\big)\) // (166.5)
Figure 2. One CLIP forward pass and loss, animated, for \(N = 4\) pairs with 3-number projected features. Left: each raw feature and its unit vector. Right: the matrix shows the cosines \(C_{ij}\) while it fills, then the logits \(\ell_{ij}\), then the row softmax \(P\) and the column softmax \(Q\); the losses of finished rows appear on the right and of finished columns at the bottom. The code box highlights the line being run, and the panel next to it shows that line’s formula with the live numbers. Every number is computed in your browser.
How to read Figure 2

Press Play, or Step through it. At \(\tau = 0.07\) (the CLIP starting value, \(s = 14.29\)) the cat and car rows and columns are easy: their losses are 0.006 or less. The dog row gives 0.118 of its probability to the puppy caption, and the puppy row gives 0.322 to the dog caption (loss 0.390). The dog caption column gives 0.245 to the puppy photo (loss 0.281). The final loss is 0.124. Now move \(\tau\) to 1 while the animation is at the end: the same cosines give 0.985, close to the chance value \(\ln 4 = 1.386\), because logits that differ by less than 2 cannot make a confident softmax. At \(\tau = 0.01\) the loss falls to 0.001. Lowering \(\tau\) never changes which caption wins a row; it only changes how sure the softmax is.

Figure 3 lets you redo this example and a harder batch with a near-duplicate: a photo of a puppy on a beach and a photo of a dog on grass, whose captions are both plausible for both photos.

true pair (diagonal)selected row or columngradient pushes the score downgradient pulls the score up
Figure 3. The CLIP loss (166.3) to (166.5) on a tiny batch with fixed cosine similarities. Click any cell to select its row (or its column, in text → image mode). The bars on the right show the softmax of the selected row or column and the gradient of formula (166.6) on each of its logits.
How to read Figure 3

With the worked-example batch at \(s = 10\) the readout shows the numbers of the box above: 0.373, 0.364 and 0.368. Now pick the 4-pair batch. Row 1 (the dog photo) gives only 0.528 to its own caption and 0.354 to the puppy caption: the near-duplicate takes a large share and gets the largest push down. The red car row is easy (0.165 loss). Slide \(s\) to 100: the easy rows reach loss 0.000, and the near-duplicate rows still cost 0.018, because 0.31 versus 0.27 is a small cosine gap. Slide \(s\) to 1: the total loss is 1.241, close to \(\ln 4 = 1.386\), whatever the cosines are.

What the gradient does

The derivative of the loss with respect to each logit has a simple form. Let \(P_{ij}\) be the row softmax (the probability that image \(i\) picks text \(j\)) and \(Q_{ij}\) the column softmax (the probability that text \(j\) picks image \(i\)):

\[ \frac{\partial \mathcal{L}_{\text{CLIP}}}{\partial \ell_{ij}} = \frac{1}{2N} \big( P_{ij} + Q_{ij} - 2\,\delta_{ij} \big), \qquad \frac{\partial \mathcal{L}_{\text{CLIP}}}{\partial \mathbf{u}_i} = s \sum_{j=1}^{N} \frac{\partial \mathcal{L}_{\text{CLIP}}}{\partial \ell_{ij}}\, \mathbf{v}_j \tag{166.6} \]
Read the symbols
\(\delta_{ij}\)
1 if \(i = j\) (a true pair), else 0
\(P_{ij}, Q_{ij}\)
row and column softmax probabilities of logit \(\ell_{ij}\)
\(\partial \mathcal{L} / \partial \mathbf{u}_i\)
the gradient on the image embedding (before the small correction from the normalization, which removes the part along \(\mathbf{u}_i\))

On the diagonal the gradient is negative (it is at most 0, since \(P_{ii}, Q_{ii} \le 1\)): gradient descent raises the score of the true pair, pulling \(\mathbf{u}_i\) towards \(\mathbf{v}_i\). Off the diagonal it is positive, and proportional to how much probability the wrong pair got: the model pushes hardest on the negatives it currently confuses, the hard negatives. Easy negatives with probability near 0 get almost no gradient. This is why the composition of a batch matters so much.

Algorithm 1: one CLIP training step (following the paper’s pseudocode)
  1. input: a batch of \(N\) images \(x_{1..N}\) and their texts \(y_{1..N}\); weights \(\theta, \phi, W_I, W_T\); log-scale \(t\)
  2. \(F_I \leftarrow f_\theta(x)\), \(F_T \leftarrow g_\phi(y)\) // features, shapes \((N, d_i)\) and \((N, d_t)\)
  3. \(U \leftarrow \text{normalize}(F_I W_I)\), \(V \leftarrow \text{normalize}(F_T W_T)\) // formula (166.1), shape \((N, d)\)
  4. \(s \leftarrow \min(e^{t}, 100)\); \(\ \ell \leftarrow s\, U V^\top\) // formula (166.2), shape \((N, N)\)
  5. labels \(\leftarrow (0, 1, \dots, N-1)\) // the diagonal
  6. \(\mathcal{L} \leftarrow \tfrac12\big(\text{CE}(\ell, \text{labels}) + \text{CE}(\ell^\top, \text{labels})\big)\) // rows, then columns: (166.3) to (166.5)
  7. backpropagate into \(\theta, \phi, W_I, W_T\) and \(t\); take an AdamW step // the temperature is a weight

The learned temperature

The worked example showed the problem: a cosine is at most 1, so with \(s = 1\) a logit can differ from another by at most 2, and the softmax over \(N\) texts can never be confident. Even a perfect geometry has a high loss. If the \(N\) embeddings are as spread out as possible (the corners of a regular simplex, with cosine \(-1/(N-1)\) between all wrong pairs) the smallest possible loss is

\[ \mathcal{L}_{\min}(s) = \ln\!\Big( 1 + (N - 1)\, e^{-s N/(N-1)} \Big) \]

For \(N = 32{,}768\) and \(s = 1\) this is 9.397, barely below the chance value \(\ln N = 10.397\). With \(s = 14.3\) it is 0.020, and with \(s = 100\) it is essentially 0. So the scale has to be large. But how large? Too large, and the softmax becomes a hard maximum: gradients go almost entirely to the single hardest negative, and noisy web pairs (captions that do not really describe the image) get huge penalties. CLIP does not tune it by hand. It learns it, through a log-parameterization that keeps it positive:

\[ s = \min\big(e^{t},\ 100\big), \qquad t_0 = \ln(1/0.07) \approx 2.659, \qquad \tau = 1/s \tag{166.7} \]
Read the symbols
\(t\)
the trained parameter: the log of the logit scale; gradient descent updates it like any weight, with \(\partial \mathcal{L}/\partial t = \sum_{ij} \frac{\partial \mathcal{L}}{\partial \ell_{ij}}\, \ell_{ij}\)
\(t_0\)
the initial value: temperature 0.07 (scale 14.3), taken from earlier instance-discrimination work (Wu et al., 2018) [Certain]
100
the cap: CLIP clips the scale at 100 (temperature 0.01) because larger values made training unstable [Certain]

The Hugging Face CLIP config stores logit_scale_init_value = 2.6592, which is exactly \(\ln(1/0.07)\) [Certain]. ALIGN (Jia et al., 2021), trained at the same time on 1.8 billion noisier pairs, made the same choice: its temperature starts at 1.0 and is learned, and the authors report that learning it worked as well as a manual sweep [Certain].

Figure 4 trains a tiny CLIP: 8 images and 8 captions with different random features (8 numbers each, as if from two different sensors), two linear towers into a 2-D shared space, and the loss (166.5). It runs three versions side by side, which differ only in the temperature.

image embedding \(\mathbf{u}\)text embedding \(\mathbf{v}\)line joins a true pair
Figure 4. Training two linear towers \(W_I, W_T \in \R^{2 \times 8}\) with Adam (learning rate 0.02) on one fixed batch of 8 pairs. Left: the embeddings of the selected run on the unit circle; the number is the pair index. Right: the loss of all three runs (log scale), the dashed line is \(\ln 8\), and the marker is the current step. All three runs start from the same seeded weights.
How to read Figure 4

At step 0 the embeddings are random and the loss of the two runs with \(s = 14.3\) is 9.509, far above \(\ln 8 = 2.079\): confident and wrong is worse than uniform. Drag the step slider. The fixed \(s = 1\) run arranges the pairs perfectly (all 8 images find their caption, mean wrong-pair cosine −0.143, the most spread possible on a circle) but its loss stays just above 1.315, the lowest possible value for 8 points on a circle at \(s = 1\). The learned run first lowers its scale (to about 10.2 near step 40), then raises it once the pairs are separated, reaching \(s \approx 31.6\) and loss 0.025 at step 400, against 0.079 for the fixed \(s = 14.3\) run. The temperature adapts to how well the model currently does.

In the real CLIP the learned scale climbs to the cap: the released OpenAI checkpoints have a logit scale of 100 [Likely]. That is one reason zero-shot probabilities from CLIP are often very peaked.

Why the batch must be huge

CLIP used a batch of 32,768 pairs [Certain]. Why so large? Each row of the loss is a \(N\)-way classification. With more negatives, the task is harder, and the model must learn finer differences to solve it. There is also a clean bound. van den Oord et al. showed that InfoNCE is a lower bound on the mutual information between the two views (how many nats of information an image and its caption share):

\[ I(X; Y) \;\ge\; \ln N - \mathcal{L}_{I \to T} \tag{166.8} \]
Read the symbols
\(I(X; Y)\)
mutual information between an image and its paired text, in nats
\(\ln N\)
the loss of a model that guesses uniformly over the batch (chance level)
\(\mathcal{L}_{I \to T}\)
the InfoNCE loss, never below 0, so the bound can never exceed \(\ln N\)

Even a perfect model can only “prove” \(\ln N\) nats with a batch of \(N\): \(\ln 256 = 5.5\), \(\ln 32{,}768 = 10.4\). A small batch saturates early: once the loss is near 0, the task gives almost no signal. A big batch keeps it informative longer. The price is the \(N \times N\) matrix. Its full size for \(N = 32{,}768\) is \(32{,}768^2 \approx 1.07\) billion logits, 4.29 GB in float32. Nobody stores it on one GPU. CLIP sharded the similarity computation: each GPU computes only the rows of its local images against all texts (and the columns of its local texts against all images) [Certain].

Algorithm 2: the CLIP loss across \(G\) GPUs (sharded similarity)
  1. input: on GPU \(r\): its local batch of \(n = N/G\) pairs
  2. compute local embeddings \(U_r, V_r\) of shape \((n, d)\) // towers run only on local data
  3. \(U_{\text{all}} \leftarrow \text{all-gather}(U_r)\), \(V_{\text{all}} \leftarrow \text{all-gather}(V_r)\) // shapes \((N, d)\); see the collectives session
  4. \(\ell^{\text{img}} \leftarrow s\, U_r V_{\text{all}}^\top\) // \((n, N)\): my images against every text
  5. \(\ell^{\text{txt}} \leftarrow s\, V_r U_{\text{all}}^\top\) // \((n, N)\): my texts against every image
  6. labels \(\leftarrow r \cdot n + (0, \dots, n-1)\) // where my positives sit among the \(N\) columns
  7. \(\mathcal{L}_r \leftarrow \tfrac12\big(\text{CE}(\ell^{\text{img}}, \text{labels}) + \text{CE}(\ell^{\text{txt}}, \text{labels})\big)\)
  8. backward; gradients reach the local towers through \(U_r, V_r\) (and, if the gather keeps gradients, through the gathered copies); then all-reduce the weight gradients as in data parallelism

The gather itself is cheap: \(32{,}768 \times 768\) numbers in bfloat16 is 50.3 MB per modality. The all-gather and all-reduce are explained in session 43 and session 44. Figure 5 shows how chance loss, the bound and the memory grow with \(N\).

chance loss \(\ln N\)proven information \(\ln N - \mathcal{L}\)measured loss \(\mathcal{L}\)
Figure 5. Batch size and the InfoNCE bound (166.8). The horizontal axis is \(\log_2 N\). The readout gives the size of the full logit matrix and the slice each GPU computes (float32, one direction), for embedding size 768 as in ViT-L/14.
How to read Figure 5

Keep the loss at 2 nats and lower \(N\): below \(N = 8\) the bound \(\ln N - L\) is negative, so the loss proves nothing. At \(N = 32{,}768\) the same loss proves at least 8.40 nats. Now look at memory: the full matrix is 4.29 GB, but with 256 GPUs each one holds a \(128 \times 32{,}768\) slice, 16.8 MB. Doubling \(N\) quadruples the full matrix but only doubles each slice if \(G\) doubles too.

The data and the recipe

The loss is simple; the scale is what made CLIP work. The facts from the paper [Certain]:

  • Data. WIT (WebImageText): 400 million (image, text) pairs from public internet sources. The authors searched for pairs whose text contains one of 500,000 queries (words that appear at least 100 times in English Wikipedia, plus some bigrams and names), and kept at most 20,000 pairs per query to balance the concepts. The total word count is similar to WebText, the GPT-2 training set.
  • Training. 32 epochs, batch 32,768, Adam with decoupled weight decay (AdamW, decay 0.2), 2,000 warm-up steps and a cosine schedule, mixed precision. Optimization tricks from session 4 all appear here.
  • Compute. The largest ResNet (RN50x64) trained for 18 days on 592 V100 GPUs; the largest ViT (ViT-L/14) for 12 days on 256 V100 GPUs, then one extra epoch at 336-pixel resolution, giving ViT-L/14@336px, the model the paper calls “CLIP”.

The data was never released. Open replications followed: LAION-400M and LAION-5B built pairs from Common Crawl and filtered them with OpenAI’s CLIP itself; OpenCLIP trained CLIP models on them and measured scaling laws (Cherti et al.); DataComp turned dataset filtering into a benchmark [Certain]. ALIGN showed the other direction: 1.8 billion pairs with only minimal filtering, an EfficientNet-L2 image tower and a BERT-Large text tower, batch 16,384, and 76.4% zero-shot ImageNet accuracy [Certain]. Quantity can beat cleanliness, up to a point. Data quality and filtering for multimodal training are the topic of session 187.

Zero-shot classification with prompts

Here is the payoff. To classify an image into \(K\) classes, write each class as a sentence, embed the sentences with the text tower, and pick the sentence closest to the image:

\[ p(y = k \mid x) = \frac{\exp\!\big(s\, \mathbf{u}^\top \mathbf{w}_k\big)}{\sum_{k'=1}^{K} \exp\!\big(s\, \mathbf{u}^\top \mathbf{w}_{k'}\big)}, \qquad \mathbf{w}_k = \mathbf{v}\big(\text{“a photo of a } \{\text{label}_k\}\text{.”}\big) \tag{166.9} \]
Read the symbols
\(\mathbf{u}\)
the unit image embedding of the test image
\(\mathbf{w}_k\)
the unit text embedding of the prompt for class \(k\)
\(K\)
the number of classes; you choose them at test time

Look at (166.9) again: it is a linear classifier whose weight rows are the text embeddings. Pretraining was a \(N\)-way classification against captions; zero-shot is a \(K\)-way classification against class sentences. Nothing new is trained. The text embeddings are computed once per class and cached, so classifying an image costs one image-tower pass and one \(K \times d\) matrix product.

Why a prompt and not just the label? Two reasons, both from the paper [Certain]. First, web captions are rarely a single word, so “dog” alone is unusual input for the text tower. Second, polysemy: one word, several meanings. ImageNet has “crane” the bird and “crane” the machine; in Oxford Pets, “boxer” is a dog breed but could also be an athlete. Using the template “A photo of a {label}.” improved ImageNet accuracy by 1.3 points. Task-specific templates help further: “A photo of a {label}, a type of pet.” for pets, “a satellite photo of a {label}.” for satellite images.

Prompt ensembling. Instead of one template, use \(M\) of them (“a photo of a big {label}.”, “a drawing of a {label}.”, …), and average their embeddings:

\[ \mathbf{w}_k = \frac{\bar{\mathbf{w}}_k}{\lVert \bar{\mathbf{w}}_k \rVert}, \qquad \bar{\mathbf{w}}_k = \frac{1}{M} \sum_{m=1}^{M} \mathbf{v}\big(\text{template}_m(\text{label}_k)\big) \tag{166.10} \]

The average is taken in embedding space, not over probabilities, so the cost at test time is the same as one template. On ImageNet, ensembling 80 templates added 3.5 points over the single default prompt; prompt engineering and ensembling together gave almost 5 points [Certain]. With all this, the best CLIP model (ViT-L/14@336px) reaches 76.2% top-1 zero-shot on ImageNet, matching the original fully supervised ResNet-50 without using any of its 1.28 million labeled training images, and 95% top-5 [Certain]. Over 20 datasets, zero-shot CLIP matched the average performance of a 4-shot linear probe on its own features [Certain].

Algorithm 3: building and running a zero-shot classifier
  1. input: class names \(\text{label}_{1..K}\), templates \(\text{template}_{1..M}\), a trained CLIP
  2. for \(k = 1 \dots K\): // once, offline
  3.   embed every filled template \(\text{template}_m(\text{label}_k)\) with the text tower, normalize each
  4.   \(\mathbf{w}_k \leftarrow\) normalize(mean of the \(M\) embeddings) // formula (166.10)
  5. stack the rows: \(W \in \R^{K \times d}\) // the zero-shot classifier
  6. for each test image \(x\): \(\mathbf{u} \leftarrow\) image embedding; scores \(\leftarrow W \mathbf{u}\) // cosine similarities
  7.   predict \(\argmax_k\) scores; probabilities \(\leftarrow \softmax(s \cdot \text{scores})\) // formula (166.9)

Figure 6 is a toy version with synthetic embeddings (16 dimensions, 5 classes, 400 test images). Each template moves the class embeddings in its own way, and the bare label word moves them the most. These numbers are made up to show the mechanism; they are not CLIP results.

  1. \(\mathbf{u} \leftarrow\) unit image embedding of the test image // one image-tower pass
  2. for \(k = 1 \dots K\): \(c_k \leftarrow \mathbf{u} \cdot \mathbf{w}_k = \sum_{d=1}^{16} u_d\, w_{k,d}\) // cosine with prompt \(k\)
  3. \(p \leftarrow \softmax(s \cdot c)\), \(s = 100\) // (166.9)
  4. \(\hat{y} \leftarrow \argmax_k c_k\) // the prediction
Figure 6. Zero-shot classification (166.9) with synthetic embeddings, animated. Each class embedding comes from a template; “Ensemble” averages four templates as in (166.10). The classifier scores one prompt per step: the strip at the bottom shows the 16 products \(u_d\, w_{k,d}\) whose sum is the cosine. Left bars: cosine similarity of the test image with each class. Right bars: softmax with \(s = 100\). The readout gives the accuracy of each classifier on all 400 synthetic test images.
How to read Figure 6

Press Play. Each step takes one class prompt and adds up its 16 products into one cosine bar (green products raise it, red ones lower it). Then the softmax turns the five cosines into probabilities, and the argmax picks the class. Image 12 is a cat. With the bare label the classifier says “car”; with “a photo of a {label}.” and with the ensemble it says “cat”. Over all 400 images the bare label gets 79.8%, the single template 86.5% and the ensemble 96.5%. Averaging the four templates cancels much of the noise that each template adds to each class, which is the same reason ensembling helped real CLIP. Notice also how peaked the probabilities are at \(s = 100\): a cosine gap of 0.05 already gives a probability ratio of \(e^{5} \approx 148\). Do not read CLIP probabilities as calibrated confidence.

What CLIP cannot do

CLIP’s strengths come from its loss and its data, and so do its weaknesses. The CLIP paper itself lists many of them [Certain unless tagged]:

LimitEvidenceWhy it happens
CountingWeak zero-shot results on counting objects in synthetic scenes (CLEVRCounts)A caption rarely states an exact count, and a batch rarely holds two images that differ only in count, so the loss never needs it [Likely]
Spatial relations and word orderWinoground: two images and two captions with the same words in a different order; the models tested, CLIP among them, did not do much better than chance. ARO: CLIP-style models show “a severe lack of order sensitivity”Retrieval on random web batches can be solved without word order. Yuksekgonul et al. show models can score well on standard retrieval without using order information, so they behave like a bag of words
Distance and geometryNear random on the distance to the nearest car (KITTI Distance)Such facts are almost never written in web captions
Fine-grained classesPoor on car models, flower species, aircraft variants compared with task-specific modelsWeb captions name the general concept more often than the exact variant
Small text and out-of-distribution imagesGood on rendered text, but only 88% on handwritten MNIST digits, below logistic regression on raw pixelsAlmost no MNIST-like images in the data; at 224 pixels small text has very few pixels (session 180)
Typographic attacksAn apple with a paper label “iPod” is classified as an iPod (Goh et al., Distill 2021)CLIP learned to read, and written words in the image can outweigh the object
Web-data biasIn the paper’s FairFace probe with added non-human and crime-related labels, 4.9% of faces were put in non-human classes, about 14% for images labeled Black; 16.5% of male images versus 9.8% of female images were put in crime-related classesThe model learns the associations of its web text. Class design matters: adding a “child” class sharply reduced such errors for people under 20
The modality gapImage embeddings and text embeddings occupy two separate regions (“cones”) of the shared space (Liang et al., 2022)Each tower starts in a narrow cone at initialization, and contrastive training with a low temperature keeps the two apart

The deeper reason behind several rows: the contrastive loss only asks “which caption in this batch?”. If no other caption in the batch differs only in count or in word order, the model gets the answer right without learning count or order. Hard negatives built on purpose (swapping words, changing numbers) help, and the ARO paper proposes exactly that [Certain]. Later sessions show how VLMs fix the rest: a language model that reads all patch tokens instead of one pooled vector (session 177), higher resolution, and better data. Common failure modes of full VLMs are collected in session 193.

The PyTorch version

A complete, deterministic CLIP on synthetic data. Ten hidden “concepts” are seen through two different random sensors: an image feature of size 32 and a text feature of size 24. Two small MLP towers map both to a 16-d shared space. We train with the symmetric loss and a learned temperature, then do zero-shot classification and retrieval. It runs on a CPU in a few seconds.

import math, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# 1) synthetic paired data: K concepts seen through two different "sensors"
K, Z, DI, DT = 10, 8, 32, 24                  # concepts, latent size, image / text feature sizes
proto = torch.randn(K, Z)                      # one latent vector per concept
to_img, to_txt = torch.randn(Z, DI), torch.randn(Z, DT)
def make_pairs(n):
    y = torch.randint(0, K, (n,))
    z = proto[y] + 0.5 * torch.randn(n, Z)     # this pair's content
    img = z @ to_img + 0.3 * torch.randn(n, DI)   # "pixels" of the image
    txt = z @ to_txt + 0.3 * torch.randn(n, DT)   # "words" of its caption
    return img, txt, y
img_tr, txt_tr, _ = make_pairs(4096)
img_te, _, y_te = make_pairs(1000)

# 2) two encoders and a learned temperature, formulas (166.1) and (166.7)
class CLIP(nn.Module):
    def __init__(self, d_emb=16):
        super().__init__()
        self.img_enc = nn.Sequential(nn.Linear(DI, 64), nn.GELU(), nn.Linear(64, d_emb))
        self.txt_enc = nn.Sequential(nn.Linear(DT, 64), nn.GELU(), nn.Linear(64, d_emb))
        self.t = nn.Parameter(torch.tensor(math.log(1 / 0.07)))     # log of the logit scale
    def forward(self, img, txt):
        u = F.normalize(self.img_enc(img), dim=-1)                  # unit image embeddings
        v = F.normalize(self.txt_enc(txt), dim=-1)                  # unit text embeddings
        s = self.t.exp().clamp(max=100.0)                           # scale = 1 / temperature
        return s * u @ v.T                                          # (N, N) logits, formula (166.2)

def clip_loss(logits):                                              # formulas (166.3)-(166.5)
    labels = torch.arange(logits.shape[0])
    return 0.5 * (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels))

model = CLIP()
opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.1)
N = 256
with torch.no_grad():
    print("loss at init:", round(clip_loss(model(img_tr[:N], txt_tr[:N])).item(), 3), " ln N =", round(math.log(N), 3))
for step in range(600):
    idx = torch.randint(0, img_tr.shape[0], (N,))
    loss = clip_loss(model(img_tr[idx], txt_tr[idx]))
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 200 == 199:
        print(step + 1, round(loss.item(), 3), "scale", round(model.t.exp().item(), 2))

# 3) zero-shot classification, formula (166.9): class "prompts" are noise-free captions
with torch.no_grad():
    prompts = proto @ to_txt                                        # one caption per concept
    W = F.normalize(model.txt_enc(prompts), dim=-1)                 # (K, d) = classifier weights
    u = F.normalize(model.img_enc(img_te), dim=-1)
    pred = (u @ W.T).argmax(dim=-1)
    print("zero-shot accuracy:", (pred == y_te).float().mean().item())
    # 4) retrieval: for each test image, is its own caption the top match among 1000?
    img_q, txt_q, _ = make_pairs(1000)
    sims = F.normalize(model.img_enc(img_q), dim=-1) @ F.normalize(model.txt_enc(txt_q), dim=-1).T
    print("image->text recall@1:", (sims.argmax(1) == torch.arange(1000)).float().mean().item(),
          " recall@5:", (sims.topk(5, dim=1).indices == torch.arange(1000)[:, None]).any(1).float().mean().item())

Output with PyTorch 2.8 on CPU: the loss at initialization is 7.522, above \(\ln 256 = 5.545\), because random embeddings in a narrow cone with scale 14.3 give confident wrong answers. Training brings it to 0.202, 0.129 and 0.097 after 200, 400 and 600 steps, and the learned scale rises from 14.3 to 17.89, 19.48 and 20.42. Retrieval of the true caption among 1,000 is easy: recall@1 0.986, recall@5 1.0. Zero-shot classification with one noise-free “prompt” per concept reaches 0.66 (chance is 0.10), much lower than retrieval, although the concepts themselves can be told apart 98% of the time from the hidden vectors. One reading [Speculation]: the loss rewards telling individual pairs apart, not grouping them by class, and the prompt is a caption without any of the details that real captions carry, a small version of CLIP’s “single word versus sentence” gap. Exact values can differ slightly across PyTorch versions. [Likely]

Where this is used

ApplicationHow the shared space is used
Zero-shot and open-set image classificationFormula (166.9) with your own class list; add or rename a class by writing a sentence (session 170)
Text-to-image and image-to-image searchEmbed the image collection once, index the vectors, embed the query text; nearest neighbours by cosine (session 169)
Vision encoder of VLMsLLaVA and many others start from a CLIP (or SigLIP) image tower and feed its patch tokens to a language model (session 178)
Text-to-video retrievalCLIP4Clip embeds sampled frames with CLIP, pools them over time and matches against the text embedding [Certain] (session 208)
Video classification and action recognitionZero-shot labels such as “a video of a person {action}.” against averaged frame embeddings; works for objects and scenes, less for actions that need motion [Likely]
Key-moment detection and Video-RAGScore every sampled frame against a text query (“a goal is scored”) and keep the peaks; frame embeddings also serve as the index for retrieval-augmented video question answering (session 210)
Near-duplicate detection and fingerprintingImage-to-image cosine of CLIP embeddings is robust to crops, resizing and re-encoding; per-frame embeddings form a video fingerprint (session 209)
Open-vocabulary detection and segmentationRegion features are compared with CLIP text embeddings of arbitrary class names (session 168)
Data filtering and evaluationKeep web pairs whose CLIP image–text cosine is high (LAION, DataComp); CLIP scores also measure how well a generated image matches its prompt
Text conditioning of image generatorsStable Diffusion v1 feeds CLIP text-encoder outputs into the denoiser through cross-attention (session 228)

Common mistakes

  • Forgetting to normalize. Without L2 normalization the dot product grows with the vector length, and the model can lower the loss by inflating norms instead of aligning directions. Normalize both sides before the dot product, and again after averaging a prompt ensemble.
  • A fixed, small logit scale. With \(s = 1\) the loss cannot go far below \(\ln N\). Learn the scale (in log space), start at \(\ln(1/0.07)\) and cap it at 100.
  • Gradient accumulation as a “bigger batch”. It does not add negatives. Each micro-batch sees only its own. Use all-gather across GPUs, or a loss that does not need the full softmax.
  • Wrong labels after all-gather. On GPU \(r\), the positives sit at columns \(r n \dots r n + n - 1\), not \(0 \dots n - 1\). Using arange(n) trains every GPU except rank 0 on wrong pairs.
  • Duplicates in a batch. Two identical captions (“image”, “photo”) or two copies of the same picture make a “negative” that is actually a positive, and the loss pushes them apart. Deduplicate the data and drop boilerplate captions.
  • Wrong preprocessing. CLIP has its own normalization mean and standard deviation and a resize-plus-center-crop pipeline (session 163). ImageNet statistics give a silently worse model.
  • Bare labels at test time. Use a sentence template, add task context, fix ambiguous class names (“crane (bird)”), and ensemble templates in embedding space.
  • Treating zero-shot probabilities as calibrated. At \(s = 100\) they are extremely peaked. Use the cosine scores for ranking, and calibrate separately if you need confidence values.

Interview questions

Write the CLIP loss for a batch of \(N\) pairs.basic
Embed images and texts, L2-normalize, form logits \(\ell_{ij} = s\, \mathbf{u}_i^\top \mathbf{v}_j\) with a learned scale \(s\). The loss is the average of two cross-entropies with the diagonal as labels: one over the rows (each image picks its text among \(N\)) and one over the columns (each text picks its image among \(N\)): \(\mathcal{L} = \tfrac12(\text{CE}(\ell, \text{arange}(N)) + \text{CE}(\ell^\top, \text{arange}(N)))\). It is a symmetric InfoNCE loss with \(N - 1\) in-batch negatives per positive.
Why is the loss symmetric? What happens with only the image-to-text direction?core
The row softmax normalizes over texts and the column softmax over images; they are different problems with different hard negatives. With only rows, nothing stops one text embedding from being the nearest text to many images while not being well placed among images, which hurts text-to-image retrieval. CLIP is used in both directions, so it trains both. A one-sided loss usually still learns a useful space but is somewhat worse in the untrained direction [Likely].
What does the temperature do, and why is it learned and capped?core
Cosines are in \([-1, 1]\), so without scaling the softmax over \(N\) items stays near uniform and the loss cannot go far below \(\ln N\) (for \(N = 32{,}768\) and \(s = 1\) the best possible loss is 9.40 versus chance 10.40). A large scale sharpens the softmax and focuses the gradient on hard negatives; too large makes the loss a hard max, unstable and very sensitive to noisy pairs. CLIP learns \(t = \ln s\), starting at \(\ln(1/0.07) = 2.659\), and clips \(s\) at 100 because larger values made training unstable.
Why does CLIP need a batch of 32,768? What limits it?core
Each row is an \(N\)-way classification; more negatives make the task harder and the signal last longer. The InfoNCE bound \(I \ge \ln N - \mathcal{L}\) caps what a batch of \(N\) can measure at \(\ln N\) nats (10.4 for 32,768). The cost is the \(N \times N\) matrix: 1.07 billion logits, 4.29 GB in float32. CLIP shards it: each GPU all-gathers the embeddings and computes only its \(n \times N\) rows (and columns). Gradient accumulation does not help because it does not add negatives.
How does zero-shot classification work, and why “a photo of a {label}.”?basic
Embed one sentence per class with the text tower; the normalized embeddings are the rows of a linear classifier. For an image, compute cosines with every row, take the argmax, and optionally a softmax with the learned scale. The template helps because web captions are sentences, not single words, and because it removes ambiguity (crane the bird versus the machine). In the paper the template gave +1.3 points on ImageNet, ensembling 80 templates in embedding space another +3.5, almost 5 in total; the best model reaches 76.2% top-1 zero-shot.
Why did OpenAI choose a contrastive objective instead of caption generation?core
Efficiency. In their comparison, a transformer that generates captions learned zero-shot ImageNet 3× slower than predicting a bag of words, and the contrastive objective was another 4× more efficient. Generation must predict the exact words of a noisy web caption; the contrastive task only has to tell which caption in the batch belongs to the image, which is enough to learn visual concepts. Later work (CoCa, BLIP) combines both losses (session 173).
Give the gradient of the CLIP loss with respect to one logit, and explain hard negatives.advanced
\(\partial \mathcal{L} / \partial \ell_{ij} = (P_{ij} + Q_{ij} - 2\delta_{ij}) / (2N)\), with \(P\) the row softmax and \(Q\) the column softmax. On the diagonal it is \(\le 0\): the true pair is pulled together. Off the diagonal it equals the probability mass the wrong pair received, so the push is concentrated on confusing pairs (hard negatives) and nearly zero for easy ones. A lower temperature concentrates it further. Random web batches contain few hard negatives that differ only in count or word order, which is one reason CLIP misses those.
CLIP fails on “a mug to the left of a laptop” versus “a laptop to the left of a mug”. Why, and how would you fix it?advanced
The contrastive loss only needs to separate a caption from the other captions in a random batch, which almost never differ only in word order, so a bag-of-words representation is enough (the ARO paper shows retrieval benchmarks can be solved without order information; Winoground shows near-chance results). The final embedding is also one pooled vector, with weak spatial detail. Fixes: composition-aware hard negatives (swap word order, swap attributes) during training, as ARO proposes; or use a model that reads patch tokens with a language model (a VLM) or a fusion model with cross-attention for such queries.
What is the modality gap?advanced
In CLIP-like models, image embeddings and text embeddings occupy two separate regions of the shared sphere; a matched pair is close relative to other pairs but far in absolute terms. Liang et al. (2022) trace it to initialization (each randomly initialized tower maps inputs into a narrow cone, and the two cones differ) and to contrastive training at low temperature, which preserves the gap. Practical consequences: cosine values like 0.3 can be a strong match; image–image and text–text cosines are not on the same scale as image–text cosines; changing the gap affects zero-shot accuracy and fairness.
Dual encoder or fusion model for a product-search system over 100 million images?core
Dual encoder for the first stage: embed the 100 million images once offline, store vectors in an approximate nearest-neighbour index, and embed only the query at search time, so each query costs one text-tower pass plus a vector search. A fusion model would need one joint forward pass per (query, image) pair, 100 million per query. Use a fusion model or a VLM as a reranker on the top few hundred candidates, where fine details (counts, relations, small text) matter. The same two-stage design is used for text retrieval (session 60).
How would you adapt CLIP to a new domain, such as medical images?core
First try zero-shot with domain-specific prompts and check failure cases; CLIP is weak where its web data is thin (the paper reports poor zero-shot results on tumor detection in PatchCamelyon). If you have labels, a linear probe on frozen image features is cheap and strong. With paired domain data (for example X-rays and their reports), continue contrastive training of both towers, as ConVIRT did for medical images before CLIP. Keep the original preprocessing, use a small learning rate, and watch for losing the general concepts.

Exercises

  1. For the 3-pair batch of the worked example with \(s = 10\), compute the loss of column 2 (text 2 choosing among the three images).
  2. What is the CLIP loss of a model that outputs the same embedding for every input, for a batch of \(N = 1{,}024\)?
  3. What is the initial logit scale and the initial temperature of CLIP? What are the largest scale and the smallest temperature allowed?
  4. For \(N = 32{,}768\) pairs on \(G = 128\) GPUs, how many logits does each GPU compute for the image-to-text direction, and how many bytes is that in float32?
  5. With \(s = 100\), two classes have cosines 0.28 and 0.25 with an image. What are the two zero-shot probabilities (only these two classes)?
  6. You have 1,000 classes and 80 templates. How many text-tower passes does building the ensemble classifier take, and how many text-tower passes per test image afterwards?
Answerscheck
1. Column 2 has logits (1.0, 2.8, 2.0) for images 1, 2, 3. Softmax of the true image: \(e^{2.8}/(e^{1.0} + e^{2.8} + e^{2.0}) = 16.44 / 26.55 = 0.619\), loss \(-\ln 0.619 = 0.479\), the value in the worked example. 2. All logits are equal, so every softmax is uniform: \(\mathcal{L} = \ln 1{,}024 = 6.931\), whatever the scale. 3. Scale \(1/0.07 = 14.29\) (so \(t_0 = 2.659\)), temperature 0.07; the cap is scale 100, temperature 0.01. 4. Each GPU has \(n = 256\) images against 32,768 texts: 8,388,608 logits, 33.6 MB in float32 (the same again for the text-to-image direction). 5. Logits 28 and 25, difference 3: \(p = 1/(1 + e^{-3}) = 0.953\) and 0.047. 6. \(1{,}000 \times 80 = 80{,}000\) text passes, once; afterwards zero text passes per image, only one image-tower pass and a \(1{,}000 \times d\) matrix product.

Key takeaways

  • CLIP is a dual encoder: an image tower and a text tower, each followed by a linear projection and L2 normalization (166.1). The towers meet only in one dot product, so embeddings can be computed once and searched fast.
  • The loss is a symmetric InfoNCE over the \(N \times N\) matrix of scaled cosines (166.2 to 166.5): rows pick texts, columns pick images, the diagonal is the label. The gradient (166.6) pulls true pairs and pushes the confusing negatives hardest.
  • The scale \(s = 1/\tau\) is learned in log space, starts at 14.3 (\(\tau = 0.07\)) and is capped at 100 (166.7). Without it the loss cannot go far below \(\ln N\).
  • Big batches give more negatives and a higher ceiling on what the loss can measure, \(\ln N\) (166.8). CLIP used 32,768 pairs and sharded the similarity matrix across GPUs.
  • Zero-shot classification is a linear classifier whose rows are text embeddings of prompts (166.9); templates and ensembling in embedding space (166.10) added almost 5 points on ImageNet, reaching 76.2% without ImageNet labels.
  • CLIP is weak at counting, spatial relations and word order, small or unusual text, and fine-grained classes, is fooled by written words, has a modality gap and inherits web biases. These limits come from the loss, the single pooled vector and the data.

Key formulas

\[ \mathbf{u}_i = \frac{W_I f(x_i)}{\lVert W_I f(x_i) \rVert}, \quad \ell_{ij} = s\, \mathbf{u}_i^\top \mathbf{v}_j, \quad s = \min(e^{t}, 100) \]
\[ \mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_{i=1}^{N} \left[ \log \frac{e^{\ell_{ii}}}{\sum_j e^{\ell_{ij}}} + \log \frac{e^{\ell_{ii}}}{\sum_j e^{\ell_{ji}}} \right] \]
\[ I(X;Y) \ge \ln N - \mathcal{L}_{I \to T}, \qquad p(y = k \mid x) = \softmax_k\big(s\, \mathbf{u}^\top \mathbf{w}_k\big) \]

References for this session

  1. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  2. Jia et al. (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
  5. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  6. Jia et al. (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  7. van den Oord et al. (2018). Representation Learning with Contrastive Predictive Coding
  8. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  9. Zhang, Jiang, Miura, Manning & Langlotz (2020). Contrastive Learning of Medical Visual Representations from Paired Images and Text
  10. Wu, Xiong, Yu & Lin (2018). Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
  11. Schuhmann et al. (2022). LAION-5B: An open large-scale dataset for training next generation image-text models
  12. Gadre et al. (2023). DataComp: In search of the next generation of multimodal datasets
  13. Cherti et al. (2023). Reproducible scaling laws for contrastive language-image learning
  14. Thrush et al. (2022). Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
  15. Yuksekgonul, Bianchi, Kalluri, Jurafsky & Zou (2023). When and why vision-language models behave like bags-of-words, and what to do about it?
  16. Liang, Zhang, Kwon, Yeung & Zou (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
  17. Goh et al. (2021). Multimodal Neurons in Artificial Neural Networks (Distill)
  18. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  19. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training