CLIP: learning images and text together
CLIP trains an image encoder and a text encoder together so that a photo and its caption land on the same point of one shared vector space. The only training signal is a symmetric contrastive loss over a batch of image–text pairs, with a learned temperature. After training, you can classify images into any set of labels by writing the labels as sentences, with no labeled training images at all. This session builds the loss by hand, trains a tiny CLIP, and ends with what CLIP cannot do.
What you will learn
- The dual-encoder design: two towers, two projections, L2 normalization, one shared space
- The symmetric InfoNCE loss over an N × N similarity matrix, computed by hand, and its gradient (pull the diagonal, push the rest)
- Why the temperature must be learned, why it starts at 0.07 and why it is capped at 100
- Why contrastive training needs huge batches (the ln N bound) and how the N × N matrix is split across GPUs
- Zero-shot classification with prompts such as “a photo of a {label}.”, and prompt ensembling in embedding space
- The real CLIP recipe: 400 million web pairs, batch 32,768, 32 epochs, and the 76.2% zero-shot ImageNet result
- The limits: counting, spatial relations and word order, small text, typographic attacks, the modality gap and web-data bias
Why this matters
Before 2021, a vision model could only name the classes it was trained on: train on 1,000 ImageNet classes and you get exactly those 1,000 outputs. CLIP (Contrastive Language–Image Pre-training, Radford et al., 2021) removed that wall. It learns from 400 million images paired with the text that appeared next to them on the web, and it puts images and sentences into one shared vector space. After training, “which class is this photo?” becomes “which sentence is closest to this photo?”, and you can write any sentence you like.
This idea is everywhere in modern multimodal systems. The image encoder of many vision-language models (VLMs) started life as a CLIP encoder: LLaVA uses CLIP ViT-L/14 [Certain] (session 181). Text-to-image search, video search and near-duplicate detection run on CLIP-style embeddings (session 169). Stable Diffusion v1 conditions its image generator on the CLIP ViT-L/14 text encoder [Certain] (session 228). Datasets such as LAION were filtered with CLIP scores [Certain]. And the next session, session 167, is a direct answer to CLIP’s loss.
The previous sessions built strong image encoders: the Vision Transformer (session 164) with labels, and MAE and DINO without labels (session 165). Self-supervised features are good, but they are not connected to words. CLIP connects them. Technically, CLIP is the same in-batch contrastive training that session 60 used for text search, with one change: the two towers read different modalities.
Two encoders, one shared space
CLIP has two separate networks, often called towers:
- An image encoder \(f_\theta\). CLIP trained five ResNets and three Vision Transformers (ViT-B/32, ViT-B/16, ViT-L/14). For a ViT, the image feature is the output at the class token. [Certain]
- A text encoder \(g_\phi\): a 12-layer transformer, 512 wide with 8 heads in the base size (63 million parameters), reading lower-cased byte-pair-encoded text of at most 76 tokens. The text feature is the top-layer activation at the end-of-sequence token [EOS]. [Certain]
The two features have different sizes (for ViT-L/14, 1,024 for the image and 768 for the text). Two learned linear projections \(W_I\) and \(W_T\) map both to the same embedding size \(d\), and each result is divided by its length (L2 normalization), so every embedding is a point on the unit sphere:
- Read the symbols
- \(x_i, y_j\)
- the \(i\)-th image and the \(j\)-th text of the batch
- \(f_\theta, g_\phi\)
- image encoder and text encoder, with weights \(\theta\) and \(\phi\)
- \(W_I, W_T\)
- linear projections into the shared space of size \(d\)
- \(\mathbf{u}_i, \mathbf{v}_j\)
- unit-length image and text embeddings in \(\R^d\); their dot product is their cosine similarity, a number in \([-1, 1]\)
The model sizes, from the CLIP paper’s Table 20 [Certain]:
| Model | Image input | Vision transformer (layers, width, heads) | Text transformer (layers, width, heads) | Shared size \(d\) |
|---|---|---|---|---|
| ViT-B/32 | 224 | 12, 768, 12 | 12, 512, 8 | 512 |
| ViT-B/16 | 224 | 12, 768, 12 | 12, 512, 8 | 512 |
| ViT-L/14 | 224 | 24, 1024, 16 | 12, 768, 12 | 768 |
| ViT-L/14@336px | 336 | 24, 1024, 16 | 12, 768, 12 | 768 |
Notice what is not there: no layer reads the image and the text together. The towers meet only at the very end, through one dot product. This is called a dual encoder (or two-tower, or late-interaction) model. It has a big practical advantage: you can embed a billion images once, store the vectors, and later compare any new text against all of them with fast vector search. A model that reads image and text together (a fusion model, see session 172) must run once per pair.
Why not train the image model to write the caption, like a captioning model? The CLIP authors tried. A transformer that predicts the caption word by word learned zero-shot ImageNet classification 3 times slower than a model that only predicts which words occur (a bag of words), and swapping that prediction for the contrastive objective gave another 4 times better efficiency [Certain, their Figure 2]. Predicting the exact words of a web caption is a very hard task, full of details that do not help recognition. Picking which caption, among the ones in the batch, belongs to this image is easier, and it is enough to learn visual concepts.
The contrastive loss: every batch is a classification problem
Take a batch of \(N\) (image, text) pairs. Embed all images and all texts with (166.1), then build the \(N \times N\) matrix of scaled dot products, the logits:
- Read the symbols
- \(\ell_{ij}\)
- score of image \(i\) with text \(j\); the true pairs are on the diagonal \(i = j\)
- \(s\)
- the logit scale, a learned positive number
- \(\tau = 1/s\)
- the temperature; the two names describe the same knob
Now read each row as a classification problem: “image \(i\), which of these \(N\) texts is yours?” The correct class is \(i\). Use a softmax over the row and the usual cross-entropy (session 3). Then read each column the other way: “text \(j\), which of these \(N\) images is yours?”:
- Read the symbols
- \(\mathcal{L}_{I \to T}\)
- image-to-text loss: cross-entropy of every row, averaged
- \(\mathcal{L}_{T \to I}\)
- text-to-image loss: cross-entropy of every column, averaged
- \(\mathcal{L}_{\text{CLIP}}\)
- the symmetric loss that CLIP minimizes; the CLIP paper gives exactly this as numpy-like pseudocode [Certain]
This family of losses is called InfoNCE (from “noise-contrastive estimation”, van den Oord et al., 2018). Each positive is compared against \(N - 1\) negatives that cost nothing extra: they are simply the other pairs of the batch. A batch of \(N\) pairs gives \(N\) positives and \(N(N-1)\) negatives.
Why symmetric? The two directions are different problems. A row softmax asks the image to prefer its caption over other captions; a column softmax asks the caption to prefer its image over other images. A model trained only on rows can still place a caption close to many images. CLIP is used in both directions (search images with text, label images with text), so it trains both. Removing one direction is a common interview follow-up; it usually still works but is a little worse [Likely].
Suppose the cosine similarities of a batch of three pairs are (rows = images, columns = texts):
Row 1 has logits (3.0, 1.0, 0.5). Softmax: \(e^{3} / (e^{3} + e^{1} + e^{0.5}) = 20.09 / 24.45 = 0.821\). Its loss is \(-\ln 0.821 = 0.197\). Row 3 has logits (0.2, 2.0, 2.5): its own caption is only slightly ahead of caption 2, so the probability is 0.586 and the loss is 0.535. Doing all three rows: \(\mathcal{L}_{I \to T} = (0.197 + 0.388 + 0.535)/3 = 0.373\). The three columns give \(\mathcal{L}_{T \to I} = (0.204 + 0.479 + 0.408)/3 = 0.364\). So \(\mathcal{L}_{\text{CLIP}} = 0.368\).
Now the same cosines with \(s = 1\): every row is almost uniform (row 1 gives 0.385, 0.315, 0.300), and the loss is 0.990, close to the “know nothing” value \(\ln 3 = 1.099\). The geometry has not changed, only the scale. Cosines live in \([-1, 1]\), so without a large scale the softmax cannot become confident. Hold this thought for the temperature section.
Figure 2 runs the whole forward pass of one training step in your browser, one operation at a time, on a batch of 4 pairs. The projected features are 3 numbers each, so you can check every product by hand. The batch has a near-duplicate: a dog photo and a puppy photo, whose captions fit both images. Watch formula (166.1) normalize each vector, the \(4 \times 4\) matrix fill in cell by cell, the temperature turn cosines into logits (166.2), and the row and column softmaxes turn logits into the two losses (166.3) and (166.4).
- input: projected features \(\tilde{\mathbf{x}}_i = W_I f_\theta(x_i)\), \(\tilde{\mathbf{y}}_j = W_T g_\phi(y_j)\); temperature \(\tau\)
- for \(i = 1 \dots N\): \(\mathbf{u}_i \leftarrow \tilde{\mathbf{x}}_i / \lVert \tilde{\mathbf{x}}_i \rVert\) // (166.1)
- for \(j = 1 \dots N\): \(\mathbf{v}_j \leftarrow \tilde{\mathbf{y}}_j / \lVert \tilde{\mathbf{y}}_j \rVert\)
- for each \(i, j\): \(C_{ij} \leftarrow \mathbf{u}_i \cdot \mathbf{v}_j\) // cosine
- \(\ell \leftarrow C / \tau\) // (166.2), \(s = 1/\tau\)
- for each row \(i\): \(P_{i,:} \leftarrow \softmax(\ell_{i,:})\); \(\ \mathcal{L}^{r}_i \leftarrow -\ln P_{ii}\) // (166.3)
- for each column \(j\): \(Q_{:,j} \leftarrow \softmax(\ell_{:,j})\); \(\ \mathcal{L}^{c}_j \leftarrow -\ln Q_{jj}\) // (166.4)
- \(\mathcal{L} \leftarrow \tfrac12\big(\text{mean}_i\, \mathcal{L}^{r}_i + \text{mean}_j\, \mathcal{L}^{c}_j\big)\) // (166.5)
How to read Figure 2
Press Play, or Step through it. At \(\tau = 0.07\) (the CLIP starting value, \(s = 14.29\)) the cat and car rows and columns are easy: their losses are 0.006 or less. The dog row gives 0.118 of its probability to the puppy caption, and the puppy row gives 0.322 to the dog caption (loss 0.390). The dog caption column gives 0.245 to the puppy photo (loss 0.281). The final loss is 0.124. Now move \(\tau\) to 1 while the animation is at the end: the same cosines give 0.985, close to the chance value \(\ln 4 = 1.386\), because logits that differ by less than 2 cannot make a confident softmax. At \(\tau = 0.01\) the loss falls to 0.001. Lowering \(\tau\) never changes which caption wins a row; it only changes how sure the softmax is.
Figure 3 lets you redo this example and a harder batch with a near-duplicate: a photo of a puppy on a beach and a photo of a dog on grass, whose captions are both plausible for both photos.
How to read Figure 3
With the worked-example batch at \(s = 10\) the readout shows the numbers of the box above: 0.373, 0.364 and 0.368. Now pick the 4-pair batch. Row 1 (the dog photo) gives only 0.528 to its own caption and 0.354 to the puppy caption: the near-duplicate takes a large share and gets the largest push down. The red car row is easy (0.165 loss). Slide \(s\) to 100: the easy rows reach loss 0.000, and the near-duplicate rows still cost 0.018, because 0.31 versus 0.27 is a small cosine gap. Slide \(s\) to 1: the total loss is 1.241, close to \(\ln 4 = 1.386\), whatever the cosines are.
What the gradient does
The derivative of the loss with respect to each logit has a simple form. Let \(P_{ij}\) be the row softmax (the probability that image \(i\) picks text \(j\)) and \(Q_{ij}\) the column softmax (the probability that text \(j\) picks image \(i\)):
- Read the symbols
- \(\delta_{ij}\)
- 1 if \(i = j\) (a true pair), else 0
- \(P_{ij}, Q_{ij}\)
- row and column softmax probabilities of logit \(\ell_{ij}\)
- \(\partial \mathcal{L} / \partial \mathbf{u}_i\)
- the gradient on the image embedding (before the small correction from the normalization, which removes the part along \(\mathbf{u}_i\))
On the diagonal the gradient is negative (it is at most 0, since \(P_{ii}, Q_{ii} \le 1\)): gradient descent raises the score of the true pair, pulling \(\mathbf{u}_i\) towards \(\mathbf{v}_i\). Off the diagonal it is positive, and proportional to how much probability the wrong pair got: the model pushes hardest on the negatives it currently confuses, the hard negatives. Easy negatives with probability near 0 get almost no gradient. This is why the composition of a batch matters so much.
- input: a batch of \(N\) images \(x_{1..N}\) and their texts \(y_{1..N}\); weights \(\theta, \phi, W_I, W_T\); log-scale \(t\)
- \(F_I \leftarrow f_\theta(x)\), \(F_T \leftarrow g_\phi(y)\) // features, shapes \((N, d_i)\) and \((N, d_t)\)
- \(U \leftarrow \text{normalize}(F_I W_I)\), \(V \leftarrow \text{normalize}(F_T W_T)\) // formula (166.1), shape \((N, d)\)
- \(s \leftarrow \min(e^{t}, 100)\); \(\ \ell \leftarrow s\, U V^\top\) // formula (166.2), shape \((N, N)\)
- labels \(\leftarrow (0, 1, \dots, N-1)\) // the diagonal
- \(\mathcal{L} \leftarrow \tfrac12\big(\text{CE}(\ell, \text{labels}) + \text{CE}(\ell^\top, \text{labels})\big)\) // rows, then columns: (166.3) to (166.5)
- backpropagate into \(\theta, \phi, W_I, W_T\) and \(t\); take an AdamW step // the temperature is a weight
The learned temperature
The worked example showed the problem: a cosine is at most 1, so with \(s = 1\) a logit can differ from another by at most 2, and the softmax over \(N\) texts can never be confident. Even a perfect geometry has a high loss. If the \(N\) embeddings are as spread out as possible (the corners of a regular simplex, with cosine \(-1/(N-1)\) between all wrong pairs) the smallest possible loss is
For \(N = 32{,}768\) and \(s = 1\) this is 9.397, barely below the chance value \(\ln N = 10.397\). With \(s = 14.3\) it is 0.020, and with \(s = 100\) it is essentially 0. So the scale has to be large. But how large? Too large, and the softmax becomes a hard maximum: gradients go almost entirely to the single hardest negative, and noisy web pairs (captions that do not really describe the image) get huge penalties. CLIP does not tune it by hand. It learns it, through a log-parameterization that keeps it positive:
- Read the symbols
- \(t\)
- the trained parameter: the log of the logit scale; gradient descent updates it like any weight, with \(\partial \mathcal{L}/\partial t = \sum_{ij} \frac{\partial \mathcal{L}}{\partial \ell_{ij}}\, \ell_{ij}\)
- \(t_0\)
- the initial value: temperature 0.07 (scale 14.3), taken from earlier instance-discrimination work (Wu et al., 2018) [Certain]
- 100
- the cap: CLIP clips the scale at 100 (temperature 0.01) because larger values made training unstable [Certain]
The Hugging Face CLIP config stores logit_scale_init_value = 2.6592, which is exactly \(\ln(1/0.07)\) [Certain]. ALIGN (Jia et al., 2021), trained at the same time on 1.8 billion noisier pairs, made the same choice: its temperature starts at 1.0 and is learned, and the authors report that learning it worked as well as a manual sweep [Certain].
Figure 4 trains a tiny CLIP: 8 images and 8 captions with different random features (8 numbers each, as if from two different sensors), two linear towers into a 2-D shared space, and the loss (166.5). It runs three versions side by side, which differ only in the temperature.
How to read Figure 4
At step 0 the embeddings are random and the loss of the two runs with \(s = 14.3\) is 9.509, far above \(\ln 8 = 2.079\): confident and wrong is worse than uniform. Drag the step slider. The fixed \(s = 1\) run arranges the pairs perfectly (all 8 images find their caption, mean wrong-pair cosine −0.143, the most spread possible on a circle) but its loss stays just above 1.315, the lowest possible value for 8 points on a circle at \(s = 1\). The learned run first lowers its scale (to about 10.2 near step 40), then raises it once the pairs are separated, reaching \(s \approx 31.6\) and loss 0.025 at step 400, against 0.079 for the fixed \(s = 14.3\) run. The temperature adapts to how well the model currently does.
In the real CLIP the learned scale climbs to the cap: the released OpenAI checkpoints have a logit scale of 100 [Likely]. That is one reason zero-shot probabilities from CLIP are often very peaked.
Why the batch must be huge
CLIP used a batch of 32,768 pairs [Certain]. Why so large? Each row of the loss is a \(N\)-way classification. With more negatives, the task is harder, and the model must learn finer differences to solve it. There is also a clean bound. van den Oord et al. showed that InfoNCE is a lower bound on the mutual information between the two views (how many nats of information an image and its caption share):
- Read the symbols
- \(I(X; Y)\)
- mutual information between an image and its paired text, in nats
- \(\ln N\)
- the loss of a model that guesses uniformly over the batch (chance level)
- \(\mathcal{L}_{I \to T}\)
- the InfoNCE loss, never below 0, so the bound can never exceed \(\ln N\)
Even a perfect model can only “prove” \(\ln N\) nats with a batch of \(N\): \(\ln 256 = 5.5\), \(\ln 32{,}768 = 10.4\). A small batch saturates early: once the loss is near 0, the task gives almost no signal. A big batch keeps it informative longer. The price is the \(N \times N\) matrix. Its full size for \(N = 32{,}768\) is \(32{,}768^2 \approx 1.07\) billion logits, 4.29 GB in float32. Nobody stores it on one GPU. CLIP sharded the similarity computation: each GPU computes only the rows of its local images against all texts (and the columns of its local texts against all images) [Certain].
- input: on GPU \(r\): its local batch of \(n = N/G\) pairs
- compute local embeddings \(U_r, V_r\) of shape \((n, d)\) // towers run only on local data
- \(U_{\text{all}} \leftarrow \text{all-gather}(U_r)\), \(V_{\text{all}} \leftarrow \text{all-gather}(V_r)\) // shapes \((N, d)\); see the collectives session
- \(\ell^{\text{img}} \leftarrow s\, U_r V_{\text{all}}^\top\) // \((n, N)\): my images against every text
- \(\ell^{\text{txt}} \leftarrow s\, V_r U_{\text{all}}^\top\) // \((n, N)\): my texts against every image
- labels \(\leftarrow r \cdot n + (0, \dots, n-1)\) // where my positives sit among the \(N\) columns
- \(\mathcal{L}_r \leftarrow \tfrac12\big(\text{CE}(\ell^{\text{img}}, \text{labels}) + \text{CE}(\ell^{\text{txt}}, \text{labels})\big)\)
- backward; gradients reach the local towers through \(U_r, V_r\) (and, if the gather keeps gradients, through the gathered copies); then all-reduce the weight gradients as in data parallelism
The gather itself is cheap: \(32{,}768 \times 768\) numbers in bfloat16 is 50.3 MB per modality. The all-gather and all-reduce are explained in session 43 and session 44. Figure 5 shows how chance loss, the bound and the memory grow with \(N\).
How to read Figure 5
Keep the loss at 2 nats and lower \(N\): below \(N = 8\) the bound \(\ln N - L\) is negative, so the loss proves nothing. At \(N = 32{,}768\) the same loss proves at least 8.40 nats. Now look at memory: the full matrix is 4.29 GB, but with 256 GPUs each one holds a \(128 \times 32{,}768\) slice, 16.8 MB. Doubling \(N\) quadruples the full matrix but only doubles each slice if \(G\) doubles too.
The data and the recipe
The loss is simple; the scale is what made CLIP work. The facts from the paper [Certain]:
- Data. WIT (WebImageText): 400 million (image, text) pairs from public internet sources. The authors searched for pairs whose text contains one of 500,000 queries (words that appear at least 100 times in English Wikipedia, plus some bigrams and names), and kept at most 20,000 pairs per query to balance the concepts. The total word count is similar to WebText, the GPT-2 training set.
- Training. 32 epochs, batch 32,768, Adam with decoupled weight decay (AdamW, decay 0.2), 2,000 warm-up steps and a cosine schedule, mixed precision. Optimization tricks from session 4 all appear here.
- Compute. The largest ResNet (RN50x64) trained for 18 days on 592 V100 GPUs; the largest ViT (ViT-L/14) for 12 days on 256 V100 GPUs, then one extra epoch at 336-pixel resolution, giving ViT-L/14@336px, the model the paper calls “CLIP”.
The data was never released. Open replications followed: LAION-400M and LAION-5B built pairs from Common Crawl and filtered them with OpenAI’s CLIP itself; OpenCLIP trained CLIP models on them and measured scaling laws (Cherti et al.); DataComp turned dataset filtering into a benchmark [Certain]. ALIGN showed the other direction: 1.8 billion pairs with only minimal filtering, an EfficientNet-L2 image tower and a BERT-Large text tower, batch 16,384, and 76.4% zero-shot ImageNet accuracy [Certain]. Quantity can beat cleanliness, up to a point. Data quality and filtering for multimodal training are the topic of session 187.
Zero-shot classification with prompts
Here is the payoff. To classify an image into \(K\) classes, write each class as a sentence, embed the sentences with the text tower, and pick the sentence closest to the image:
- Read the symbols
- \(\mathbf{u}\)
- the unit image embedding of the test image
- \(\mathbf{w}_k\)
- the unit text embedding of the prompt for class \(k\)
- \(K\)
- the number of classes; you choose them at test time
Look at (166.9) again: it is a linear classifier whose weight rows are the text embeddings. Pretraining was a \(N\)-way classification against captions; zero-shot is a \(K\)-way classification against class sentences. Nothing new is trained. The text embeddings are computed once per class and cached, so classifying an image costs one image-tower pass and one \(K \times d\) matrix product.
Why a prompt and not just the label? Two reasons, both from the paper [Certain]. First, web captions are rarely a single word, so “dog” alone is unusual input for the text tower. Second, polysemy: one word, several meanings. ImageNet has “crane” the bird and “crane” the machine; in Oxford Pets, “boxer” is a dog breed but could also be an athlete. Using the template “A photo of a {label}.” improved ImageNet accuracy by 1.3 points. Task-specific templates help further: “A photo of a {label}, a type of pet.” for pets, “a satellite photo of a {label}.” for satellite images.
Prompt ensembling. Instead of one template, use \(M\) of them (“a photo of a big {label}.”, “a drawing of a {label}.”, …), and average their embeddings:
The average is taken in embedding space, not over probabilities, so the cost at test time is the same as one template. On ImageNet, ensembling 80 templates added 3.5 points over the single default prompt; prompt engineering and ensembling together gave almost 5 points [Certain]. With all this, the best CLIP model (ViT-L/14@336px) reaches 76.2% top-1 zero-shot on ImageNet, matching the original fully supervised ResNet-50 without using any of its 1.28 million labeled training images, and 95% top-5 [Certain]. Over 20 datasets, zero-shot CLIP matched the average performance of a 4-shot linear probe on its own features [Certain].
- input: class names \(\text{label}_{1..K}\), templates \(\text{template}_{1..M}\), a trained CLIP
- for \(k = 1 \dots K\): // once, offline
- embed every filled template \(\text{template}_m(\text{label}_k)\) with the text tower, normalize each
- \(\mathbf{w}_k \leftarrow\) normalize(mean of the \(M\) embeddings) // formula (166.10)
- stack the rows: \(W \in \R^{K \times d}\) // the zero-shot classifier
- for each test image \(x\): \(\mathbf{u} \leftarrow\) image embedding; scores \(\leftarrow W \mathbf{u}\) // cosine similarities
- predict \(\argmax_k\) scores; probabilities \(\leftarrow \softmax(s \cdot \text{scores})\) // formula (166.9)
Figure 6 is a toy version with synthetic embeddings (16 dimensions, 5 classes, 400 test images). Each template moves the class embeddings in its own way, and the bare label word moves them the most. These numbers are made up to show the mechanism; they are not CLIP results.
- \(\mathbf{u} \leftarrow\) unit image embedding of the test image // one image-tower pass
- for \(k = 1 \dots K\): \(c_k \leftarrow \mathbf{u} \cdot \mathbf{w}_k = \sum_{d=1}^{16} u_d\, w_{k,d}\) // cosine with prompt \(k\)
- \(p \leftarrow \softmax(s \cdot c)\), \(s = 100\) // (166.9)
- \(\hat{y} \leftarrow \argmax_k c_k\) // the prediction
How to read Figure 6
Press Play. Each step takes one class prompt and adds up its 16 products into one cosine bar (green products raise it, red ones lower it). Then the softmax turns the five cosines into probabilities, and the argmax picks the class. Image 12 is a cat. With the bare label the classifier says “car”; with “a photo of a {label}.” and with the ensemble it says “cat”. Over all 400 images the bare label gets 79.8%, the single template 86.5% and the ensemble 96.5%. Averaging the four templates cancels much of the noise that each template adds to each class, which is the same reason ensembling helped real CLIP. Notice also how peaked the probabilities are at \(s = 100\): a cosine gap of 0.05 already gives a probability ratio of \(e^{5} \approx 148\). Do not read CLIP probabilities as calibrated confidence.
What CLIP cannot do
CLIP’s strengths come from its loss and its data, and so do its weaknesses. The CLIP paper itself lists many of them [Certain unless tagged]:
| Limit | Evidence | Why it happens |
|---|---|---|
| Counting | Weak zero-shot results on counting objects in synthetic scenes (CLEVRCounts) | A caption rarely states an exact count, and a batch rarely holds two images that differ only in count, so the loss never needs it [Likely] |
| Spatial relations and word order | Winoground: two images and two captions with the same words in a different order; the models tested, CLIP among them, did not do much better than chance. ARO: CLIP-style models show “a severe lack of order sensitivity” | Retrieval on random web batches can be solved without word order. Yuksekgonul et al. show models can score well on standard retrieval without using order information, so they behave like a bag of words |
| Distance and geometry | Near random on the distance to the nearest car (KITTI Distance) | Such facts are almost never written in web captions |
| Fine-grained classes | Poor on car models, flower species, aircraft variants compared with task-specific models | Web captions name the general concept more often than the exact variant |
| Small text and out-of-distribution images | Good on rendered text, but only 88% on handwritten MNIST digits, below logistic regression on raw pixels | Almost no MNIST-like images in the data; at 224 pixels small text has very few pixels (session 180) |
| Typographic attacks | An apple with a paper label “iPod” is classified as an iPod (Goh et al., Distill 2021) | CLIP learned to read, and written words in the image can outweigh the object |
| Web-data bias | In the paper’s FairFace probe with added non-human and crime-related labels, 4.9% of faces were put in non-human classes, about 14% for images labeled Black; 16.5% of male images versus 9.8% of female images were put in crime-related classes | The model learns the associations of its web text. Class design matters: adding a “child” class sharply reduced such errors for people under 20 |
| The modality gap | Image embeddings and text embeddings occupy two separate regions (“cones”) of the shared space (Liang et al., 2022) | Each tower starts in a narrow cone at initialization, and contrastive training with a low temperature keeps the two apart |
The deeper reason behind several rows: the contrastive loss only asks “which caption in this batch?”. If no other caption in the batch differs only in count or in word order, the model gets the answer right without learning count or order. Hard negatives built on purpose (swapping words, changing numbers) help, and the ARO paper proposes exactly that [Certain]. Later sessions show how VLMs fix the rest: a language model that reads all patch tokens instead of one pooled vector (session 177), higher resolution, and better data. Common failure modes of full VLMs are collected in session 193.
The PyTorch version
A complete, deterministic CLIP on synthetic data. Ten hidden “concepts” are seen through two different random sensors: an image feature of size 32 and a text feature of size 24. Two small MLP towers map both to a 16-d shared space. We train with the symmetric loss and a learned temperature, then do zero-shot classification and retrieval. It runs on a CPU in a few seconds.
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) synthetic paired data: K concepts seen through two different "sensors" K, Z, DI, DT = 10, 8, 32, 24 # concepts, latent size, image / text feature sizes proto = torch.randn(K, Z) # one latent vector per concept to_img, to_txt = torch.randn(Z, DI), torch.randn(Z, DT) def make_pairs(n): y = torch.randint(0, K, (n,)) z = proto[y] + 0.5 * torch.randn(n, Z) # this pair's content img = z @ to_img + 0.3 * torch.randn(n, DI) # "pixels" of the image txt = z @ to_txt + 0.3 * torch.randn(n, DT) # "words" of its caption return img, txt, y img_tr, txt_tr, _ = make_pairs(4096) img_te, _, y_te = make_pairs(1000) # 2) two encoders and a learned temperature, formulas (166.1) and (166.7) class CLIP(nn.Module): def __init__(self, d_emb=16): super().__init__() self.img_enc = nn.Sequential(nn.Linear(DI, 64), nn.GELU(), nn.Linear(64, d_emb)) self.txt_enc = nn.Sequential(nn.Linear(DT, 64), nn.GELU(), nn.Linear(64, d_emb)) self.t = nn.Parameter(torch.tensor(math.log(1 / 0.07))) # log of the logit scale def forward(self, img, txt): u = F.normalize(self.img_enc(img), dim=-1) # unit image embeddings v = F.normalize(self.txt_enc(txt), dim=-1) # unit text embeddings s = self.t.exp().clamp(max=100.0) # scale = 1 / temperature return s * u @ v.T # (N, N) logits, formula (166.2) def clip_loss(logits): # formulas (166.3)-(166.5) labels = torch.arange(logits.shape[0]) return 0.5 * (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) model = CLIP() opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.1) N = 256 with torch.no_grad(): print("loss at init:", round(clip_loss(model(img_tr[:N], txt_tr[:N])).item(), 3), " ln N =", round(math.log(N), 3)) for step in range(600): idx = torch.randint(0, img_tr.shape[0], (N,)) loss = clip_loss(model(img_tr[idx], txt_tr[idx])) opt.zero_grad(); loss.backward(); opt.step() if step % 200 == 199: print(step + 1, round(loss.item(), 3), "scale", round(model.t.exp().item(), 2)) # 3) zero-shot classification, formula (166.9): class "prompts" are noise-free captions with torch.no_grad(): prompts = proto @ to_txt # one caption per concept W = F.normalize(model.txt_enc(prompts), dim=-1) # (K, d) = classifier weights u = F.normalize(model.img_enc(img_te), dim=-1) pred = (u @ W.T).argmax(dim=-1) print("zero-shot accuracy:", (pred == y_te).float().mean().item()) # 4) retrieval: for each test image, is its own caption the top match among 1000? img_q, txt_q, _ = make_pairs(1000) sims = F.normalize(model.img_enc(img_q), dim=-1) @ F.normalize(model.txt_enc(txt_q), dim=-1).T print("image->text recall@1:", (sims.argmax(1) == torch.arange(1000)).float().mean().item(), " recall@5:", (sims.topk(5, dim=1).indices == torch.arange(1000)[:, None]).any(1).float().mean().item())
Output with PyTorch 2.8 on CPU: the loss at initialization is 7.522, above \(\ln 256 = 5.545\), because random embeddings in a narrow cone with scale 14.3 give confident wrong answers. Training brings it to 0.202, 0.129 and 0.097 after 200, 400 and 600 steps, and the learned scale rises from 14.3 to 17.89, 19.48 and 20.42. Retrieval of the true caption among 1,000 is easy: recall@1 0.986, recall@5 1.0. Zero-shot classification with one noise-free “prompt” per concept reaches 0.66 (chance is 0.10), much lower than retrieval, although the concepts themselves can be told apart 98% of the time from the hidden vectors. One reading [Speculation]: the loss rewards telling individual pairs apart, not grouping them by class, and the prompt is a caption without any of the details that real captions carry, a small version of CLIP’s “single word versus sentence” gap. Exact values can differ slightly across PyTorch versions. [Likely]
Where this is used
| Application | How the shared space is used |
|---|---|
| Zero-shot and open-set image classification | Formula (166.9) with your own class list; add or rename a class by writing a sentence (session 170) |
| Text-to-image and image-to-image search | Embed the image collection once, index the vectors, embed the query text; nearest neighbours by cosine (session 169) |
| Vision encoder of VLMs | LLaVA and many others start from a CLIP (or SigLIP) image tower and feed its patch tokens to a language model (session 178) |
| Text-to-video retrieval | CLIP4Clip embeds sampled frames with CLIP, pools them over time and matches against the text embedding [Certain] (session 208) |
| Video classification and action recognition | Zero-shot labels such as “a video of a person {action}.” against averaged frame embeddings; works for objects and scenes, less for actions that need motion [Likely] |
| Key-moment detection and Video-RAG | Score every sampled frame against a text query (“a goal is scored”) and keep the peaks; frame embeddings also serve as the index for retrieval-augmented video question answering (session 210) |
| Near-duplicate detection and fingerprinting | Image-to-image cosine of CLIP embeddings is robust to crops, resizing and re-encoding; per-frame embeddings form a video fingerprint (session 209) |
| Open-vocabulary detection and segmentation | Region features are compared with CLIP text embeddings of arbitrary class names (session 168) |
| Data filtering and evaluation | Keep web pairs whose CLIP image–text cosine is high (LAION, DataComp); CLIP scores also measure how well a generated image matches its prompt |
| Text conditioning of image generators | Stable Diffusion v1 feeds CLIP text-encoder outputs into the denoiser through cross-attention (session 228) |
Common mistakes
- Forgetting to normalize. Without L2 normalization the dot product grows with the vector length, and the model can lower the loss by inflating norms instead of aligning directions. Normalize both sides before the dot product, and again after averaging a prompt ensemble.
- A fixed, small logit scale. With \(s = 1\) the loss cannot go far below \(\ln N\). Learn the scale (in log space), start at \(\ln(1/0.07)\) and cap it at 100.
- Gradient accumulation as a “bigger batch”. It does not add negatives. Each micro-batch sees only its own. Use all-gather across GPUs, or a loss that does not need the full softmax.
- Wrong labels after all-gather. On GPU \(r\), the positives sit at columns \(r n \dots r n + n - 1\), not \(0 \dots n - 1\). Using
arange(n)trains every GPU except rank 0 on wrong pairs. - Duplicates in a batch. Two identical captions (“image”, “photo”) or two copies of the same picture make a “negative” that is actually a positive, and the loss pushes them apart. Deduplicate the data and drop boilerplate captions.
- Wrong preprocessing. CLIP has its own normalization mean and standard deviation and a resize-plus-center-crop pipeline (session 163). ImageNet statistics give a silently worse model.
- Bare labels at test time. Use a sentence template, add task context, fix ambiguous class names (“crane (bird)”), and ensemble templates in embedding space.
- Treating zero-shot probabilities as calibrated. At \(s = 100\) they are extremely peaked. Use the cosine scores for ranking, and calibrate separately if you need confidence values.
Interview questions
Write the CLIP loss for a batch of \(N\) pairs.basic
Why is the loss symmetric? What happens with only the image-to-text direction?core
What does the temperature do, and why is it learned and capped?core
Why does CLIP need a batch of 32,768? What limits it?core
How does zero-shot classification work, and why “a photo of a {label}.”?basic
Why did OpenAI choose a contrastive objective instead of caption generation?core
Give the gradient of the CLIP loss with respect to one logit, and explain hard negatives.advanced
CLIP fails on “a mug to the left of a laptop” versus “a laptop to the left of a mug”. Why, and how would you fix it?advanced
What is the modality gap?advanced
Dual encoder or fusion model for a product-search system over 100 million images?core
How would you adapt CLIP to a new domain, such as medical images?core
Exercises
- For the 3-pair batch of the worked example with \(s = 10\), compute the loss of column 2 (text 2 choosing among the three images).
- What is the CLIP loss of a model that outputs the same embedding for every input, for a batch of \(N = 1{,}024\)?
- What is the initial logit scale and the initial temperature of CLIP? What are the largest scale and the smallest temperature allowed?
- For \(N = 32{,}768\) pairs on \(G = 128\) GPUs, how many logits does each GPU compute for the image-to-text direction, and how many bytes is that in float32?
- With \(s = 100\), two classes have cosines 0.28 and 0.25 with an image. What are the two zero-shot probabilities (only these two classes)?
- You have 1,000 classes and 80 templates. How many text-tower passes does building the ensemble classifier take, and how many text-tower passes per test image afterwards?
Answerscheck
Key takeaways
- CLIP is a dual encoder: an image tower and a text tower, each followed by a linear projection and L2 normalization (166.1). The towers meet only in one dot product, so embeddings can be computed once and searched fast.
- The loss is a symmetric InfoNCE over the \(N \times N\) matrix of scaled cosines (166.2 to 166.5): rows pick texts, columns pick images, the diagonal is the label. The gradient (166.6) pulls true pairs and pushes the confusing negatives hardest.
- The scale \(s = 1/\tau\) is learned in log space, starts at 14.3 (\(\tau = 0.07\)) and is capped at 100 (166.7). Without it the loss cannot go far below \(\ln N\).
- Big batches give more negatives and a higher ceiling on what the loss can measure, \(\ln N\) (166.8). CLIP used 32,768 pairs and sharded the similarity matrix across GPUs.
- Zero-shot classification is a linear classifier whose rows are text embeddings of prompts (166.9); templates and ensembling in embedding space (166.10) added almost 5 points on ImageNet, reaching 76.2% without ImageNet labels.
- CLIP is weak at counting, spatial relations and word order, small or unusual text, and fine-grained classes, is fooled by written words, has a modality gap and inherits web biases. These limits come from the loss, the single pooled vector and the data.
Key formulas
References for this session
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Jia et al. (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Jia et al. (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- van den Oord et al. (2018). Representation Learning with Contrastive Predictive Coding
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Zhang, Jiang, Miura, Manning & Langlotz (2020). Contrastive Learning of Medical Visual Representations from Paired Images and Text
- Wu, Xiong, Yu & Lin (2018). Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
- Schuhmann et al. (2022). LAION-5B: An open large-scale dataset for training next generation image-text models
- Gadre et al. (2023). DataComp: In search of the next generation of multimodal datasets
- Cherti et al. (2023). Reproducible scaling laws for contrastive language-image learning
- Thrush et al. (2022). Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
- Yuksekgonul, Bianchi, Kalluri, Jurafsky & Zou (2023). When and why vision-language models behave like bags-of-words, and what to do about it?
- Liang, Zhang, Kwon, Yeung & Zou (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Goh et al. (2021). Multimodal Neurons in Artificial Neural Networks (Distill)
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training