compare vision features with linear probes
In this lab you run three frozen image encoders, DINOv2, CLIP and SigLIP 2, on the same CIFAR-10 images and measure what their features know. You train linear probes the right way, compare accuracy from 1 to 500 labels per class and against zero-shot, test whether the gaps are real, measure the speed, and probe the patch tokens on a simple localization task. Then you decide which encoder you would put in a VLM.
What you will learn
- Which output vector to take from each encoder: DINOv2’s CLS, CLIP’s CLS before the projection, SigLIP’s attention-pooled output
- How to extract and cache frozen features once, with each model’s own normalization and no center crop for dense tasks
- How to train a linear probe properly: standardize, sweep the L2 strength on a validation split, test once
- How accuracy grows from 1 to 500 labels per class, and how many labels zero-shot is worth
- How to tell a real gap from noise with the paired bootstrap on the same test images
- How patch size sets the token count and the compute per image (about 46 against 35 GFLOPs)
- How to probe dense patch tokens on a localization task, and why DINOv2 leads there by a wide margin
- How to argue for an encoder choice in a VLM: SigLIP 2 by default, fused with DINOv2 when spatial detail matters
Why this matters
Every vision-language model starts with one decision: which image encoder do we use? The encoder turns pixels into vectors, and the language model can only reason about what those vectors contain. In this lab you take three famous frozen encoders, DINOv2, CLIP and SigLIP 2, run them on the same images, and measure what their features know with the simplest possible test: a linear probe.
A linear probe is one linear layer trained on top of a frozen encoder. The encoder does not change. If a single matrix can read the class of the image from the features, the information is there and it is easy to get out. Because the probe is so small, it is cheap (it trains in well under a second here), it has almost no hyper-parameters, and it compares encoders fairly: all three get exactly the same reader. This is the standard way papers report feature quality; both the CLIP paper and the DINOv2 paper do it [Certain].
You met the three encoders in the last sessions. DINOv2 learned from images alone, by matching a teacher network across crops and masked patches (session 165). CLIP learned to match images with their captions using a softmax contrastive loss (session 166). SigLIP 2 uses a sigmoid loss instead, plus captioning and self-distillation losses to improve its patch features (session 167). In the previous lab you used CLIP and SigLIP without any labels at all (session 170). Here you add a few labels and see how far each encoder gets.
You will:
- Look inside each encoder: how many tokens it makes, and which vector you should take as “the image feature” (Figure 1).
- Extract features once for 5,000 training and 2,000 test images of CIFAR-10, and cache them.
- Train linear probes the right way: standardize, sweep the regularization on a validation split, and test once (Figure 2).
- Compare accuracy with 1 to 500 labels per class, against zero-shot, and test whether the differences are real (Figure 3).
- Measure speed: compute per image, images per second, and probe training time.
- Probe the patch tokens on a simple localization task: which part of the image shows which object? (Figure 4.)
- Decide which encoder you would put in a VLM, and why (Figure 5).
We ran every script on this page ourselves, on an Apple M5 Pro laptop (PyTorch MPS backend), and the numbers in the tables are our measurements. Your numbers on a CUDA GPU will match closely for accuracy and will differ for speed.
Setup
Hardware. Any machine with 16 GB of memory. One NVIDIA GPU should make the feature extraction take about a minute per encoder [Likely]; on an Apple laptop it took us a few minutes per encoder; on a CPU it takes longer but works. The probes themselves run on the CPU in seconds.
pip install torch transformers datasets numpy
We used PyTorch 2.15 and transformers 5.23. The CLIP checkpoint on the Hub has only a pytorch_model.bin file (no safetensors), which recent transformers versions still load [Certain].
The three encoders. We take the base size of each (ViT-B: 12 layers, width 768), so the comparison is about the training method, not the model size. We counted the parameters of each vision tower by loading it.
| Encoder (Hub id) | Trained with | Patch, tokens at 224 px | Vision parameters | Global feature we probe |
|---|---|---|---|---|
DINOv2 ViT-B/14facebook/dinov2-base | Images only: self-distillation (DINO + iBOT), 142M curated images | 14 px, \(16 \times 16 = 256\) patches + 1 CLS | 86.58M | CLS token after the final LayerNorm (768) |
CLIP ViT-B/16openai/clip-vit-base-patch16 | 400M image–text pairs, softmax contrastive loss | 16 px, \(14 \times 14 = 196\) patches + 1 CLS | 85.80M (+0.39M projection) | CLS after post_layernorm (768), before the projection to 512 |
SigLIP 2 ViT-B/16google/siglip2-base-patch16-224 | Image–text pairs, sigmoid loss + captioning + self-distillation and masked prediction | 16 px, \(14 \times 14 = 196\) patches, no CLS | 92.88M (7.09M of it is the pooling head) | Output of the attention-pooling head (768) |
DINOv2’s official recipe trains on images of 518 px, but the model works at any size that is a multiple of 14; at 224 px it makes 256 patches [Certain]. SigLIP 2 adds the captioning decoder (LocCa) and, for the last 20% of training, self-distillation and masked-prediction losses; the paper says these improve localization and dense features [Certain].
The data. CIFAR-10 (uoft-cs/cifar10 on the Hub): 50,000 training and 10,000 test images of 32 × 32 pixels in 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). We draw a balanced subset: 500 images per class for training (5,000) and 200 per class for testing (2,000), with fixed seeds. CIFAR-10 is small and easy, which is the point: it runs on a laptop, and the differences that remain are still informative. Every image is resized to 224 × 224 with bicubic interpolation before it enters an encoder.
The table you will fill. Our results are in Step 4; copy the empty version into your notes and fill it with your own run.
| Encoder | Probe, 500 labels per class | Probe, 16 labels per class | Zero-shot | Patch accuracy (localization) | GFLOPs per image | Images per second |
|---|---|---|---|---|---|---|
| DINOv2 B/14 | not possible (no text tower) | |||||
| CLIP B/16 | ||||||
| SigLIP 2 B/16 |
Step 1: what is “the feature” of an image?
A ViT cuts the image into square patches, turns each patch into a token, and runs a transformer over all of them (session 164). At the output you have one vector per token. The number of patch tokens is
- Read the symbols
- \(H\)
- the image side in pixels (224 here)
- \(p\)
- the patch side in pixels: 14 for DINOv2, 16 for CLIP and SigLIP 2
- \(N\)
- the number of patch tokens; DINOv2 and CLIP add one extra CLS token
For classification we need one vector per image. Call the output tokens \(\mathbf{h}_1, \dots, \mathbf{h}_N \in \R^{d}\) with \(d = 768\). There are three common ways to pool them:
- Read the symbols
- \(\mathbf{h}_{\text{CLS}}\)
- the output of the extra “class” token that DINOv2 and CLIP were trained to use as the image summary
- \(\mathbf{z}_{\text{mean}}\)
- the plain average of the patch tokens
- \(\mathbf{q}\)
- one learned query vector (a “probe” token) in SigLIP’s multihead attention pooling (MAP) head; it attends to all patches
- \(W_Q, W_K, W_V\)
- the head’s projections (the real head has 12 attention heads; one is shown); its output goes through an MLP
Use the pooling the model was trained with. CLIP’s loss only ever saw the CLS token (after a LayerNorm and a projection), so the CLS is where its knowledge is collected. SigLIP 2 has no CLS token at all; its loss saw the MAP output. DINOv2 trains both its CLS token (the DINO loss) and its patch tokens (the iBOT loss), and its paper reports linear probes on the CLS token, optionally concatenated with the mean of the patch tokens [Certain]. For CLIP, the CLIP paper probes the features before the final projection to 512 [Certain]. Figure 1 shows the three paths.
How to read Figure 1
Switch between the encoders. DINOv2 uses smaller patches (14 px), so at the same image size it makes 256 patch tokens instead of 196: a finer grid for dense tasks, and about 32% more compute. CLIP has a CLS token that collects the summary; SigLIP 2 has none and uses a small attention-pooling head instead. All three give a 768-number global vector, so the probes in Step 3 have the same size: \(768 \times 10 + 10 = 7{,}690\) weights.
Step 2: extract the features once
The encoders are frozen, so their outputs never change. Compute them once, store them on disk, and run every probe from the cache. A probe then costs a few small matrix products instead of a pass through an 86M-parameter network. This is also how the DINOv2 authors make their probe sweep cheap: they run the backbone once and feed the output to many linear classifiers [Certain].
- input: images \(x_1, \dots, x_n\), encoder \(f\) with its own mean and standard deviation for normalization
- for each batch of images do
- resize to \(224 \times 224\) (bicubic), scale to \([0, 1]\), subtract the model’s mean \(\mu\), divide by its std \(\sigma\) // each model has its own values
- \(\mathbf{h} \leftarrow f(x)\) with gradients off and the model in
eval()mode - keep the global feature (formula 171.2) in FP32 and the patch tokens in FP16
- end for
- save the arrays together with the labels and the exact image order
Three details matter more than they look.
- Normalization differs. DINOv2 uses the ImageNet mean and std (0.485, 0.456, 0.406 / 0.229, 0.224, 0.225), CLIP its own (0.481, 0.458, 0.408 / 0.269, 0.261, 0.276), and SigLIP 2 maps pixels to \([-1, 1]\) (mean and std 0.5). Feeding one model’s normalization to another costs accuracy quietly, with no error message.
- Resize, then do not crop. The DINOv2 image processor on the Hub resizes the short side to 256 and then crops the center 224 [Certain]. For classification that is fine. For the localization task in Step 6 a crop would cut off the borders of the image, so we resize directly to 224 for all three models and skip the processors.
- Know which layer you read. In Hugging Face, DINOv2’s
last_hidden_statealready has the final LayerNorm applied, CLIP’s does not (itspost_layernormis applied only to the pooled CLS), and SigLIP’s does. We apply CLIP’spost_layernormto its patch tokens ourselves so all three are treated the same.
Global features: \(7{,}000 \times 768 \times 4\ \text{bytes} = 21.5\ \text{MB}\) per encoder. Patch tokens for the 1,300 localization images of Step 6: DINOv2 has \(1{,}300 \times 256 \times 768 = 255.6\) million numbers, 511 MB in FP16; CLIP and SigLIP 2 have \(196\) patches, so 391 MB each. Patch features are large: for a real dataset you store them only when you need them, or you reduce them first.
Step 3: the linear probe, done properly
The probe is multinomial logistic regression: a linear layer followed by softmax, trained with cross-entropy (session 3). First standardize every feature dimension with statistics from the training set only:
Then fit the weights \(W \in \R^{d \times C}\) and bias \(\mathbf{b} \in \R^{C}\) by minimizing the regularized loss
- Read the symbols
- \(\tilde{\mathbf{z}}_j\)
- the standardized feature of training image \(j\) (formula 171.3)
- \(y_j\)
- its class, one of \(C = 10\)
- \(\lambda\)
- the L2 regularization strength: larger values keep the weights small and the probe simple
- \(\lVert W \rVert_F^2\)
- the sum of all squared weights
The loss (171.4) is convex: it has a single minimum and no bad local ones, so the result does not depend on the random seed or the starting point. We start from zero weights and use L-BFGS, a quasi-Newton optimizer that reaches the minimum in a few hundred steps on the full batch. The CLIP paper does the same with scikit-learn’s L-BFGS [Certain].
The only hyper-parameter is \(\lambda\). Never choose it on the test set. Hold out part of the training data as a validation split, try a grid of values, and keep the best:
Then refit on all training data with \(\lambda^\star\) and evaluate on the test set once. The CLIP paper sweeps 96 values between \(10^{-6}\) and \(10^{6}\) with a binary search [Certain]; six values, one per decade, are enough for us. The vectors \(\mu_{\text{train}}\) and \(\sigma_{\text{train}}\) in (171.3) hold one mean and one standard deviation per feature dimension.
- input: cached features \(Z_{\text{tr}}, Z_{\text{te}}\), labels, grid of \(\lambda\) values
- split the training rows into fit (80%) and validation (20%) // fixed seed
- for each \(\lambda\) in the grid do
- standardize with the fit rows’ mean and std (171.3); minimize (171.4) on the fit rows with L-BFGS
- record the validation accuracy
- end for
- \(\lambda^\star \leftarrow\) the best value (171.5); refit on all training rows, with new statistics
- return the test accuracy, computed one time
Why standardize? The 768 dimensions of these features have very different scales; for example, ViTs often have a few dimensions with huge values. Without standardization the same \(\lambda\) punishes some dimensions far more than others, and the optimizer converges slowly. With it, one \(\lambda\) grid works for all three encoders. Figure 2 lets you feel what \(\lambda\) and the number of labels do.
How to read Figure 2
With “Well separated”, even 1 label per class gives a high test accuracy: good features make probes easy. Switch to “Overlapping” and set 5 labels per class: with \(\lambda = 10^{-5}\) the boundaries chase the few points and the test accuracy is 73.7%; with \(\lambda = 10^{-1}\) it is 77.7%. Now set 60 labels: every \(\lambda\) from \(10^{-5}\) to \(10^{-1}\) gives 80.0%, so the choice hardly matters when labels are many. “Not linear” puts one class in a ring around another: no straight line separates them, so even 60 labels leave a big error. A linear probe measures what is linearly readable; a feature that needs a curved boundary scores low even if the information is there.
Step 4: the results, and are the differences real?
We probed several global features for each encoder, each time with 5,000 training labels and the \(\lambda\) chosen on validation. Our measurements on the 2,000 test images:
| Encoder | Feature | Test accuracy | \(\lambda^\star\) |
|---|---|---|---|
| DINOv2 B/14 | CLS | 98.5% | \(10^{-5}\) |
| CLS and patch mean, concatenated (1,536) | 98.35% | \(10^{-3}\) | |
| CLIP B/16 | CLS after post_layernorm (768) | 95.75% | \(10^{-2}\) |
| Projected embedding (512) | 95.25% | \(10^{-2}\) | |
| SigLIP 2 B/16 | MAP head output | 96.5% | \(10^{-1}\) |
| Mean of the patch tokens | 96.4% | \(10^{-5}\) |
The order is DINOv2 first, then SigLIP 2, then CLIP. These numbers agree with the papers. The DINOv2 paper reports 98.7% for its ViT-B/14 on the full CIFAR-10 (50,000 training images) [Certain], and the CLIP paper reports 96.2% for CLIP ViT-B/16 [Certain]. We used only 10% of the training labels and get within about half a point of both. For CLIP, the feature before the projection is better than the 512-number embedding, as the CLIP paper suggests: the projection was trained only to match text, and it throws some visual detail away.
How much noise is in a test accuracy?
A test set of \(n\) images gives an accuracy \(\hat{p}\) with a standard error of
But to compare two encoders, use the fact that they saw the same test images. Only the images where they disagree carry information. Let \(r^{A}_j, r^{B}_j \in \{0, 1\}\) say whether each encoder got image \(j\) right. The paired difference and its bootstrap interval are
- Read the symbols
- \(\hat{p}\)
- the measured test accuracy
- \(n\)
- the number of test images (2,000)
- \(r^{A}_j\)
- 1 if encoder A classified image \(j\) correctly, else 0
- resampled test sets
- draw \(n\) images with replacement, 2,000 times, and recompute \(\hat{\Delta}\) on each draw (the paired bootstrap)
One accuracy: DINOv2 has \(\hat{p} = 0.985\), so \(\operatorname{SE} = \sqrt{0.985 \times 0.015 / 2000} = 0.0027\), or \(\pm 1.96 \times 0.0027 = \pm 0.53\) points for a 95% interval. CLIP has \(\operatorname{SE} = 0.0045\) (\(\pm 0.88\) points).
DINOv2 against CLIP: DINOv2 alone is right on 72 images and CLIP alone on 17. The difference is \((72 - 17) / 2000 = 2.75\) points, with a bootstrap interval of 1.85 to 3.65 points. It is real. DINOv2 against SigLIP 2: 58 against 18 images, \(2\) points, interval 1.20 to 2.90: also real.
SigLIP 2 against CLIP: 54 against 39 images, \(0.75\) points, interval −0.10 to 1.75. The interval contains zero, so on this test set we cannot say that SigLIP 2’s probe beats CLIP’s. Do not write “SigLIP 2 is better” in a report from this run alone.
Few labels, and zero labels
In practice you often have only a handful of labels per class. We repeated the probe with 1, 2, 4, …, 128 labels per class (five random draws each, from a pool of 4,000 training images; \(\lambda\) still chosen on the 1,000 validation images) and 400 per class (the whole pool). We also ran the zero-shot classifier of the last lab, with the single prompt “a photo of a {label}.”: CLIP reaches 89.9% and SigLIP 2 95.05%, with no labels at all. The CLIP paper reports 91.6% zero-shot for B/16 on the full test set with an ensemble of prompts [Certain]; our single prompt is a little lower.
How to read Figure 3
Move the slider. With 1 label per class SigLIP 2 is ahead; its features were trained to line up with words, so one example per class already lands near the right group. From about 4 labels per class DINOv2 catches up and then leads, by about 1.3 to 2 points over SigLIP 2 at 16 to 128 labels. CLIP’s zero-shot accuracy equals a probe with somewhere between 4 and 8 labels per class; SigLIP 2’s zero-shot equals a probe with about 16. If your classes can be named in words, zero-shot is a strong free baseline; if they cannot (a defect type, a specific product), only the probe helps.
Step 5: speed
Accuracy is half the story. In a VLM, the encoder runs on every image, and in video on every frame. The cost of a ViT forward pass is mostly the linear layers, plus attention, which grows with the square of the number of tokens. Per layer and per token, the attention projections cost \(2 \cdot 4d^2\) FLOPs and the MLP (width \(4d\)) costs \(2 \cdot 8d^2\), so
- Read the symbols
- \(L\)
- the number of layers (12)
- \(N\)
- the number of tokens, including the CLS token if there is one
- \(d\)
- the width (768)
- \(4N^2 d\)
- the two attention matrix products \(QK^\top\) and \(AV\)
CLIP, \(N = 197\): linear part \(12 \times 24 \times 197 \times 768^2 = 33.46\) GFLOPs, attention \(12 \times 4 \times 197^2 \times 768 = 1.43\) GFLOPs. With the patch embedding (0.23 GFLOPs) the total is 35.1 GFLOPs.
DINOv2 at the same 224 px, \(N = 257\): \(43.66 + 2.43 + 0.23 =\) 46.3 GFLOPs, 1.32 times CLIP. The model has the same size; the cost comes only from the smaller patch (14 instead of 16 px). SigLIP 2 has 196 tokens and a pooling head of about 0.47 GFLOPs: 35.4 GFLOPs.
Our measured throughput (batch 64, FP32, best of three runs of 10 batches, Apple M5 Pro GPU through PyTorch MPS, while other jobs shared the machine):
| Encoder | Tokens | GFLOPs per image (171.8) | Images per second (measured) | Probe fit, \(5{,}000 \times 768\) (seconds, CPU) |
|---|---|---|---|---|
| DINOv2 B/14 | 257 | 46.3 | 98 | 0.20 |
| CLIP B/16 | 197 | 35.1 | 130 | 0.18 |
| SigLIP 2 B/16 | 196 | 35.4 | 125 | 0.08 |
Treat our absolute speeds as rough: the machine was shared, and an NVIDIA GPU in BF16 is much faster. The ratio is the useful part. The probe itself is nearly free: under half a second, thousands of times cheaper than extracting the features. That is why you can afford a full \(\lambda\) sweep, many few-shot draws and many encoders.
Step 6: dense features on a simple localization task
A VLM does not read only the global vector. It reads the patch tokens, one per region of the image (session 177). So we must also test whether each patch token knows what is at its own position, not just what the whole image shows.
The task: build a 224 × 224 mosaic from four CIFAR-10 images of four different classes, each resized to 112 × 112, one per quadrant. Then train a linear probe on single patch tokens to predict the class of the quadrant the patch lies in. Both grids align with the quadrants: \(112 / 14 = 8\) and \(112 / 16 = 7\) patches per side. We train on 1,000 mosaics built from training images and test on 300 mosaics built from test images. One probe sees every patch of every mosaic, so DINOv2’s probe has 256,000 training rows and the other two 196,000.
- Read the symbols
- \(\tilde{\mathbf{h}}_i\)
- the standardized output token of patch \(i\) (no pooling)
- \(y_{m,i}\)
- the class of the quadrant that contains patch \(i\) of mosaic \(m\)
- \(M, N\)
- the number of test mosaics (300) and of patches per mosaic
- \(\hat{S}_c, S_c\)
- the sets of patches predicted as class \(c\), and truly of class \(c\), over all test mosaics; the mean of \(\text{IoU}_c\) over the 10 classes is the mIoU
- input: mosaics with quadrant labels, a frozen encoder
- run the encoder; keep all \(N\) patch tokens of each mosaic // no pooling
- give every patch the label of its quadrant; flatten to one row per patch
- standardize (171.3), choose \(\lambda\) on held-out mosaics, fit (171.4) // the same probe as before
- predict every test patch; report patch accuracy, accuracy near the quadrant borders, and mIoU (171.9)
Our results on the 300 test mosaics. “Border” patches are the two rows and two columns of patches that touch the lines between quadrants; “interior” patches are all the others.
| Encoder | Grid | Patch accuracy | Interior patches | Border patches | mIoU |
|---|---|---|---|---|---|
| DINOv2 B/14 | 16 × 16 | 80.7% | 83.6% | 71.1% | 68.0% |
| CLIP B/16 | 14 × 14 | 66.7% | 70.3% | 56.7% | 50.4% |
| SigLIP 2 B/16 | 14 × 14 | 70.1% | 73.3% | 61.0% | 54.3% |
Now the gap is large: DINOv2 is about 11 points ahead of SigLIP 2 and 14 points ahead of CLIP, compared with 2 to 3 points on whole-image classification. This matches the published picture. In the DINOv2 paper, a linear layer on frozen DINOv2 ViT-g/14 patch tokens reaches 49.0 mIoU on ADE20k, against 39.3 for the much larger OpenCLIP ViT-G/14 [Certain]. CLIP’s loss only rewards the CLS token, so its patch tokens are free to mix information from the whole image. SigLIP 2’s extra local losses help: it beats CLIP here, as its paper reports for segmentation (77.1 against 74.5 mIoU on PASCAL for its So/14 model against CLIP L/14) [Certain]. Every encoder is worst at the borders, where a patch’s attention mixes two neighbors. Figure 4 shows real test mosaics with the real predictions of each probe.
How to read Figure 4
Step through the mosaics. DINOv2 has the highest patch accuracy on 5 of these 6 mosaics (on mosaic 1, SigLIP 2 is slightly ahead). Its errors sit mostly near the quadrant borders and in the hardest quadrant, often a look-alike pair such as cat and dog. CLIP’s maps are speckled: single patches inside a quadrant take the class of another quadrant, because CLIP’s patch tokens carry a lot of information from the whole image. In the PCA view, DINOv2 and SigLIP 2 already show the four quadrants as different colors, with no labels at all. CLIP’s map is mostly one color with a few very bright patches: a few tokens with extreme values stretch the color range. Such outlier tokens are typical of large ViTs trained without registers [Likely] (session 165).
Step 7: which encoder would you put in a VLM?
Put the measurements together.
- Global classification with labels: DINOv2 leads (98.5% against 96.5% and 95.75%). The lead over SigLIP 2 is real but small.
- Very few labels, or none: SigLIP 2 leads at 1 label per class, and only CLIP and SigLIP 2 can do zero-shot at all. Their features already live in a space shared with text.
- Dense, local information: DINOv2 leads by a wide margin (80.7% patch accuracy against 70.1% and 66.7%).
- Speed at the same resolution: CLIP and SigLIP 2 are about 1.3 times cheaper than DINOv2 because of the larger patch.
So which one goes into a VLM? The connector of a VLM (a small MLP in LLaVA, session 178) has to map visual tokens into the LLM’s word space. Features that are already aligned with language make that much easier, and in practice text-trained encoders win as the single encoder. The Prismatic VLMs study found that vision-language contrastive backbones (CLIP, SigLIP) are significantly better than the alternatives as a VLM’s only encoder, and that fusing DINOv2 with SigLIP features gives significant gains across the board; its final models use the fused DINOv2 + SigLIP features [Certain]. The “Eyes Wide Shut” study traced many VLM failures on simple visual questions to CLIP’s blind spots and improved them by mixing in DINOv2 features [Certain]. LLaVA uses a CLIP ViT-L/14, and PaliGemma uses a SigLIP So400m encoder [Certain].
Our answer, which you should defend in your own words: a SigLIP 2 encoder as the default, because its features are aligned with text (good for the connector and for few-shot and zero-shot use), it beats CLIP on dense features, and it is cheaper per token than DINOv2. Add DINOv2 features (concatenated per patch) when the product needs grounding, counting, segmentation or fine spatial detail, and you can pay for a second encoder. Pick DINOv2 alone for vision-only systems with labels: classification heads, retrieval of near-duplicates, segmentation and tracking. These are design judgments based on a small experiment and on the studies above, not laws [Likely]. Figure 5 lets you change the priorities.
How to read Figure 5
The default weights describe a general-purpose VLM: text alignment matters most, dense features next. Set “Zero-shot, text alignment” to 0 and “Dense patch features” to 5: DINOv2 wins, which is the vision-only case. Set the dense weight to 0 and compute to 5: SigLIP 2 wins clearly. CLIP wins only if compute is nearly all you care about, and then only by a hair (35.1 against 35.4 GFLOPs): SigLIP 2 is at least as good on every other measured axis.
The full program
This is the script that produced every number on this page (sections 1 to 5 in one file, about 10 minutes on our laptop). It is deterministic up to tiny floating-point differences between GPU types: fixed seeds for the subsets, mosaics, splits and bootstrap, and a convex probe that starts from zero.
# lab.py: compare frozen DINOv2, CLIP and SigLIP 2 features with linear probes (CIFAR-10). import json, time, numpy as np, torch, torch.nn.functional as F from datasets import load_dataset from transformers import AutoModel, AutoProcessor torch.manual_seed(0) dev = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu" ds = load_dataset("uoft-cs/cifar10") CLASSES = ds["train"].features["label"].names # ---------- 1. data: balanced subsets and 2x2 mosaics ---------- def subset(split, per_class, seed): y = np.array(ds[split]["label"]); g = np.random.default_rng(seed) idx = g.permutation(np.concatenate([g.permutation(np.where(y == c)[0])[:per_class] for c in range(10)])) x = np.stack([np.array(ds[split][int(i)]["img"].convert("RGB")) for i in idx]) return torch.from_numpy(x).permute(0, 3, 1, 2).contiguous(), torch.from_numpy(y[idx]) def mosaics(X, y, n, seed): # four images of four different classes g = np.random.default_rng(seed); by = [np.where(y.numpy() == c)[0] for c in range(10)] cls = np.stack([g.choice(10, 4, replace=False) for _ in range(n)]) tiles = torch.stack([torch.stack([X[g.choice(by[c])] for c in row]) for row in cls]) return tiles, torch.from_numpy(cls) # (n,4,3,32,32), (n,4): TL, TR, BL, BR Xtr, ytr = subset("train", 500, 0) # 5,000 labeled training images Xte, yte = subset("test", 200, 1) # 2,000 test images Mtr, Mtr_y = mosaics(Xtr, ytr, 1000, 2) Mte, Mte_y = mosaics(Xte, yte, 300, 3) def pixels(x, mean, std, size=224): # our own resize: no center crop x = F.interpolate(x.float() / 255, size=(size, size), mode="bicubic", align_corners=False).clamp(0, 1) return (x - torch.tensor(mean).view(1, 3, 1, 1)) / torch.tensor(std).view(1, 3, 1, 1) def mosaic_pixels(M, mean, std): t = pixels(M.flatten(0, 1), mean, std, 112).view(-1, 2, 2, 3, 112, 112) return torch.cat([torch.cat([t[:, r, 0], t[:, r, 1]], -1) for r in range(2)], -2) # ---------- 2. three frozen encoders, one feature function ---------- MODELS = {"dinov2": ("facebook/dinov2-base", [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), "clip": ("openai/clip-vit-base-patch16", [0.48145466, 0.4578275, 0.40821073], [0.26862954, 0.26130258, 0.27577711]), "siglip2": ("google/siglip2-base-patch16-224", [0.5] * 3, [0.5] * 3)} @torch.no_grad() def encode(name, m, pix): if name == "dinov2": h = m(pixel_values=pix).last_hidden_state # final LayerNorm applied return {"global": h[:, 0], "cls+mean": torch.cat([h[:, 0], h[:, 1:].mean(1)], 1), "patch": h[:, 1:]} o = m.vision_model(pixel_values=pix) if name == "clip": h = m.vision_model.post_layernorm(o.last_hidden_state) return {"global": o.pooler_output, "proj512": m.visual_projection(o.pooler_output), "patch": h[:, 1:]} return {"global": o.pooler_output, "mean": o.last_hidden_state.mean(1), "patch": o.last_hidden_state} def run_all(name, m, X, mean, std, keys, bs=100): out = {} for i in range(0, len(X), bs): f = encode(name, m, pixels(X[i:i + bs], mean, std).to(dev)) for k in keys: out.setdefault(k, []).append(f[k].float().cpu()) return {k: torch.cat(v) for k, v in out.items()} def sync(): if dev == "cuda": torch.cuda.synchronize() if dev == "mps": torch.mps.synchronize() R = {"device": dev}; FEAT = {} for name, (hub, mean, std) in MODELS.items(): m = AutoModel.from_pretrained(hub).to(dev).eval() vis = m if name == "dinov2" else m.vision_model keys = [k for k in encode(name, m, pixels(Xtr[:2], mean, std).to(dev)) if k != "patch"] FEAT[name] = {"tr": run_all(name, m, Xtr, mean, std, keys), "te": run_all(name, m, Xte, mean, std, keys)} FEAT[name]["mtr"] = torch.cat([encode(name, m, mosaic_pixels(Mtr[i:i + 50], mean, std).to(dev))["patch"].half().cpu() for i in range(0, len(Mtr), 50)]) FEAT[name]["mte"] = torch.cat([encode(name, m, mosaic_pixels(Mte[i:i + 50], mean, std).to(dev))["patch"].half().cpu() for i in range(0, len(Mte), 50)]) pix = pixels(Xtr[:64], mean, std).to(dev) # speed: best of 3 runs of 10 batches for _ in range(3): encode(name, m, pix) best = 1e9 for _ in range(3): sync(); t0 = time.perf_counter() for _ in range(10): encode(name, m, pix) sync(); best = min(best, time.perf_counter() - t0) R[name] = {"params_vision": sum(p.numel() for p in vis.parameters()), "img_per_s": 640 / best, "tokens": FEAT[name]["mtr"].shape[1]} if name != "dinov2": # zero-shot with one prompt proc = AutoProcessor.from_pretrained(hub) kw = dict(padding="max_length", max_length=64) if name == "siglip2" else dict(padding=True) tok = proc(text=[f"a photo of a {c}." for c in CLASSES], return_tensors="pt", **kw).to(dev) with torch.no_grad(): T = m.get_text_features(**tok) T = F.normalize(getattr(T, "pooler_output", T).float().cpu(), dim=-1) I = F.normalize(FEAT[name]["te"]["proj512" if name == "clip" else "global"], dim=-1) R[name]["zeroshot"] = ((I @ T.T).argmax(1) == yte).float().mean().item() print(name, R[name], flush=True) del m # ---------- 3. the linear probe ---------- def fit_probe(X, y, lam, C=10, iters=200): W = torch.zeros(X.shape[1], C, requires_grad=True); b = torch.zeros(C, requires_grad=True) opt = torch.optim.LBFGS([W, b], lr=1, max_iter=iters, history_size=20, line_search_fn="strong_wolfe") def closure(): opt.zero_grad() loss = F.cross_entropy(X @ W + b, y) + lam * (W * W).sum() loss.backward(); return loss opt.step(closure) return W.detach(), b.detach() def stdz(tr, *rest): # statistics from the training split only mu, sd = tr.mean(0), tr.std(0) + 1e-6 return [(t - mu) / sd for t in (tr, *rest)] LAMS = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1.0] perm = torch.randperm(5000, generator=torch.Generator().manual_seed(0)) va, pool = perm[:1000], perm[1000:] # 1,000 for choosing lambda, 4,000 for few-shot draws def choose_lambda(Ftr, tr_idx): # sweep lambda, score on the validation images A, V = stdz(Ftr[tr_idx], Ftr[va]) scores = {} for l in LAMS: W, b = fit_probe(A, ytr[tr_idx], l) scores[l] = ((V @ W + b).argmax(1) == ytr[va]).float().mean().item() return max(LAMS, key=scores.get) for name in MODELS: tr_f, te_f = FEAT[name]["tr"], FEAT[name]["te"] R[name]["probe"] = {} for k in tr_f: # every global feature variant, all 5,000 labels lam = choose_lambda(tr_f[k], pool) A, T = stdz(tr_f[k], te_f[k]) t0 = time.perf_counter(); W, b = fit_probe(A, ytr, lam); secs = time.perf_counter() - t0 right = ((T @ W + b).argmax(1) == yte) R[name]["probe"][k] = {"acc": right.float().mean().item(), "lam": lam, "fit_s": secs} if k == "global": R[name]["right"] = right.int().numpy() R[name]["fewshot"] = {} for k in [1, 2, 4, 8, 16, 32, 64, 128, 400]: # labels per class accs = [] for s in range(5 if k < 400 else 1): g = np.random.default_rng(100 + s); yp = ytr[pool].numpy() idx = pool[torch.from_numpy(np.concatenate([g.permutation(np.where(yp == c)[0])[:k] for c in range(10)]))] lam = choose_lambda(tr_f["global"], idx) A, T = stdz(tr_f["global"][idx], te_f["global"]) W, b = fit_probe(A, ytr[idx], lam) accs.append(((T @ W + b).argmax(1) == yte).float().mean().item()) R[name]["fewshot"][k] = [float(np.mean(accs)), float(np.std(accs))] print(name, R[name]["probe"], R[name]["fewshot"], flush=True) # ---------- 4. is the difference real? paired bootstrap ---------- g = np.random.default_rng(0); R["paired"] = {} for a, b in [("dinov2", "clip"), ("dinov2", "siglip2"), ("siglip2", "clip")]: d = R[a]["right"] - R[b]["right"] bs = [d[g.integers(0, len(d), len(d))].mean() for _ in range(2000)] R["paired"][f"{a}-{b}"] = {"diff": float(d.mean()), "ci95": [float(np.percentile(bs, 2.5)), float(np.percentile(bs, 97.5))], "only_a": int((d == 1).sum()), "only_b": int((d == -1).sum())} print(R["paired"], flush=True) # ---------- 5. dense features: label every patch of a mosaic ---------- def quadrant_labels(Y, grid): r = torch.arange(grid) q = (r[:, None] >= grid // 2).long() * 2 + (r[None, :] >= grid // 2).long() return Y[:, q.flatten()] for name in MODELS: Ptr, Pte = FEAT[name]["mtr"].float(), FEAT[name]["mte"].float() n, P, d = Ptr.shape; grid = int(P ** 0.5) Ltr, Lte = quadrant_labels(Mtr_y, grid).reshape(-1), quadrant_labels(Mte_y, grid) Xp, Xq = stdz(Ptr.reshape(-1, d), Pte.reshape(-1, d)) cut = 800 * P # lambda chosen on the last 200 training mosaics scores = {} for l in [1e-5, 1e-4, 1e-3]: W, b = fit_probe(Xp[:cut], Ltr[:cut], l, iters=150) scores[l] = ((Xp[cut:] @ W + b).argmax(1) == Ltr[cut:]).float().mean().item() lam = max(scores, key=scores.get) W, b = fit_probe(Xp, Ltr, lam, iters=150) pred = (Xq @ W + b).argmax(1).view(-1, grid, grid); gt = Lte.view(-1, grid, grid) ok = (pred == gt).float(); r = torch.arange(grid) border = ((r[:, None] - (grid / 2 - 0.5)).abs() <= 1) | ((r[None, :] - (grid / 2 - 0.5)).abs() <= 1) iou = [((pred == c) & (gt == c)).sum().item() / max(1, ((pred == c) | (gt == c)).sum().item()) for c in range(10)] R[name]["dense"] = {"grid": grid, "lam": lam, "patch_acc": ok.mean().item(), "border_acc": ok[:, border].mean().item(), "interior_acc": ok[:, ~border].mean().item(), "miou": float(np.mean(iou)), "examples": pred[:6].tolist()} # 3-component PCA of the patch tokens, for a picture Z = Pte[:6].reshape(-1, d); Z = Z - Pte.reshape(-1, d).mean(0) U, S, V = torch.pca_lowrank(Pte.reshape(-1, d) - Pte.reshape(-1, d).mean(0), q=3, center=False) pc = Z @ V; pc = (pc - pc.min(0).values) / (pc.max(0).values - pc.min(0).values) R[name]["dense"]["pca"] = (pc * 255).round().int().view(6, grid, grid, 3).tolist() print(name, {k: v for k, v in R[name]["dense"].items() if k not in ("examples", "pca")}, flush=True) R["mosaic_truth"] = Mte_y[:6].tolist(); R["mosaic_tiles"] = Mte[:6].tolist() for name in MODELS: R[name].pop("right") json.dump(R, open("results.json", "w"))
What to try next: probe the 4th-to-last layer as well as the last one (the DINOv2 protocol also tests the last four layers [Certain]); repeat at 448 px; swap in the SigLIP 2 So400m or DINOv3 checkpoints; replace CIFAR-10 by a fine-grained dataset (flowers, cars, aircraft), where the gaps between encoders are usually larger.
Where this is used
| Setting | What is probed or reused | Typical choice |
|---|---|---|
| Choosing the vision tower of a VLM | Global and patch probes, then a small VLM training run per candidate | SigLIP or SigLIP 2 as default; DINOv2 + SigLIP fusion for spatial tasks (Prismatic, Cambrian-1 style studies) |
| Cheap image classifiers in production | A linear head on cached frozen features | DINOv2 with labels; CLIP or SigLIP zero-shot as a fallback when labels are missing |
| Video action recognition | A linear probe on frame features averaged over time | DINOv2’s paper probes 8 frames per clip on UCF-101, Kinetics-400 and Something-Something v2 [Certain] |
| Video retrieval and Video-RAG | Text-aligned frame embeddings in a vector index | CLIP or SigLIP frame embeddings; a probe or fine-tune on top for domain labels |
| Key-moment detection and shot classification | A per-frame linear probe, smoothed over time | Whichever encoder wins the probe on your frames; test it like this lab |
| Near-duplicate detection and video fingerprinting | Global features compared by cosine similarity | DINOv2 global features (strong instance-level retrieval in its paper) |
| Segmentation and object tracking in video | Patch tokens with a linear or small decoder | DINOv2 or DINOv3 patch features |
Common mistakes
- Choosing \(\lambda\) (or the feature, or the layer) on the test set. With a few configurations you overfit the test set by a point or more. Use a validation split, then test once.
- Skipping standardization. A few feature dimensions with huge values dominate, L-BFGS converges slowly, and one \(\lambda\) grid no longer fits all encoders.
- Using the wrong normalization constants. Each encoder has its own mean and std; a mismatch silently costs accuracy.
- Center-cropping for dense tasks. A crop moves the content against your labels and cuts off image borders. Resize instead, or crop the labels too.
- Probing the projected CLIP embedding when you want visual quality. The 512-number projection was trained for text matching; the 768-number CLS before it scores higher.
- Comparing with unequal labels or images. All encoders must see the same training images and the same test images, in the same order, so you can use paired tests.
- Reading small gaps as wins. With 2,000 test images, a gap under about 1 point between two strong encoders is often noise; check the paired interval (171.7).
- Judging an encoder only by its global probe. A VLM reads the patch tokens; an encoder can be close on classification and far behind on dense tasks, as in Step 6.
Interview questions
What is a linear probe, and why is it used to compare encoders?basic
Why do we standardize features before the probe, and where do the statistics come from?basic
How should you choose the regularization strength of a probe?basic
Which output of CLIP, DINOv2 and SigLIP should you probe, and why?core
DINOv2 beats SigLIP 2 by 2 points on a linear probe. Should a VLM use DINOv2?core
Two probes reach 96.5% and 95.75% on 2,000 test images. Is the first better?core
Why are CLIP’s patch tokens weaker for localization than DINOv2’s?core
Why does DINOv2 cost about a third more compute than CLIP B/16 at 224 px, with the same model size?core
When does a zero-shot classifier beat a linear probe?advanced
A linear probe gives low accuracy on some features. Can you conclude the information is not there?advanced
How would you run this comparison for a video VLM?advanced
Exercises
- How many trainable parameters does the probe have for DINOv2 CLS + mean features (1,536 numbers) and 10 classes?
- Using formula (171.1), how many patch tokens do DINOv2 B/14 and SigLIP 2 B/16 make at 384 × 384? Which input sizes give an equal number?
- Compute the 95% half-width of the accuracy of a probe with 92% accuracy on 500 test images (formula 171.6). Would a gap of 2 points between two such probes be convincing without a paired test?
- In Figure 2, choose “Overlapping” and 2 labels per class. Find the \(\lambda\) with the best test accuracy. Then set 60 labels per class. How much does the best \(\lambda\) matter now?
- In the mosaic task, a model that always predicts the class of the whole image (if each mosaic had one dominant class) would be wrong on three quadrants. What patch accuracy would a probe reach that always predicts the class of the top-left quadrant, everywhere?
- Change the code to probe the mean of DINOv2’s patch tokens alone. Predict first: higher or lower than the CLS? (We measured 97.85% in an earlier run with the same splits.)
Answerscheck
Key takeaways
- A linear probe (171.4) on frozen, cached, standardized (171.3) features is the cheap, fair and reproducible way to compare encoders; choose \(\lambda\) on validation (171.5) and test once.
- Probe the vector each model was trained to output: DINOv2’s CLS, CLIP’s CLS before the projection, SigLIP’s attention-pooled output (171.2).
- On CIFAR-10 with 500 labels per class we measured DINOv2 98.5%, SigLIP 2 96.5%, CLIP 95.75%. Only gaps whose paired interval (171.7) excludes zero are real; SigLIP 2 against CLIP was not.
- With 1 label per class SigLIP 2 led; zero-shot CLIP and SigLIP 2 equal probes with about 4 to 16 labels per class.
- On patch-level localization DINOv2 led by 11 to 14 points, the biggest gap in the lab; patch size (14 against 16 px) also costs it about 1.32 times the compute (171.8).
- For a VLM: a text-aligned encoder such as SigLIP 2 by default, fused with DINOv2 when spatial detail matters.
Key formulas
References for this session
- Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Tschannen et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Tschannen et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Karamcheti et al. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
- Tong et al. (2024). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
- Tong et al. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- Krizhevsky (2009). Learning Multiple Layers of Features from Tiny Images
- Wolf et al. (2020). Transformers: State-of-the-Art Natural Language Processing