Self-supervised vision: MAE, DINO, DINOv2, DINOv3
Labels are expensive, but images are free. This session shows how a vision transformer learns from images alone: MAE hides most patches and rebuilds the pixels, DINO teaches a student to match a slowly moving teacher on different crops, and DINOv2 and DINOv3 scale this up to give clean features for every patch. You will also learn when to use one global vector and when to use the patch tokens, for VLMs and for video.
What you will learn
- How MAE masks 75% of patches, why the encoder skips mask tokens, and why this makes training about 3× cheaper
- Why MAE features are excellent after fine-tuning but weaker when frozen
- How DINO trains a student against an EMA teacher with multi-crop, and how centering and sharpening prevent collapse
- What DINOv2 adds: the iBOT patch loss, KoLeo, Sinkhorn-Knopp centering, curated data, distillation, and registers
- Why dense features degrade in long training, and how DINOv3’s Gram anchoring fixes it
- When to use global features (retrieval, near-duplicates) and when to use dense patch tokens (segmentation, tracking, VLM input)
Why this matters
Labels are expensive. ImageNet has about 1.3 million labeled training images, but the web has billions of images with no labels at all. Self-supervised learning (SSL) trains a vision encoder on images alone: the image itself gives the training signal. The result is a backbone whose features work for classification, retrieval, segmentation, depth and tracking, often with the backbone frozen.
In session 164 you saw how a ViT turns an image into patch tokens. A ViT trained only with class labels learns what it needs to tell 1,000 classes apart, and little more. This session covers the two main ways to train a ViT without labels, and the four models that every vision engineer is expected to know:
- MAE (masked autoencoder, 2021): hide most of the patches and rebuild their pixels. Like BERT for images.
- DINO (2021): a student network learns to give the same output as a teacher network on different views of the same image. The teacher is an average of past students.
- DINOv2 (2023) and DINOv3 (2025): DINO plus a patch-level loss, curated data at scale, and (in v3) a new loss called Gram anchoring that keeps the per-patch features clean. These are the strongest general-purpose frozen image features today for dense tasks. [Likely]
Why should an LLM or VLM engineer care? Because the vision encoder decides what the language model can see. CLIP-style encoders (next session, session 166) are trained to match captions, and they miss details that captions do not mention. Several VLM studies add DINOv2 features next to CLIP or SigLIP features to improve grounding and spatial questions [Likely]. In video, DINO patch features can track an object across frames with no training at all.
Learning from images alone
A self-supervised method invents a task whose answer is already in the data. This task is called a pretext task. We do not care about the task itself; we care about the features the network must learn to solve it. Two families matter today:
- Reconstruct what is missing (masked image modeling). Remove part of the input and predict it. MAE predicts pixels. BEiT and iBOT predict tokens or features instead.
- Agree across views (joint embedding). Make two random crops of the same image, with different colors and blur, and ask the network for the same output on both. The network must learn what stays the same: the object, not the crop position or the color. DINO belongs here, as do SimCLR, MoCo and BYOL.
The second family has a famous danger: collapse. If the network outputs the same constant vector for every image, all views agree perfectly and the loss is minimal, but the features are useless. Contrastive methods (SimCLR, MoCo, CLIP) avoid this with negatives: they also push apart views of different images. DINO avoids it without negatives, with two simple tricks we will study in detail.
MAE: hide most patches, rebuild the pixels
The masked autoencoder (He et al., 2021) is BERT’s masked-word idea applied to image patches, with two important changes. First, it hides far more: 75% of the patches by default. Second, the big encoder never sees the hidden patches at all.
The steps, for a 224×224 image and 16×16 patches (so \(14 \times 14 = 196\) patches):
- Patchify the image into 196 patches, each with \(16 \cdot 16 \cdot 3 = 768\) pixel values.
- Mask: choose 75% of the patches uniformly at random and remove them. 49 visible patches remain.
- Encode: a large ViT runs on the 49 visible patches only (each with its position embedding). No mask tokens enter the encoder.
- Decode: put the 49 encoded tokens back in their places, fill the 147 empty places with a shared, learned mask token, add position embeddings to all 196, and run a small transformer decoder.
- Predict pixels: a linear layer maps each decoder output to 768 numbers, the pixels of that patch. The loss is the mean squared error, computed on the masked patches only.
The loss, for one image with the set \(\mathcal{M}\) of masked patch indices, is:
The paper also tests a normalized pixel target: each patch is normalized by its own mean and standard deviation before it is used as the target. This makes the model care about local contrast and edges, not the overall brightness of the patch, and it improves the features. [Certain]
- Read the symbols
- \(\mathbf{x}_i\)
- the true pixels of patch \(i\), flattened to a vector of \(D_p = 16 \cdot 16 \cdot 3 = 768\) numbers
- \(\tilde{\mathbf{x}}_i\)
- the target: the same pixels, normalized inside the patch (or the raw pixels, in the simpler version)
- \(\hat{\mathbf{x}}_i\)
- the decoder’s prediction for patch \(i\)
- \(\mu_i, \sigma_i^2\)
- mean and variance of the 768 pixel values of patch \(i\)
- \(\mathcal{M}\)
- the masked (hidden) patches; visible patches give no loss
- \(\epsilon\)
- a small number (for example \(10^{-6}\)) that avoids division by zero on flat patches
Why 75%, when BERT masks only 15%?
Words carry a lot of meaning each; a sentence with 15% missing words is already a hard puzzle. Images are very redundant: a missing patch of sky or grass can be guessed from its neighbours by simple interpolation, with no understanding of the scene. With 75% removed at random, most missing patches have no visible neighbour nearby, and the model must use knowledge of objects and scenes. The MAE paper found that 75% works well for both fine-tuning and linear probing. [Certain] Try it in Figure 2.
Figure 2 runs one MAE training step on a toy image, one line of the algorithm at a time. There is no trained decoder in your browser, so a stand-in plays its part: it fills each hidden patch with the average color of the nearest visible patches. That is the best a model with no understanding of the scene can do, so its loss tells you how hard the task is.
- patchify: \(\mathbf{x}_1, \dots, \mathbf{x}_N\), each with \(D_p\) numbers
- \(\pi \leftarrow\) random shuffle of the indices; \(\mathcal{M} \leftarrow\) first \(M = rN\) of \(\pi\)
- visible \(\leftarrow\) the other \(V = N - M\) patches; drop \(\mathcal{M}\)
- \(\mathbf{z} \leftarrow E(\text{visible})\) // cost by (165.3)
- put a mask token at every \(i \in \mathcal{M}\); \(\hat{\mathbf{x}} \leftarrow D(\text{all } N)\)
- for \(i \in \mathcal{M}\): \(\ell_i \leftarrow \frac{1}{D_p}\lVert \hat{\mathbf{x}}_i - \mathbf{x}_i \rVert^2\); \(S \leftarrow S + \ell_i\)
- \(\mathcal{L}_{\text{MAE}} \leftarrow S / |\mathcal{M}|\) // (165.1)
How to read Figure 2
Press Play or Step. Step 2 shuffles the patch indices and marks the first \(M\) in red: at 75% with the first mask, \(\pi\) starts with patches #26, #31, #137, #148 (patch #\(i\) sits in row \(\lfloor i/14 \rfloor + 1\) and column \((i \bmod 14) + 1\)). Step 3 removes them, and step 4 puts the visible count into the encoder cost (165.3): 49 tokens cost 24.4% of a full pass. Steps 6 to 19 score one row each: row 1 has 11 hidden patches with \(S = 0.2337\), and the worst patch of the image is in row 5, column 12, at the lower edge of the sun, which the stand-in fills with pure sun yellow (\(\ell = 0.1694\)). The last step gives \(\mathcal{L} = 3.7188 / 147 = 0.0253\), which is 0.432 of the image variance.
Now move the slider. At 25% masking the stand-in error is only 0.197 of the variance: the task is too easy and teaches little. At 90% it is 0.654: now you need to know that a roof sits on a house and grass sits below the sky. Switch to Grid at 75%: the error drops to 0.257, because every hidden patch has a visible neighbour. Block masking (large square holes) goes the other way, to 0.854, because whole objects disappear. The MAE paper saw the same thing: grid masking is an easier task and gave worse features than random masking (in its ablation, linear probing 66.0% vs 73.5%). [Certain] Changing a control keeps the current step, so you can stop on a row and compare.
Why the encoder skips the mask tokens
BERT feeds a [MASK] token into the encoder. MAE does not, for two reasons. First, at test time there are no mask tokens, so an encoder trained on 75% mask tokens would see very different inputs in training and at test time. Second, compute: a transformer block costs about \(24 n d^2\) FLOPs for its linear layers and \(4 n^2 d\) for attention, for \(n\) tokens of width \(d\). With a fraction \(r\) of the \(N\) patches removed:
The linear part shrinks by \(1 - r\) and the attention part by \((1-r)^2\). The decoder is small: 8 blocks of width 512, which is about 9% of the per-token FLOPs of a ViT-L encoder (24 blocks of width 1024). [Certain]
\(N = 196\), \(r = 0.75\), so the encoder sees \(n = 49\) tokens. For one ViT-L block (\(d = 1024\)): linear layers \(24 \cdot 49 \cdot 1024^2 \approx 1.23 \times 10^9\) FLOPs instead of \(4.93 \times 10^9\); attention \(4 \cdot 49^2 \cdot 1024 \approx 9.8 \times 10^6\) instead of \(1.57 \times 10^8\). The encoder does 24.4% of the work of a full pass.
Decoder per token vs encoder per token, linear parts only: \(\frac{8 \cdot 512^2}{24 \cdot 1024^2} = \frac{1}{12} \approx 8.3\%\), close to the paper’s 9%.
Total pretraining FLOPs (encoder + decoder) compared with an encoder that also processes 147 mask tokens: \(\frac{24\,C_{\text{blk}}(196, 1024) + 8\,C_{\text{blk}}(196, 512)}{24\,C_{\text{blk}}(49, 1024) + 8\,C_{\text{blk}}(196, 512)} \approx 3.29\). The paper reports 3.3× fewer FLOPs and a 2.8× wall-clock speedup for this setting. [Certain]
What MAE features are good at, and what not
MAE features shine after fine-tuning. With only ImageNet-1K images, a ViT-H MAE fine-tuned at 448×448 reached 87.8% top-1 on ImageNet. [Certain] But as frozen features they are weaker: a ViT-L MAE gets 75.8% with a linear probe (a linear classifier trained on frozen features), and in the DINOv2 paper a ViT-H/14 MAE gets only 49.4% with a k-NN classifier, against 83.5% for DINOv2 ViT-g/14. [Certain] The reason: predicting pixels forces the network to keep low-level detail, and the classes are not linearly separated in its last layer. The MAE paper shows that fine-tuning only the last block already lifts accuracy from 73.5% to 81.0%. [Certain]
DINO: a student matches a teacher
DINO (Caron et al., 2021) stands for self-distillation with no labels. In ordinary knowledge distillation (session 59), a big trained teacher gives soft targets to a small student. In DINO there is no trained teacher. Both networks have the same architecture (a ViT backbone plus a small MLP “projection head”), and the teacher is built from the student itself.
One training step:
- From one image, make several random crops with color jitter, blur and flips: 2 global crops (resized to 224×224, each covering a large part of the image) and several local crops (96×96, covering small parts). This is called multi-crop.
- The teacher sees only the 2 global crops. The student sees all crops.
- Each network outputs \(K\) numbers (the paper uses \(K = 65{,}536\)), turned into a probability distribution with a softmax. Think of the \(K\) outputs as soft “pseudo-classes” or prototypes that the model invents.
- The loss asks the student’s distribution on every crop to match the teacher’s distribution on a different global crop. A local crop of a dog’s ear must produce the same pseudo-class as the whole dog: “local-to-global” correspondence.
- Only the student gets gradients. The teacher’s weights are then updated as an exponential moving average (EMA) of the student’s weights.
The math
Let \(g_{\theta_s}\) be the student and \(g_{\theta_t}\) the teacher. For an input crop \(x\), the student’s probability for prototype \(k\) is a softmax with temperature \(\tau_s\):
The teacher’s output first has a center vector \(\mathbf{c}\) subtracted, then goes through a softmax with a much lower temperature \(\tau_t\):
With the set \(V\) of all crops and the two global crops \(x^g_1, x^g_2\), the loss is the cross-entropy \(H(a, b) = -\sum_k a^{(k)} \log b^{(k)}\), summed over teacher–student pairs of different crops:
- Read the symbols
- \(K\)
- number of output dimensions (prototypes); 65,536 in DINO
- \(\tau_s, \tau_t\)
- student and teacher temperatures; DINO uses \(\tau_s = 0.1\) and warms \(\tau_t\) up from 0.04 to 0.07 over the first 30 epochs [Certain]. A lower temperature gives a sharper (more peaked) distribution
- \(\mathbf{c}\)
- the center: a running mean of the teacher outputs over batches (formula 165.8)
- \(V\)
- all crops of one image: 2 global and several local (for example 6 or 10)
- \(H(P_t, P_s)\)
- cross-entropy; the teacher side is treated as a constant (stop-gradient)
The teacher is an exponential moving average of the student. After every optimizer step on the student:
DINO increases \(\lambda\) from 0.996 to 1 with a cosine schedule. [Certain] With \(\lambda = 0.996\), the teacher averages roughly the last \(1/(1-\lambda) = 250\) student versions. The paper notes that this averaging acts like an ensemble of past models, and the teacher is better than the student during training, so it gives good targets. [Certain]
Avoiding collapse: centering and sharpening
There are two ways the system can cheat and output useless features:
- Collapse to one dimension: the teacher puts all mass on the same prototype for every image. Student and teacher agree perfectly, and the features carry no information.
- Collapse to uniform: the teacher outputs a flat distribution for every image. Again agreement is easy and useless.
DINO uses two opposite forces. Centering subtracts the running mean \(\mathbf{c}\), so no prototype can win for every image; it prevents one-dimension collapse but pushes toward uniform. Sharpening (a low \(\tau_t\)) makes each teacher output peaked; it prevents uniform collapse but pushes toward one dimension. Together, with the EMA teacher, they balance. [Certain] The center is updated with its own EMA over the batch of \(B\) teacher outputs:
Here \(m\) is a rate close to 1 (0.9 in the official code [Likely]). The paper explains collapse with a split of the cross-entropy:
where \(h(P_t)\) is the entropy of the teacher output. A KL that drops to 0 means student and teacher output a constant: collapse. Without centering, \(h(P_t)\) goes to 0 (one dimension); without sharpening, it goes to \(\log K\) (uniform). [Certain]
How to read Figure 4
Start at the default (\(\tau_t = 0.04\), centering on): each image is confident (mean entropy 0.097 nats) and the six images use six different prototypes, so the batch-average entropy is 1.834, close to \(\ln 6 = 1.792\) plus a little spread. Turn centering off: the shared bias wins everywhere, all six images pick the gray prototype: one-dimension collapse. Turn centering back on and raise \(\tau_t\) to 1.0: each distribution becomes almost flat (entropy 2.074, the maximum is \(\ln 8 = 2.079\)): uniform collapse. Only both tricks together keep the outputs informative.
Teacher logits for a global crop: \(g_t = (0.6, 0.5, -0.2)\). The center (running mean over images) is \(\mathbf{c} = (0.3, 0.0, -0.1)\): prototype 1 is high for every image. Centered: \((0.3, 0.5, -0.1)\). Divide by \(\tau_t = 0.04\): \((7.5, 12.5, -2.5)\). Softmax: \(P_t = (0.0067, 0.9933, 0.0000)\). Without centering, the target would be \((0.9241, 0.0759, 0.0000)\): prototype 1 again, as for every other image.
Student logits for a local crop: \(g_s = (0.2, 0.6, 0.0)\). Divide by \(\tau_s = 0.1\) and softmax: \(P_s = (0.0179, 0.9796, 0.0024)\). Cross-entropy: \(H = -(0.0067 \ln 0.0179 + 0.9933 \ln 0.9796) = 0.0474\). The teacher entropy is \(h(P_t) = 0.0402\), so \(\KL(P_t \Vert P_s) = 0.0072\): the student agrees, but not perfectly, and the gradient pushes its second logit up.
The whole loop, running
Figure 4 froze one moment. Figure 5 runs the full DINO loop of Algorithm 2 on a toy: 6 images, \(K = 6\) prototypes, and a student that is a single linear layer \(g_{\theta}(\mathbf{x}) = \mathbf{W}\mathbf{x} + \mathbf{b}\) on 6-dimensional inputs. Each image \(b\) is a fixed unit vector \(\mathbf{e}_b\); a “view” is \(\mathbf{e}_b\) plus Gaussian noise of size \(\sigma = 0.2\), which plays the role of a random crop. The student starts with a shared bias \(b^{(1)} = 1\) toward prototype 1, the same trap as in Figure 4. The teacher starts as a copy of the student, and the center starts at \(\mathbf{c} = \mathbf{0}\).
Only one more formula is needed. For a softmax with temperature \(\tau_s\), the gradient of the cross-entropy with respect to the student logits is \(\partial H / \partial \mathbf{z}_s = (P_s - P_t)/\tau_s\): the student is pushed up where the teacher puts more mass than the student does. The teacher gets no gradient; it only moves through the EMA.
- for each image: views \(\mathbf{x}^1, \mathbf{x}^2 \leftarrow \mathbf{e}_b + \sigma\varepsilon\)
- teacher logits \(\mathbf{z}_t \leftarrow \mathbf{W}_t\mathbf{x} + \mathbf{b}_t\) // no gradient
- \(P_t \leftarrow \softmax((\mathbf{z}_t - \mathbf{c})/\tau_t)\) // (165.5)
- \(\mathbf{z}_s \leftarrow \mathbf{W}_s\mathbf{x} + \mathbf{b}_s\); \(P_s \leftarrow \softmax(\mathbf{z}_s/\tau_s)\) // (165.4)
- \(\mathcal{L} \leftarrow\) mean of \(H(P_t(\mathbf{x}^1), P_s(\mathbf{x}^2))\) and the swap // (165.6)
- \(\theta_s \leftarrow \theta_s - \eta\nabla\mathcal{L}\), with \(\partial H/\partial\mathbf{z}_s = (P_s - P_t)/\tau_s\)
- \(\theta_t \leftarrow \lambda\theta_t + (1-\lambda)\theta_s\) // (165.7)
- \(\mathbf{c} \leftarrow m\mathbf{c} + (1-m)\,\text{mean}(\mathbf{z}_t)\) // (165.8)
How to read Figure 5
Press Step to walk through iteration 1, one line of the algorithm at a time; the live panel puts the numbers of the tracked image into each formula. In iteration 1 the center is still zero, so the shared bias wins: all six teacher outputs put almost all mass on prototype 1, and the batch loss is 0.4695. Line 7 moves the teacher bias only a little: \(b_t^{(1)} = 0.95 \cdot 1 + 0.05 \cdot 1.0147 = 1.0007\). Line 8 starts to learn the bias into the center: \(c^{(1)} = 0.091\).
Now press “Train +50 iterations” several times and watch the chart. With the defaults the center soaks up the shared bias, the images spread over the prototypes, and by iteration 300 all six images pick six different prototypes: mean teacher entropy 0.033 nats, entropy of the batch average 1.791 (close to the maximum \(\ln 6 = 1.792\)), loss 0.057. Press Reset and try the failures. Centering off: every image stays on prototype 1 and the loss falls to 0.001 by iteration 100 and to 0 after that: one-dimension collapse, the perfect but useless agreement of (165.9). \(\tau_t = 1\): the teacher output is flat for every image and the loss sits at \(\ln 6 = 1.792\): uniform collapse. \(\lambda = 0\) (the teacher is a copy of the student): the targets move as fast as the student and after 300 iterations the run is back on one prototype. \(\lambda = 0.99\): the teacher is very slow, and after 300 iterations only 4 prototypes are in use. This is a toy, not the real model, but the forces are the same.
What DINO learned without labels
A ViT-B/8 trained with DINO reached 80.1% ImageNet top-1 with a linear probe and 77.4% with a simple k-NN classifier on frozen features. [Certain] More surprising: the self-attention of the CLS token in the last layer forms clean object masks, though the model never saw a segmentation label. The paper calls this an “emerging property”. [Certain] The same features also work well for image retrieval and copy detection, and DINO features were already used to propagate segmentation masks through DAVIS videos. [Certain]
DINOv2: DINO plus patch-level learning, at scale
DINO features are good for a whole image. Detection, segmentation, depth and tracking need good features for every patch. DINOv2 (Oquab et al., 2023) combines several known ideas and scales them up. [Certain]
- Image-level loss: the DINO loss on the CLS token, as above.
- Patch-level loss (iBOT): mask some patches of the student’s input (not the teacher’s). For each masked patch, the student must predict the teacher’s prototype distribution for that same patch, which the teacher computes from the visible image. This is masked image modeling, but in feature space, not pixel space.
- Separate heads for the DINO and iBOT losses (untied weights), which worked better at scale.
- Sinkhorn-Knopp centering (from SwAV) instead of the running-mean center: 3 iterations that make the teacher’s outputs over the batch use all prototypes about equally.
- KoLeo regularizer: spreads the features of a batch evenly (formula 165.11).
- Short high-resolution phase at 518×518 at the end of training, because small objects vanish at low resolution.
- Curated data: LVD-142M, 142 million images selected automatically from a pool of about 1.2 billion web images, by retrieving images that are close to images in curated datasets, after deduplication.
- Big model, then distillation: a ViT-g/14 with 1.1B parameters (embedding size 1536, 24 heads) is trained first; ViT-S, B and L are distilled from it.
The iBOT loss, for the set \(\mathcal{M}\) of patches masked in the student’s input:
- Read the symbols
- \(p_{t,i}\)
- teacher’s centered, sharpened distribution for patch \(i\), computed from the unmasked image
- \(p_{s,i}\)
- student’s distribution for patch \(i\), computed from the input where patch \(i\) was replaced by a mask token
The KoLeo regularizer looks at a batch of \(n\) L2-normalized features \(\mathbf{x}_1, \dots, \mathbf{x}_n\) and rewards each point for being far from its nearest neighbour:
The full DINOv2 objective is a weighted sum; the KoLeo weight is 0.1 [Certain]:
Four unit vectors in 2-D: \((1, 0)\), \((0.8, 0.6)\), \((0, 1)\), \((-1, 0)\). Nearest-neighbour distances: the first two are 0.6325 apart; the third is 0.8944 from the second; the fourth is \(\sqrt{2} = 1.4142\) from the third. \(\mathcal{L}_{\text{KoLeo}} = -\tfrac14(\ln 0.6325 + \ln 0.6325 + \ln 0.8944 + \ln 1.4142) = 0.1703\). Moving the second point away from the first lowers the loss. The DINOv2 ablation shows KoLeo improves retrieval (Oxford-M) from 55.6 to 63.9 mAP while classification stays about the same. [Certain]
The ablations show which part does what: removing the iBOT masked-modeling term costs almost 3 mIoU on ADE20k segmentation (47.1 vs 44.2), so the patch-level loss is what makes the dense features good. [Certain] With a frozen backbone and a linear probe, DINOv2 ViT-g/14 reaches 86.5% on ImageNet, close to OpenCLIP ViT-G/14 (86.2%), which was trained with text. [Certain]
Registers: a side story about artifacts
Large DINOv2 models show a few high-norm tokens in their patch feature maps, mostly in boring background areas. Darcet et al. (2023) found that the model re-uses these low-information patches as scratch space for global computation; they are about 2% of the tokens and their norm is about 10 times higher than normal. The fix is simple: add a few extra learnable tokens, called registers, that are not image patches. The model then uses the registers as scratch space, and the patch maps become clean. With 4 registers the extra FLOPs are below 2%. [Certain] DINOv3 uses 4 registers by default. [Certain]
DINOv3: 7B parameters and Gram anchoring
DINOv3 (Siméoni et al., 2025) scales the DINOv2 recipe and fixes a problem that appears at scale. [Certain]
| Teacher model | DINOv2 | DINOv3 |
|---|---|---|
| Backbone, parameters | ViT-g, 1.1B | ViT-7B, 6.7B |
| Blocks, width, heads | 40, 1536, 24 | 40, 4096, 32 |
| Patch size | 14 | 16 |
| Position embeddings | learned | RoPE (axial, with box jittering) |
| Registers | 4 | 4 |
| Training images | LVD-142M | LVD-1689M (1,689 million curated images) |
| Schedules | cosine | constant learning rate, weight decay and EMA (0.999), after warmup; 1M iterations |
Batch: 4,096 images, each with 2 global crops of 256×256 and 8 local crops of 112×112. With 16-pixel patches, that is \(2 \cdot 16^2 + 8 \cdot 7^2 = 904\) patch tokens per image, the same as DINOv2 with 224/98 crops and 14-pixel patches, and about 3.7 million patch tokens per batch. [Certain]
The problem: dense features decay during long training
Training DINOv2-style models longer keeps improving ImageNet classification, but after about 200k iterations the segmentation quality of the patch features goes down, and for the 7B model it falls below its early level. [Certain] The cosine similarity maps between one patch and all others, clean at 200k iterations, become noisy by 600k: unrelated patches look similar. The paper also sees the cosine similarity between the CLS token and the patch tokens slowly grow: every patch starts to look like the global image summary, and locality is lost. [Certain]
The fix: Gram anchoring
The Gram matrix of an image is the matrix of all pairwise dot products of its \(P\) patch features. If the features are L2-normalized, entry \((i, j)\) is the cosine similarity of patch \(i\) and patch \(j\). Gram anchoring pulls the student’s Gram matrix toward that of an earlier checkpoint (the Gram teacher), whose patch features were still clean:
- Read the symbols
- \(\mathbf{X}_S \in \R^{P \times d}\)
- the student’s L2-normalized patch features for one global crop: \(P\) patches, width \(d\)
- \(\mathbf{X}_G \in \R^{P \times d}\)
- the Gram teacher’s patch features for the same crop (an early EMA teacher)
- \(\lVert \cdot \rVert_F\)
- the Frobenius norm: the square root of the sum of all squared entries
Why the Gram matrix and not the features themselves? Because it only fixes how patches relate to each other. The features may move freely, for example rotate, as the global loss keeps improving, as long as the pattern of similarities stays the same. Any rotation \(\mathbf{Q}\) (\(\mathbf{Q}\mathbf{Q}^\top = \mathbf{I}\)) leaves the loss unchanged: \((\mathbf{X}\mathbf{Q})(\mathbf{X}\mathbf{Q})^\top = \mathbf{X}\mathbf{X}^\top\).
In DINOv3, Gram anchoring starts after 1M iterations, as a refinement step; the Gram teacher is refreshed to the current EMA teacher every 10k iterations, at most three times, and the loss weight is \(w_{\text{Gram}} = 2\). [Certain] The refinement objective is:
(\(\mathcal{L}_{\text{DKoleo}}\) is a distributed version of KoLeo.) Even applied late, the Gram loss “repairs” badly degraded patch features. A further improvement feeds images at twice the resolution to the Gram teacher and down-samples its feature map by 2, which gives smoother targets; this adds about 2 mIoU on ADE20k. [Certain] Explore the idea in Figure 6.
How to read Figure 6
Click a patch on the left. At drift 0 the dog patch is similar to the other dog patches (mean cosine 0.837) and not to other regions (−0.022), and every patch’s nearest neighbour is in its own region (100%). Raise the drift to 0.5: other regions climb to 0.659 and the patches move toward the global direction (mean cosine 0.847). At 0.8 the gap is almost gone (0.871 vs 0.857) and the nearest-neighbour score falls to 45.8%: a linear segmentation head would struggle, but a CLS-based classifier might not notice. Now tick Gram anchoring: the features are far from the Gram teacher (mean distance 1.448), yet the Gram loss is exactly 0 and the similarity map is the clean one again. This is a toy, not the real model.
After training: high resolution, a model family, text
DINOv3 adds a short high-resolution adaptation phase (global crops up to 768, still with Gram anchoring), then distills the 7B teacher into a family: ViT-S (21M), ViT-S+ (29M), ViT-B (86M), ViT-L (300M), ViT-H+ (840M), and ConvNeXt models from 29M to 198M parameters. [Certain] With a frozen backbone, the 7B model reaches 88.4% ImageNet linear-probe accuracy and 83.3 J&F on DAVIS video segmentation tracking at high resolution, 6.7 points above DINOv2. [Certain] A text encoder is also trained on top of a frozen DINOv3 ViT-L (the dino.txt recipe) for zero-shot use, and a separate 7B model is trained on 493 million satellite images. [Certain]
Global vs dense features: which to use for VLMs and video
A ViT encoder gives two kinds of output. A global feature is one vector for the whole image: the CLS token, or the mean of the patch tokens. Dense features are the patch tokens themselves: one vector per patch, \(14 \times 14 = 196\) of them for 224×224 with 16-pixel patches, 1,024 at 512×512.
| Need | Use | Why |
|---|---|---|
| Image or video retrieval, near-duplicate detection, clustering, k-NN classification | global (CLS or mean-pooled), L2-normalized | One vector per image fits a vector index; DINO global features are strong for copy detection [Certain] |
| Semantic segmentation, depth, keypoint matching | dense patch tokens, often from DINOv2 or DINOv3, plus a light head | The answer differs per pixel; the patch-level losses (iBOT, Gram) make these maps good |
| Video object tracking (mask propagation) | dense tokens of each frame, compared across frames | A patch in frame \(t+1\) takes the label of its most similar patches in frame \(t\); no training needed |
| VLM vision input | dense tokens (usually all patches, after a projector) | The LLM must read details and positions; one global vector would throw them away (session 177) |
| Choosing the encoder for a VLM | CLIP or SigLIP for text-aligned semantics; add DINOv2 for spatial detail | The “Eyes Wide Shut” study found image pairs that CLIP sees as similar but DINOv2 does not, and mixing in DINOv2 features improved visual grounding [Certain] |
A useful way to remember it: CLIP-style training asks “does this image match this caption?”, so it learns what captions talk about. DINO-style training asks “is this the same image, seen from another crop?”, so it learns shape, parts, position and texture, including things no caption mentions. VLM papers such as Prismatic VLMs and Cambrian-1 compared many encoders and found that combining a language-supervised encoder with DINOv2 helps on spatial and localization tasks. [Likely]
Putting it together: pseudo-code and PyTorch
- input: batch of images, encoder \(E\) (large ViT), decoder \(D\) (small), mask ratio \(r = 0.75\)
- patchify each image into \(N\) patches; embed each patch and add its position embedding
- draw a random number per patch; sort; keep the first \((1-r)N\) patches // uniform random masking
- \(\mathbf{z} \leftarrow E(\text{visible tokens})\) // no mask tokens in the encoder
- insert a shared mask token in every removed place; undo the shuffle; add position embeddings
- \(\hat{\mathbf{x}} \leftarrow D(\text{all } N \text{ tokens})\), then a linear layer to \(p^2 \cdot 3\) pixels per patch
- target \(\leftarrow\) per-patch normalized pixels // formula (165.2)
- loss \(\leftarrow\) MSE on masked patches only // formula (165.1); backward; AdamW step
- after pretraining: discard \(D\); fine-tune or probe \(E\)
- input: student \(\theta_s\), teacher \(\theta_t\), center \(\mathbf{c}\), temperatures \(\tau_s, \tau_t\), EMA rates \(\lambda, m\)
- for each image: 2 global crops and several local crops, with color jitter, blur, flips
- teacher on global crops: \(P_t = \softmax((g_{\theta_t} - \mathbf{c})/\tau_t)\), no gradient // (165.5)
- student on all crops: \(P_s = \softmax(g_{\theta_s}/\tau_s)\) // (165.4)
- \(\mathcal{L} \leftarrow\) cross-entropy over all pairs of different crops // (165.6)
- DINOv2: mask some student patches of the global crops; add the iBOT loss on them // (165.10)
- DINOv2: add \(0.1 \times\) KoLeo on the student CLS features // (165.11)
- backward; optimizer step on \(\theta_s\) only
- \(\theta_t \leftarrow \lambda\theta_t + (1-\lambda)\theta_s\) // (165.7); \(\mathbf{c} \leftarrow m\mathbf{c} + (1-m)\,\text{mean}(g_{\theta_t})\) // (165.8)
- DINOv3 refinement: also add \(w_{\text{Gram}}\mathcal{L}_{\text{Gram}}\) against an early teacher // (165.13)
The core pieces in PyTorch. It runs on a CPU in a few seconds, and prints the same numbers every time:
import copy, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # ---- 1) MAE: patchify, random masking, loss on masked patches only ---- def patchify(img, p=16): # (B,3,H,W) -> (B,N,p*p*3) B, C, H, W = img.shape x = img.reshape(B, C, H // p, p, W // p, p) return x.permute(0, 2, 4, 3, 5, 1).reshape(B, (H // p) * (W // p), p * p * C) def random_masking(tokens, ratio=0.75): # the shuffle trick from the MAE paper B, N, D = tokens.shape keep = int(N * (1 - ratio)) noise = torch.rand(B, N) # one random number per patch ids_shuffle = noise.argsort(dim=1) # random order ids_restore = ids_shuffle.argsort(dim=1) # to undo the shuffle later ids_keep = ids_shuffle[:, :keep] visible = torch.gather(tokens, 1, ids_keep[..., None].expand(-1, -1, D)) mask = torch.ones(B, N); mask[:, :keep] = 0 mask = torch.gather(mask, 1, ids_restore) # 1 = masked, in the original order return visible, mask, ids_restore img = torch.randn(2, 3, 224, 224) target = patchify(img) # (2, 196, 768) mu, var = target.mean(-1, keepdim=True), target.var(-1, keepdim=True) target = (target - mu) / (var + 1e-6).sqrt() # normalized pixels, formula (165.2) emb = nn.Linear(768, 64) visible, mask, ids_restore = random_masking(emb(target)) print("visible tokens:", visible.shape[1], "masked:", int(mask[0].sum())) # 49, 147 pred = torch.randn_like(target) # stand-in for the decoder output loss = (((pred - target) ** 2).mean(-1) * mask).sum() / mask.sum() # (165.1) print("MAE loss on masked patches:", round(loss.item(), 3)) # 2.009 # ---- 2) DINO: centering + sharpening, cross-entropy, EMA teacher ---- K = 256 student = nn.Sequential(nn.Linear(32, 64), nn.GELU(), nn.Linear(64, K)) teacher = copy.deepcopy(student) for p in teacher.parameters(): p.requires_grad = False center = torch.zeros(K) def dino_loss(s_out, t_out, center, tau_s=0.1, tau_t=0.04): t = F.softmax((t_out - center) / tau_t, dim=-1).detach() # (165.5): center, sharpen, stop-grad log_s = F.log_softmax(s_out / tau_s, dim=-1) # (165.4) return -(t * log_s).sum(-1).mean() opt = torch.optim.AdamW(student.parameters(), lr=1e-3) x = torch.randn(16, 32) for step in range(3): v1 = x + 0.1 * torch.randn_like(x) # two "views" of the same batch v2 = x + 0.1 * torch.randn_like(x) s1, s2 = student(v1), student(v2) with torch.no_grad(): t1, t2 = teacher(v1), teacher(v2) loss = 0.5 * (dino_loss(s2, t1, center) + dino_loss(s1, t2, center)) # (165.6) with 2 views opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): lam = 0.996 for pt, ps in zip(teacher.parameters(), student.parameters()): pt.mul_(lam).add_((1 - lam) * ps) # (165.7) EMA teacher center = 0.9 * center + 0.1 * torch.cat([t1, t2]).mean(0) # (165.8) print(f"step {step}: DINO loss {loss.item():.3f}") # 2.230, 2.129, 2.010 # ---- 3) KoLeo (165.11) and Gram anchoring (165.13) ---- def koleo(z): z = F.normalize(z, dim=-1) d = torch.cdist(z, z); d.fill_diagonal_(float("inf")) return -torch.log(d.min(dim=1).values + 1e-8).mean() def gram_loss(xs, xg): # xs, xg: (P, d) patch features of one image xs, xg = F.normalize(xs, dim=-1), F.normalize(xg, dim=-1) return ((xs @ xs.T - xg @ xg.T) ** 2).sum() z = torch.tensor([[1., 0.], [0.8, 0.6], [0., 1.], [-1., 0.]]) print("KoLeo:", round(koleo(z).item(), 4)) # 0.1703, as in the worked example xg = torch.randn(196, 64) Q, _ = torch.linalg.qr(torch.randn(64, 64)) # a random rotation print("Gram loss after rotation:", round(gram_loss(xg @ Q, xg).item(), 6)) # 0.0 print("Gram loss vs noisy copy:", round(gram_loss(xg + torch.randn_like(xg), xg).item(), 1)) # 597.7
The printed values come from PyTorch 2.8 on a CPU; other versions may differ in the last digits. For real work, load pretrained weights instead of training: for example the DINOv2 models through torch.hub or Hugging Face, and take last_hidden_state[:, 0] as the global feature and the remaining tokens (after the registers, if any) as the dense map. [Likely]
Where this is used
| Application | Which model and which features | Note |
|---|---|---|
| Backbone pretraining for a fine-tuned detector or classifier | MAE encoder, fully fine-tuned | SAM’s image encoder is an MAE-pretrained ViT-H [Likely] |
| Vision encoder inside a VLM | DINOv2 patch tokens, often next to SigLIP tokens | Better spatial grounding than a CLIP-only encoder [Likely]; see session 178 |
| Monocular depth estimation | DINOv2 encoder with a DPT-style head | Depth Anything builds on a DINOv2 encoder [Likely] |
| Video object segmentation and tracking | DINO, DINOv2, DINOv3 dense tokens; label propagation across frames | DINOv3 7B: 83.3 J&F on DAVIS at high resolution, with no video training [Certain]; see session 204 |
| Video retrieval, near-duplicate detection, fingerprinting | DINO global features per frame, L2-normalized, pooled over time | Strong copy-detection features [Certain]; see session 209 |
| Action recognition and video classification | frozen frame features plus a small attentive probe | DINOv3 is evaluated this way on UCF101, Something-Something V2 and Kinetics-400 [Certain] |
| Key-moment and shot-change detection | cosine distance between global features of consecutive frames | Cheap and label-free; a sudden drop in similarity marks a cut |
| Domains without labels: satellite, medical, biology | DINO-style SSL trained on the domain itself | DINOv3 trains a 7B model on 493M satellite images [Certain] |
Common mistakes
- Using MAE features frozen and expecting CLIP-level linear-probe results. MAE is designed to be fine-tuned. Probe it with a few trainable blocks, or choose DINOv2 or DINOv3 for frozen use.
- Feeding mask tokens into the MAE encoder. It wastes about 3× compute and creates a train–test mismatch. Drop the masked patches; add mask tokens only in the decoder.
- Computing the MAE loss on all patches. The visible patches are trivial to copy and dilute the signal; the paper computes the loss on masked patches only.
- Letting gradients flow into the DINO teacher, or copying the student into the teacher every step. The teacher must be an EMA with no gradient; the paper reports that copying the student fails to converge. [Certain]
- Removing centering or using a high teacher temperature. You get one-dimension or uniform collapse (Figure 4). Monitor the teacher entropy and the KL of formula (165.9) during training.
- Using the CLS token for a dense task. Segmentation, depth and tracking need the patch tokens. Also skip the register tokens when you reshape patch tokens into a 2-D map.
- Wrong patch size or resolution when reshaping. DINOv2 uses 14-pixel patches, DINOv3 uses 16. A 224×224 input gives a 16×16 map in DINOv2 and a 14×14 map in DINOv3.
- Forgetting normalization in retrieval. L2-normalize global features before cosine search and k-NN, as in the DINO evaluation.
Interview questions
Why does MAE mask 75% of patches while BERT masks 15% of tokens?basic
Why does the MAE encoder not use mask tokens, and what does that buy?core
MAE fine-tunes very well but has weak linear-probe accuracy. Why?core
How does DINO avoid collapse without negative pairs?core
Why is the teacher an EMA of the student? Why not just use the student?advanced
What is multi-crop and why does it help DINO?core
What does DINOv2 add to DINO, and which part matters for segmentation?core
What is the KoLeo regularizer?advanced
What is Gram anchoring and why does it act on the Gram matrix, not on the features?advanced
You build a VLM. Do you use CLIP/SigLIP or DINOv2 as the vision encoder, and which tokens?core
How would you track an object through a video using only a frozen DINO model?core
What are registers in a ViT, and why do DINOv2 and DINOv3 need them?advanced
Exercises
- An MAE uses 448×448 images and 16×16 patches, mask ratio 75%. How many patches are there, and how many does the encoder see? By what factor does the attention part of one encoder block shrink?
- A DINO teacher has \(\lambda = 0.999\). About how many past student versions does it average? Compare with \(\lambda = 0.996\).
- In Figure 4, set \(\tau_t = 0.07\) with centering on. Is the output healthy? What is the mean per-image entropy?
- For the worked DINO example, suppose the student instead outputs \(g_s = (0.6, 0.2, 0.0)\). Without computing exactly, will the cross-entropy go up or down, and why?
- Two student patch features are \(\mathbf{x}_1 = (1, 0)\) and \(\mathbf{x}_2 = (0.6, 0.8)\); the Gram teacher has \(\mathbf{x}_1 = (1, 0)\), \(\mathbf{x}_2 = (1, 0)\), and a third patch \((0, 1)\) in both, while the student’s third patch is \((0, 1)\). Compute the Gram loss (165.13).
- DINOv3 uses 16-pixel patches. How many patch tokens does a 1024×1024 image produce, and how many entries does its Gram matrix have?
Answerscheck
Key takeaways
- MAE removes 75% of patches, runs a large encoder on the rest, and a small decoder rebuilds the pixels; the loss is on masked patches only (165.1). It is about 3× cheaper to train and excellent after fine-tuning, but weaker as frozen features.
- DINO trains a student to match an EMA teacher (165.7) across global and local crops (165.6). Centering plus sharpening (165.5, 165.8) avoid collapse without negatives.
- DINOv2 adds a patch-level iBOT loss (165.10), KoLeo (165.11), Sinkhorn-Knopp centering, curated LVD-142M data and a 1.1B model distilled into smaller ones; its frozen features are strong for both global and dense tasks.
- DINOv3 scales to 6.7B parameters and 1.7B images and adds Gram anchoring (165.13), which keeps patch similarity structure clean during long training.
- Use global features for retrieval and near-duplicates; use dense patch tokens for segmentation, depth, tracking and as VLM input. DINO features complement CLIP-style features in VLMs.
Key formulas
References for this session
- He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
- Caron et al. (2021). Emerging Properties in Self-Supervised Vision Transformers
- Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision
- Siméoni et al. (2025). DINOv3
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Zhou et al. (2021). iBOT: Image BERT Pre-Training with Online Tokenizer
- Caron et al. (2020). Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Sablayrolles et al. (2018). Spreading vectors for similarity search
- Darcet, Oquab, Mairal, Bojanowski (2023). Vision Transformers Need Registers
- Tong et al. (2024). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
- Karamcheti et al. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
- Tong et al. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- Kirillov et al. (2023). Segment Anything
- Yang et al. (2024). Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language