LLM & VLM courseBlog
169Track 7: Vision and VLMs · session 7 of 38

Vision applications: retrieval, zero-shot classification, near-duplicates

One image embedding can answer three different questions: which images look like this one (retrieval), which label fits this image (classification), and is this image an edited copy of one I already know (copy detection). The vector can be the same; the decision rule and the metric change. This session builds the index, the re-ranking step, zero-shot, few-shot and linear-probe classifiers, and the SSCD copy detector, and it computes recall@k, mAP and micro-AP by hand.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 165 (self-supervised vision), 166 (CLIP) and 151 (vector search at scale)

What you will learn

  • Three kinds of “same”: same class, same object, same source image, and which descriptor fits each
  • Image retrieval as a system: embed once, index (flat, IVF, PQ, HNSW), search, then re-rank with query expansion or a stronger model
  • Recall@k, precision@k, AP and mAP for ranking, and micro-AP for detection, all computed by hand on one example
  • Zero-shot, few-shot and linear-probe classification on frozen features, and why zero-shot CLIP beats a 1-shot linear probe
  • Mixing the text prior with a few labeled images: prompt-as-pseudo-shots and Tip-Adapter
  • Copy detection with global descriptors: GeM pooling, the SSCD recipe (InfoNCE plus KoLeo entropy) and score normalization
  • Why a single threshold for all queries is hard, and how precision collapses when the reference set grows

Why this matters

The last sessions built image encoders: ViT, DINOv2, CLIP and SigLIP. Each one turns an image into one vector. This session is about what you do with that vector. Three products cover most of it: image search (“show me photos like this one”, or “show me photos of a red bicycle”), classification without training (“which of my 300 product categories is this?”), and near-duplicate and copy detection (“is this upload an edited copy of an image we already removed?”).

All three run on the same machinery: embed once, store the vectors, compare with a dot product. But they ask different questions, so they need different decision rules and different metrics. A search engine cares about the order of results for each query. A classifier cares about the top-1 label. A copy detector must make a yes/no decision with one threshold for millions of queries, most of which have no copy at all. Mixing these up is the most common error in practice and in interviews: people evaluate a copy detector with mAP, or tune a retrieval model on classification accuracy, and are surprised in production.

Real systems built this way: DINOv2’s training set LVD-142M was made by removing near-duplicates from 1.2 billion web images with the SSCD copy detector, then retrieving images close to curated datasets with embedding search in FAISS [Certain] (session 165). Content moderation at social networks matches uploads against databases of known harmful images; the 2021 Image Similarity Challenge (ISC21) was built to benchmark exactly this [Certain]. CLIP-style zero-shot classifiers and text-to-image search are the subject of the next lab, session 170, and the same ideas extend to video in session 208 and session 209.

One embedding, three kinds of “same”

When we say two images are “similar”, we can mean three very different things. The copy detection literature puts them on a line [Certain, following Pizzi et al., 2022]:

  • Same class (semantic similarity): two different cats. This is what CLIP, SigLIP and DINOv2 embeddings capture best. Used for search by meaning and for classification.
  • Same object or place (instance matching): two photos of the same building from different viewpoints. Classical “landmark retrieval”, with benchmarks such as Oxford and Paris. Descriptors are trained so that viewpoint changes do not matter, but different buildings of the same style do.
  • Same source image (copy detection): one photo, then the same photo cropped, with text added, filtered, or screenshotted. SSCD defines it strictly: images are copies if and only if they come from the same 2-D image source [Certain]. Two different photos of the same landmark are not copies.
same class same object same source image LOL two different cats one house, two viewpoints a photo, then cropped + text + emoji descriptor: CLIP, SigLIP, DINOv2 descriptor: GeM retrieval CNN descriptor: SSCD decision: rank, or argmax over labels decision: rank per query decision: one global threshold metric: recall@k, accuracy metric: mAP metric: micro-AP looser: what is in the picturestricter: which exact pixels
Figure 1. Three kinds of “same”, from loose to strict. Each needs a descriptor that is invariant to the right things (viewpoint, edits) and sensitive to the rest, a decision rule, and a metric that matches the decision.

The key point: a descriptor is good only relative to a task. A CLIP embedding puts two different cats close together, which is right for search by meaning and wrong for copy detection: it would flag every cat photo as a copy of every other. SSCD does the opposite. Its authors write that an ideal copy descriptor maps copies together “while keeping even semantically similar (same ‘class’) images far apart”, the opposite of what classification wants [Certain]. Their entropy loss lowers ImageNet linear-probe accuracy while raising copy detection accuracy [Certain, their Figure 3].

Image retrieval: embed once, search many times

Retrieval has two phases. Offline, embed every image of the collection (the database or gallery), L2-normalize the vectors and put them in an index, a data structure built for fast nearest-neighbour search. Online, embed the query (an image, or a text with a CLIP-style text tower) and ask the index for the \(k\) database vectors with the highest score:

\[ s(\mathbf{q}, \mathbf{x}_i) = \mathbf{q}^\top \mathbf{x}_i = \cos(\mathbf{q}, \mathbf{x}_i), \qquad \mathcal{T}_k(\mathbf{q}) = \text{the } k \text{ indices } i \text{ with the largest } s(\mathbf{q}, \mathbf{x}_i) \tag{169.1} \]
Read the symbols
\(\mathbf{q}\)
the unit-length query embedding (image or text)
\(\mathbf{x}_i\)
the unit-length embedding of database image \(i\), \(i = 1 \dots N\)
\(s\)
the score; for unit vectors the dot product equals the cosine, and ranking by cosine equals ranking by L2 distance, because \(\lVert \mathbf{q} - \mathbf{x} \rVert^2 = 2 - 2\,\mathbf{q}^\top \mathbf{x}\)
\(\mathcal{T}_k(\mathbf{q})\)
the top-\(k\) result list, in order

The dual-encoder design of CLIP (session 166) is what makes this cheap: the database side never sees the query, so its vectors are computed once. The cost of search is then a question of memory and arithmetic:

\[ \text{memory} = N \cdot d \cdot b \ \text{bytes}, \qquad \text{flat search} = 2 N d \ \text{FLOPs per query}, \qquad \text{IVF scans} \approx N \cdot \frac{n_{\text{probe}}}{n_{\text{list}}} \ \text{vectors} \tag{169.2} \]
Read the symbols
\(N, d\)
number of database images and embedding size
\(b\)
bytes per number: 4 for float32, 2 for float16; with product quantization each vector is stored as \(M\) one-byte codes instead of \(d \cdot b\) bytes
\(n_{\text{list}}, n_{\text{probe}}\)
IVF (inverted file) splits the database into \(n_{\text{list}}\) clusters with k-means and, at query time, scans only the \(n_{\text{probe}}\) clusters whose centers are closest to the query
Worked example: 100 million images, \(d = 768\)

Float32 storage: \(10^8 \times 768 \times 4 = 307.2\) GB. Float16 halves it to 153.6 GB. Flat (exact) search costs \(2 \times 10^8 \times 768 = 153.6\) GFLOP per query and must read all 153.6 GB; at a memory bandwidth of 2 TB/s that is about 77 ms per query just to stream the vectors, before any batching. With IVF, \(n_{\text{list}} = 65{,}536\) and \(n_{\text{probe}} = 32\), each query scans \(10^8 \times 32 / 65{,}536 \approx 48{,}800\) vectors, a 2,048 times smaller fraction. With product quantization (PQ) into \(M = 64\) one-byte codes, storage falls to \(10^8 \times 64 = 6.4\) GB (plus 0.8 GB for 8-byte ids), small enough for one GPU. The price is recall: IVF can miss a neighbour that sits in an unscanned cluster, and PQ scores are approximate.

The index families (flat, IVF, PQ, IVF-PQ, HNSW graphs) and their recall-versus-speed trade-offs are the subject of session 151; FAISS implements all of them [Certain]. What matters here is the system shape. DINOv2’s data pipeline used exactly this: GPU FAISS indices with inverted files and product quantization codes, on 20 nodes with 8 V100 GPUs each, in less than two days for the whole LVD-142M build [Certain].

Algorithm 1: a two-stage image retrieval system
  1. offline input: database images \(x_{1..N}\), an encoder \(f\) (CLIP, SigLIP, DINOv2, SSCD …)
  2. for each image: \(\mathbf{x}_i \leftarrow f(x_i) / \lVert f(x_i) \rVert\) // same preprocessing as at query time
  3. optionally learn PCA-whitening on a separate training set and apply it, then L2-normalize again
  4. build an index over \(\{\mathbf{x}_i\}\): flat for \(N\) up to a few million, IVF-PQ or HNSW beyond // formula (169.2)
  5. online input: a query \(q\) (image, or text for a CLIP-style model)
  6. \(\mathbf{q} \leftarrow\) normalized embedding of \(q\); \(\ C \leftarrow \mathcal{T}_{K}(\mathbf{q})\) from the index with \(K\) large (100 to 1,000) // stage 1: fast, approximate, high recall
  7. re-rank \(C\) with a more expensive score: exact cosine, query expansion, spatial verification, or a cross-encoder / VLM // stage 2: slow, precise, only on \(K\) items
  8. return the top \(k \ll K\) of the re-ranked list

Measuring retrieval: recall@k, AP, mAP and micro-AP

Before re-ranking, we need a ruler. For one query \(q\), let \(\text{rel}_q(i) = 1\) if the result at rank \(i\) is relevant and 0 otherwise, and let \(R_q\) be the number of relevant items in the whole database (found or not).

Recall@k has two definitions in use, and papers do not always say which. Image–text benchmarks (Flickr30k, MS-COCO) use the “hit” version: the fraction of queries with at least one relevant item in the top \(k\). Classical information retrieval uses the fraction of the \(R_q\) relevant items found in the top \(k\):

\[ \text{R@}k_{\text{hit}} = \frac{1}{|Q|} \sum_{q \in Q} \mathbb{1}\Big[ \sum_{i=1}^{k} \text{rel}_q(i) \ge 1 \Big], \qquad \text{R@}k_{\text{frac}}(q) = \frac{1}{R_q} \sum_{i=1}^{k} \text{rel}_q(i) \tag{169.3} \]

Precision@k is \(\frac1k \sum_{i \le k} \text{rel}_q(i)\). Average precision summarizes the whole ranking in one number: walk down the list, and every time you meet a relevant item, record the precision at that rank; then divide by \(R_q\), so relevant items that were never returned count as zero:

\[ \text{AP}(q) = \frac{1}{R_q} \sum_{i=1}^{n} \text{P@}i(q) \cdot \text{rel}_q(i) \tag{169.4} \]
\[ \text{mAP} = \frac{1}{|Q^{+}|} \sum_{q \in Q^{+}} \text{AP}(q), \qquad Q^{+} = \{ q : R_q \ge 1 \} \tag{169.5} \]
Read the symbols
\(n\)
the length of the returned list
\(\text{P@}i(q)\)
precision of the first \(i\) results of query \(q\)
\(Q^{+}\)
queries that have at least one relevant item; mAP is undefined for a query with \(R_q = 0\), so such queries are simply skipped

mAP averages over queries, so it is also called macro-AP. Only the order inside each query matters: you could add 0.5 to every score of one query and mAP would not change. That is exactly right for search, and exactly wrong for detection. A copy detector outputs (query, reference, confidence) triples and must decide “copy” or “not copy” with one threshold for all queries. ISC21 therefore uses micro-AP: pool the candidate pairs of all queries, sort them by confidence, and compute one precision–recall curve [Certain]:

\[ \mu\text{AP} = \sum_{i=1}^{M} p(i)\, \Delta r(i), \qquad p(i) = \frac{\text{TP}(i)}{i}, \quad r(i) = \frac{\text{TP}(i)}{P} \tag{169.6} \]
Read the symbols
\(M\)
the number of returned pairs, over all queries together, sorted by confidence (highest first)
\(\text{TP}(i)\)
the number of true copy pairs among the first \(i\) pairs
\(P\)
the total number of true copy pairs in the ground truth
\(\Delta r(i)\)
\(r(i) - r(i-1)\): it is \(1/P\) when pair \(i\) is a true copy and 0 otherwise

Two consequences. First, a pair returned for a distractor query (a query that has no copy in the database) is always a false positive and lowers µAP; mAP ignores such queries entirely. Second, confidences must be comparable across queries: a query whose wrong candidates all score 0.83 pushes false positives above the true copies of other queries.

Worked example: three queries, five results each

Query 1 has \(R = 3\) relevant items and its list is (relevant, no, relevant, no, relevant) with scores 0.92, 0.88, 0.85, 0.80, 0.78. Its AP is \(\frac13\big(\frac11 + \frac23 + \frac35\big) = 0.756\). Query 2 has \(R = 2\), but only one is returned, at rank 2: AP \(= \frac12 \cdot \frac12 = 0.250\). Query 3 is a distractor: \(R = 0\), all five results are wrong, with scores 0.83 down to 0.62. mAP skips it: \((0.756 + 0.250)/2 = 0.503\).

For µAP, pool all 15 pairs and sort them. The order begins 0.92 (copy), 0.88, 0.85 (copy), 0.83 (from query 3), 0.80, 0.78 (copy), … and the fifth and last true pair, query 2’s 0.69, is at position 10. With \(P = 5\): \(\mu\text{AP} = \frac15\big(\frac11 + \frac23 + \frac36 + \frac{4}{10}\big) = 0.513\). Remove the distractor query and the same true pairs give 0.568. Same rankings per query, lower µAP: the distractor’s high scores cost 0.055.

relevant (true copy)not relevant
Figure 2. Ranking metrics (169.3) to (169.5) per query and micro-AP (169.6) over the pooled pairs. Top: one row per query, results in rank order with their confidence. Bottom: all pairs of the included queries, sorted by confidence, as the micro-AP sees them. “Missed” counts relevant items that are in the database but not in the list (query 2 has one).
How to read Figure 2

The start state is the worked example: AP 0.756 and 0.250, mAP 0.503, µAP 0.513. Click the cell “0.69” of query 2 to make it not relevant: query 2 now has AP 0, mAP drops to 0.378, and µAP becomes 0.542 (one fewer true pair, so \(P = 4\)). Reset, then click query 3’s first cell (0.83) to make it a true copy: query 3 now counts in mAP with AP 1.000, mAP rises to 0.669, and µAP rises to 0.597 because a high-confidence false positive became a true positive. Press “Query 3 removed” to see the pooled list without the distractor.

Re-ranking: spend more compute on fewer candidates

Stage 1 is built for recall: get the right images somewhere in the top 100 to 1,000, fast. Stage 2 is built for precision: put them at the very top. Because it only touches \(K\) candidates, it can be 1,000 times more expensive per item. Four common re-rankers:

  • Exact re-scoring. If stage 1 used compressed PQ codes, recompute the exact cosine with the full vectors for the \(K\) candidates. Cheap and almost always worth it.
  • Query expansion (QE). The top results of a good query are usually correct. Average them with the query and search again: the new query is closer to the center of the relevant group, so it also finds relevant images that looked different from the original query. Average query expansion (AQE) averages the top \(n_{\text{QE}}\) results with equal weights; \(\alpha\)-weighted query expansion (αQE, Radenović et al.) weights each by its similarity to the power \(\alpha\), which makes it much less sensitive to the choice of \(n_{\text{QE}}\) [Certain].
  • Neighbourhood methods. k-reciprocal re-ranking (Zhong et al., 2017) trusts a candidate more if the query is also among the candidate’s own nearest neighbours [Certain]. Diffusion methods spread scores along the graph of database neighbours.
  • Stronger models. Spatial verification with local features (match keypoints and check that one geometric transform explains them, classically with RANSAC) for instance and copy matching; a cross-encoder or a VLM for text-to-image search, where word order and counting matter and the dual encoder is weak (session 166, section on limits).
\[ \mathbf{q}' = \frac{\mathbf{q} + \sum_{i=1}^{n_{\text{QE}}} w_i\, \mathbf{x}_{(i)}}{\big\lVert \mathbf{q} + \sum_{i=1}^{n_{\text{QE}}} w_i\, \mathbf{x}_{(i)} \big\rVert}, \qquad w_i = \big( \mathbf{q}^\top \mathbf{x}_{(i)} \big)^{\alpha} \tag{169.7} \]
Read the symbols
\(\mathbf{x}_{(i)}\)
the database vector at rank \(i\) of the first search
\(n_{\text{QE}}\)
how many top results join the new query
\(\alpha\)
sharpness of the weights; \(\alpha = 0\) gives plain AQE (all weights 1); a large \(\alpha\) lets only the most similar results vote (negative similarities are clipped to 0)
\(\mathbf{q}'\)
the expanded query, used for a second search

Query expansion has a failure mode called query drift: if the first results are mostly wrong, averaging them pulls the query towards the wrong group, and the second search is worse than the first. Figure 3 shows both cases on a toy 2-D embedding, where every vector is a point on the unit circle and the cosine is the cosine of the angle between two points.

class A (relevant)class Bclass Cquery and expanded query
Figure 3. \(\alpha\)-weighted query expansion (169.7) on 60 database points in three classes (20 each, seeded). The query belongs to class A, so the 20 class-A points are relevant. Left: the circle; ringed points are the top 5 of the first search. Right: AP, precision@5 and the hit recall@1 before and after expansion. Points are drawn slightly off the circle only so that they do not overlap.
How to read Figure 3

At the start (query at 25°, \(n_{\text{QE}} = 5\), \(\alpha = 3\)) all top 5 are class-A points. Expansion moves the query from 25° to 17.6°, towards the center of class A, and AP rises from 0.790 to 0.923. Move the query to 35°: only 2 of the top 5 are class A, and expansion drags the query to 40.3°, towards class B. This is query drift: AP falls from 0.506 to 0.405 and precision@5 from 0.4 to 0.2. At 40° even the top-1 result flips from relevant to not relevant. Back at 25°, set \(n_{\text{QE}} = 15\): with \(\alpha = 0\) (plain AQE) the far, weak neighbours vote as much as the close ones and the gain shrinks (AP 0.839); \(\alpha = 10\) gives them small weights and recovers part of it (0.868). In this toy the effect is small; on real benchmarks Radenović et al. found AQE hard to tune across datasets and αQE more stable [Certain].

Algorithm 2: re-ranking with \(\alpha\)-weighted query expansion
  1. input: unit query \(\mathbf{q}\), index over \(\{\mathbf{x}_i\}\), \(n_{\text{QE}}\), \(\alpha\), final list size \(k\)
  2. \((s_{1..n}, j_{1..n}) \leftarrow\) top \(n_{\text{QE}}\) scores and ids for \(\mathbf{q}\) // first search
  3. \(w_i \leftarrow \max(s_i, 0)^{\alpha}\) for \(i = 1 \dots n_{\text{QE}}\)
  4. \(\mathbf{q}' \leftarrow \text{normalize}\big(\mathbf{q} + \sum_i w_i\, \mathbf{x}_{j_i}\big)\) // formula (169.7)
  5. return the top \(k\) for \(\mathbf{q}'\) // second search; for copy detection keep the first-search scores as confidences

A related trick from the same paper, database-side augmentation (DBA), applies the same averaging to every database vector offline, with its own neighbours [Certain]. It costs nothing at query time. Both QE and DBA assume that neighbours share the label; they help landmark and class retrieval and can hurt copy detection, where a neighbour that looks alike is precisely the thing that must not match.

Classification on frozen features: zero-shot, few-shot, linear probe

Now the second job: assign a label. The encoder is frozen (its weights do not change); we only build a small classifier on top of its embeddings. There are three regimes, ordered by how many labeled images you have.

Zero-shot (no labeled images). With a CLIP or SigLIP model, embed one prompt per class, “a photo of a {label}.”, and use the normalized text embeddings \(\mathbf{t}_k\) as the rows of a linear classifier: predict \(\argmax_k \mathbf{u}^\top \mathbf{t}_k\) (formula 166.9 in session 166). Templates and prompt ensembles help; the classes are defined by words.

Linear probe (many labels). Train multinomial logistic regression on frozen image features:

\[ \min_{W}\ \frac{1}{n} \sum_{i=1}^{n} -\log \softmax\big(W \mathbf{u}_i\big)_{y_i} + \lambda \lVert W \rVert^2 \tag{169.8} \]
Read the symbols
\(\mathbf{u}_i, y_i\)
frozen embedding and label of labeled image \(i\); \(n\) images in total
\(W\)
a \(K \times d\) weight matrix, one row per class (plus a bias in practice)
\(\lambda\)
L2 regularization strength; with few labels it must be large, or \(W\) fits noise

A linear probe is also the standard way to compare encoders: CLIP evaluates every model this way, on the features before the final projection, with scikit-learn’s L-BFGS logistic regression (at most 1,000 iterations) and a validation sweep of \(\lambda\) from \(10^{-6}\) to \(10^{6}\) [Certain]. session 171 runs this protocol.

Few-shot (1 to 16 labels per class). A linear probe trained from scratch on 1 example per class is very noisy. A simple alternative is the class prototype: the normalized mean of the class’s labeled embeddings, used exactly like a prompt embedding. Even better, keep the text prompt as a prior and let the labeled images move it, as if the prompt were worth \(\kappa\) labeled images:

\[ \mathbf{w}_k = \frac{\kappa\, \mathbf{t}_k + \sum_{i:\, y_i = k} \mathbf{u}_i}{\big\lVert \kappa\, \mathbf{t}_k + \sum_{i:\, y_i = k} \mathbf{u}_i \big\rVert} \tag{169.9} \]
Read the symbols
\(\mathbf{t}_k\)
the zero-shot text embedding of class \(k\)
\(\kappa\)
how many labeled images the prompt is worth; with no labels, \(\mathbf{w}_k = \mathbf{t}_k\) (zero-shot); with \(\kappa = 0\), \(\mathbf{w}_k\) is the prototype

Formula (169.9) is a teaching device, not a named method. Published methods do the same in other ways: Tip-Adapter keeps the zero-shot logits and adds a “cache model” built from the few-shot images, with no training [Certain]:

\[ \text{logits}(\mathbf{f}) = \mathbf{f}\, W_c^\top + \alpha\, \varphi\big(\mathbf{f}\, F_{\text{train}}^\top\big)\, L_{\text{train}}, \qquad \varphi(x) = e^{-\beta (1 - x)} \tag{169.10} \]
Read the symbols
\(\mathbf{f}\)
the normalized test image feature (a row vector)
\(W_c\)
CLIP’s zero-shot text classifier, one row per class
\(F_{\text{train}}, L_{\text{train}}\)
the features of all few-shot training images, and their one-hot labels
\(\alpha, \beta\)
how much the cache counts, and how sharply it decays with distance

CoOp learns the words of the prompt itself as continuous vectors, from a few labeled shots [Certain]. WiSE-FT goes further: it fine-tunes the whole model, then averages the fine-tuned weights with the zero-shot weights, to keep robustness to distribution shift [Certain].

What the CLIP paper measured

The CLIP paper compares the three regimes on the same feature space [Certain]:

  • Averaged over 20 datasets, zero-shot CLIP matches a 4-shot linear probe on CLIP’s own features. It also nearly matches the best 16-shot linear probe over all public models in their suite (a BiT-M ResNet-152x2).
  • The number of labeled examples per class that a linear probe needs to match zero-shot varies widely: median 5.4, mean 20.8, from below 1 (Flowers102 and EuroSAT, where zero-shot is worse than a 1-shot probe) to 184. On ImageNet, zero-shot matches a 16-shot linear probe.
  • Why can zero-shot beat 1-shot? The paper’s explanation: words specify the concept directly, while one example image contains many concepts and many hypotheses fit it. They tried using the zero-shot weights as a prior for few-shot training (an L2 penalty towards them), but the hyperparameter search often chose such a strong penalty that the result was “just” the zero-shot classifier.
  • With all labels, the linear probe wins: on ImageNet, ViT-L/14@336px goes from 76.2% zero-shot to 85.4% with a logistic-regression probe, +9.2 points. But its average accuracy on ImageNet distribution-shift test sets slightly decreases.

Figure 4 reproduces this pattern with synthetic embeddings, where you control how good the prompts are.

zero-shot (prompts only)prototypes (\(\kappa = 0\))prompt + shots (169.9)linear probe (169.8)
Figure 4. Classification on frozen synthetic embeddings: 10 classes in 32 dimensions, 500 test images. Image embeddings have isotropic noise plus strong noise along 4 shared “nuisance” directions (think lighting or background); prompts have no nuisance but are offset from the true class center by the prompt noise. The linear probe is logistic regression with weight decay, trained with Adam for 200 full-batch steps. The labeled shots are fixed seeded draws, so the curves do not change when you move the sliders of the prompt.
How to read Figure 4

With the default prompt noise (1.4), zero-shot gets 55.8%. One labeled image per class is much worse: prototypes 25.6% and the linear probe 26.0%, because a single image carries its own nuisance. Prompt + shots with \(\kappa = 4\) keeps the prior and reaches 58.4%. At 2 shots the probe (45.4%) is still below zero-shot; at 4 shots it passes (69.0%), which is the same shape as CLIP’s “zero-shot ≈ 4-shot probe” average. At 32 shots the probe reaches 97.0%, because it learns to ignore the nuisance directions, which a mean-based classifier cannot do (prototypes 75.4%). Set the prompt noise to 0: zero-shot is 85.6%, and adding one noisy shot with \(\kappa = 4\) lowers it to 82.8%; with a perfect prompt, a larger \(\kappa\) is better. Set the noise to 3: zero-shot drops to 28.6%, and from 4 shots on the prompt no longer helps (52.8% with \(\kappa = 4\) against 53.0% for prototypes).

Algorithm 3: choosing a classifier on a frozen encoder
  1. input: frozen encoder, class names, \(n\) labeled images per class (possibly 0), a validation split
  2. if the encoder has a text tower: build the zero-shot classifier \(\mathbf{t}_{1..K}\) with templates // always a baseline
  3. if \(n = 0\): return zero-shot
  4. if \(n\) is small (1 to 16): build prompt + shots (169.9) or Tip-Adapter (169.10); pick \(\kappa\) (or \(\alpha, \beta\)) on validation
  5. else: train the linear probe (169.8), sweeping \(\lambda\) on validation // usually wins with many labels
  6. compare every candidate with zero-shot on the validation split, and also on a shifted split if you have one // probes can lose robustness

Copy detection with global descriptors: SSCD

The third job is the strictest. The ISC21 benchmark (DISC21) states it like this [Certain]: a reference set of 1 million images; a separate training set of 1 million images from the same distribution, with no labels; and query sets of 50,000 images, of which 10,000 are edited copies of reference images and 40,000 are distractors with no copy in the reference set. Edits include crops, rotations, filters, text, emoji and overlays, screenshots and even deep fakes. ISC21 had two tracks: a matching track with any method, and a descriptor track, where each image is one vector of at most 256 dimensions [Certain].

Production systems usually work in two stages: a fast global descriptor (one vector per image) produces a shortlist, and a slower verification stage, often local feature matching, checks it [Certain, as described by Pizzi et al.]. SSCD (Self-Supervised Copy Detection) is a strong global descriptor. Its recipe, starting from SimCLR (session 165) [Certain]:

  • Backbone and pooling. A ResNet-50 trunk, then GeM pooling instead of average pooling, then a linear projection to a 512-d descriptor (the SimCLR MLP head is replaced and kept at inference), then L2 normalization.
  • Augmentations that look like real copies: strong blur, then rotations, text, emoji and JPEG compression, and finally mixup and cutmix, which create partial copies.
  • Loss. InfoNCE between two augmented views of each image, with temperature 0.05 (SimCLR uses 0.1), plus a KoLeo entropy term with weight \(\lambda = 30\).
  • Post-processing. PCA-whitening learned on the DISC21 training set, then L2 normalization; optionally score normalization with a background set.
  • Training. Batch 4,096, LARS optimizer, 100 epochs, resolution 224; inference with the short side resized to 288.

GeM (generalized mean) pooling computes, for each channel of the final feature map, a power mean over all positions:

\[ g_c = \Big( \frac{1}{|\Omega|} \sum_{\omega \in \Omega} x_{c,\omega}^{\,p} \Big)^{1/p} \tag{169.11} \]
Read the symbols
\(x_{c,\omega}\)
activation of channel \(c\) at spatial position \(\omega\) (non-negative after ReLU)
\(\Omega\)
all spatial positions of the feature map
\(p\)
\(p = 1\) is average pooling, \(p \to \infty\) is max pooling; SSCD fixes \(p = 3\), because learning \(p\) made it grow without bound during contrastive training [Certain]

For a channel with values (1, 2, 3, 10) over four positions, average pooling gives 4.000, GeM with \(p = 3\) gives \(((1 + 8 + 27 + 1000)/4)^{1/3} = 6.374\), and max pooling gives 10. GeM leans towards the strongest local evidence, which survives crops and overlays better than an average over the whole image.

The KoLeo term (from the Kozachenko–Leonenko estimator of differential entropy) pushes each descriptor away from its nearest neighbour in the batch that is not one of its own copies:

\[ \mathcal{L}_{\text{KoLeo}} = -\frac{1}{N} \sum_{i=1}^{N} \log \Big( \min_{j \notin \hat{P}_i} \lVert \mathbf{z}_i - \mathbf{z}_j \rVert \Big) \tag{169.12} \]
\[ \mathcal{L}_{\text{SSCD}} = \mathcal{L}_{\text{InfoNCE}} + \lambda\, \mathcal{L}_{\text{KoLeo}}, \qquad \lambda = 30 \tag{169.13} \]
Read the symbols
\(\mathbf{z}_i\)
the unit descriptor of augmented image \(i\) in the batch
\(\hat{P}_i\)
image \(i\) itself and its positives (other augmentations of the same source, including mixed images)
\(\lambda\)
the entropy weight; the paper found accuracy rising up to 30 and training unstable above 40 [Certain]

Because the loss is the log of the nearest-neighbour distance, it pushes hard on vectors that are very close and almost not at all on vectors that are already far apart. The effect is a uniform spread of descriptors over the sphere. This has two benefits for copy detection [Certain]: distances in dense and sparse regions become comparable, so one threshold works better across queries; and the descriptor uses all its dimensions (plain SimCLR collapsed to about 256 of its 512 dimensions).

The results on DISC21 (µAP, %, ResNet-50 unless stated; “+SN” is with score normalization) [Certain, Pizzi et al. Table 2]:

DescriptorTrained onDimsµAPµAP +SN
SimCLR trunk featuresImageNet204813.133.9
Multigrain (supervised)ImageNet204820.541.7
DINO ViT-B/16ImageNet150032.253.8
SimCLRCD (all SSCD changes except KoLeo)ImageNet204839.856.8
SSCD, strong blurImageNet51250.464.5
SSCD, advanced augmentations + mixupDISC2151261.572.5
SSCDlarge (ResNeXt-101)DISC21102463.775.3

Two lessons. A good general-purpose descriptor (DINO) is far from a good copy descriptor: SSCD with a smaller network and fewer dimensions beats it by 18 to 29 points of µAP. And score normalization adds 11 to 22 points to every row, which tells you how much of the difficulty is calibration across queries, not ranking.

Score normalization: one threshold for all queries

Some images live in crowded parts of the embedding space (sunsets, cats, text screenshots): every image near them scores high. Others are unique. With one global threshold, the crowded queries produce confident false positives. Score normalization subtracts a per-query bias estimated from a background set of images known to contain no copies (for DISC21, the training set) [Certain]:

\[ \hat{s}(q, r) = s(q, r) - \beta \cdot \underbrace{\frac{1}{1 + n_{\text{end}} - n} \sum_{i=n}^{n_{\text{end}}} s(q, b_i)}_{\text{bias}(q)} \tag{169.14} \]
Read the symbols
\(s(q, r)\)
similarity of query \(q\) and reference image \(r\)
\(b_i\)
the \(i\)-th nearest neighbour of \(q\) in the background set
\(n \dots n_{\text{end}}\)
which background neighbours are averaged
\(\beta \ge 0\)
how much of the bias is removed

SSCD folds the bias into the vectors, so a normal inner-product index still works: the query becomes \([\mathbf{z}_q,\ -\text{bias}(q)]\) and each reference becomes \([\mathbf{z}_r,\ 1]\) [Certain]. The price: the vectors are no longer unit length, and an image that is truly similar to the background distribution becomes harder to detect.

Worked example: why precision collapses with scale

ISC21’s own example [Certain]: with 50,000 queries (10,000 true copies) against 1 million references, one threshold gives recall 0.6 at precision 0.8. That is 6,000 true positives and \(6{,}000/0.8 - 6{,}000 = 1{,}500\) false positives. Now grow the reference set to 10 million images without adding copies. Recall cannot change (there are no new positives), but false matches grow roughly with the number of references: 15,000. Precision becomes \(6{,}000 / 21{,}000 = 0.286\). This is why copy detection at web scale runs at very high thresholds (high precision, low recall), and why a verification stage follows the shortlist.

true copy queries (top-1 is the source)distractor queries and wrong top-1threshold
Figure 5. A synthetic copy detection benchmark: 1,000 reference images and 1,000 background images in 32 dimensions, where 30% of all images come from one dense region (“popular content”). 100 queries are edited copies of references (edit = added noise), 400 are distractors. Each query returns its top-1 reference with its score. The readout gives micro-AP (169.6) over these 500 pairs, and precision and recall at the threshold, also extrapolated to a 10 times larger reference set.
How to read Figure 5

At the start (edit strength 0.8, \(\beta = 0\), threshold 0.60) the histograms overlap: distractors from the dense region score as high as true copies. µAP is 0.203, and the threshold gives precision 0.353 (85 true and 156 false detections). Raising the threshold makes it worse, not better: at 0.80 precision is 0.234, because the highest-scoring pairs are distractors from the dense region. Now slide \(\beta\) to 0.7: µAP rises to 0.784. All scores shift down, so move the threshold to 0.34: precision 0.945 at recall 0.690. With 10 times more references the same threshold gives precision 0.633 only. Too much normalization also hurts: \(\beta = 1.5\) gives µAP 0.716. Raise the edit strength to 2: only 30 of 100 copies still have their source as top-1, and no normalization can rescue the others. Lower it to 0.2: µAP is 1.000 even without normalization.

Algorithm 4: a copy detection pipeline with micro-AP evaluation
  1. input: reference images, background images (no copies), queries, descriptor \(f\) (SSCD), \(\beta\), \(n \dots n_{\text{end}}\)
  2. embed and whiten all images; index the references; index the background separately
  3. for each query \(q\): \(\text{bias}(q) \leftarrow \text{mean}\big(s(q, b_n), \dots, s(q, b_{n_{\text{end}}})\big)\) from the background index
  4.   candidates \(\leftarrow\) top-\(k\) references of \(q\); \(\hat{s} \leftarrow s - \beta\, \text{bias}(q)\) // formula (169.14)
  5.   optionally verify the top candidates with local feature matching
  6. pool all (query, reference, \(\hat{s}\)) triples of all queries; sort by \(\hat{s}\), highest first
  7. walk down the list: \(\text{TP}(i)\), \(p(i) = \text{TP}(i)/i\), and add \(p(i)/P\) at each true pair // formula (169.6)
  8. choose the operating threshold on a held-out set for the precision you need, then re-check it when the reference set grows

Robustness to edits: what global descriptors handle and what they miss

Every invariance a copy descriptor has, it learned from an augmentation. The SSCD paper and the ISC21 paper discuss these cases [Certain unless tagged]:

EditGlobal descriptor (SSCD)Why, and what helps
Resize, JPEG, blur, color filters, grayscaleRobustDirectly in the training augmentations; strong blur gives a low-frequency bias that ignores high-frequency noise
Text, emoji, small overlaysRobustTrained with text and emoji augmentations; side effect: SSCD is nearly text-insensitive, so it is imprecise on images made only of text
Moderate crops, flips, rotationsMostly robustRandom crops, flips and rotations in training; very strong crops leave little shared content [Likely]
Partial copies: collages, an image pasted into a screenshot or memeWeakOne vector for the whole image is dominated by the rest of the composite; mixup and cutmix help a little; local descriptors or region crops are needed
Different photos of the same scene (landmarks)False positivesNot copies by definition, but they look alike; verification or stricter thresholds needed
Adversarial edits made to evade detectionUnknown in generalCopy detection is adversarial; publishing a detector also informs evaders, which the SSCD authors discuss as an ethical trade-off

The PyTorch version

One deterministic script on synthetic embeddings, CPU only, a few seconds. It builds a database, runs exact search with recall@k and mAP, re-ranks with αQE, compares zero-shot, prototypes, prompt + shots and a linear probe, computes GeM, and measures micro-AP of a copy detector with and without score normalization.

import torch, torch.nn.functional as F
torch.manual_seed(0)

# 1) synthetic CLIP-like embeddings: K classes, unit vectors in D dims
K, D = 20, 64
centers = F.normalize(torch.randn(K, D), dim=-1)
nuisance = torch.linalg.qr(torch.randn(D, 4))[0].T   # 4 directions such as lighting or background
def sample(n_per_class, noise=1.6):
    y = torch.arange(K).repeat_interleave(n_per_class)
    x = centers[y] + noise * torch.randn(len(y), D) / D ** 0.5 + 0.5 * torch.randn(len(y), 4) @ nuisance
    return F.normalize(x, dim=-1), y
db, y_db = sample(50)                          # 1,000 indexed images
qs, y_q = sample(5)                            # 100 query images
prompts = F.normalize(centers + 1.4 * torch.randn(K, D) / D ** 0.5, dim=-1)  # "a photo of a {label}."

# 2) exact search: cosine = dot product of unit vectors, formula (169.1)
def search(q, k):
    return (q @ db.T).topk(k, dim=-1)               # scores and indices, shape (Q, k)
def recall_at_k(idx, yq):                           # hit version of formula (169.3)
    return (y_db[idx] == yq[:, None]).any(1).float().mean().item()
def mean_ap(q, yq):                                 # formulas (169.4)-(169.5) over the full ranking
    order = (q @ db.T).argsort(dim=-1, descending=True)
    rel = (y_db[order] == yq[:, None]).float()
    prec = rel.cumsum(1) / torch.arange(1, rel.shape[1] + 1)
    return ((prec * rel).sum(1) / rel.sum(1)).mean().item()
_, idx = search(qs, 10)
print("recall@1 %.3f  recall@10 %.3f  mAP %.3f" % (recall_at_k(idx[:, :1], y_q), recall_at_k(idx, y_q), mean_ap(qs, y_q)))

# 3) re-ranking with alpha-weighted query expansion, formula (169.7)
def alpha_qe(q, n=10, alpha=3.0):
    s, i = search(q, n)
    w = s.clamp(min=0) ** alpha                     # top neighbours vote with weight sim^alpha
    return F.normalize(q + (w[..., None] * db[i]).sum(1), dim=-1)
q2 = alpha_qe(qs)
_, idx2 = search(q2, 10)
print("after aQE: recall@1 %.3f  recall@10 %.3f  mAP %.3f" % (recall_at_k(idx2[:, :1], y_q), recall_at_k(idx2, y_q), mean_ap(q2, y_q)))

# 4) zero-shot vs few-shot vs linear probe, formulas (169.8)-(169.9)
test, y_test = sample(50)
acc = lambda W: ((test @ W.T).argmax(1) == y_test).float().mean().item()
print("zero-shot (prompts):          %.3f" % acc(prompts))
for shots in [1, 4, 16, 64]:
    x, y = sample(shots)
    protos = F.normalize(torch.stack([x[y == k].mean(0) for k in range(K)]), dim=-1)
    prior = F.normalize(torch.stack([4 * prompts[k] + x[y == k].sum(0) for k in range(K)]), dim=-1)
    probe = torch.zeros(K, D, requires_grad=True)
    opt = torch.optim.Adam([probe], lr=0.05)
    for _ in range(300):                            # logistic regression on frozen features
        loss = F.cross_entropy(20 * x @ probe.T, y) + 1e-3 * probe.pow(2).sum()
        opt.zero_grad(); loss.backward(); opt.step()
    print("%2d-shot: prototypes %.3f  prompt+shots %.3f  linear probe %.3f" % (shots, acc(protos), acc(prior), acc(probe.detach())))

# 5) GeM pooling of a CNN feature map, formula (169.11)
fmap = torch.tensor([1., 2., 3., 10.]).view(1, 1, 2, 2)
gem = lambda f, p: f.clamp(min=1e-6).pow(p).mean(dim=(-2, -1)).pow(1 / p)
print("GeM p=1 %.3f  p=3 %.3f  max %.1f" % (gem(fmap, 1).item(), gem(fmap, 3).item(), fmap.max().item()))

# 6) copy detection: micro-AP with and without score normalization, formulas (169.6) and (169.14)
def micro_ap(scores, is_pos, n_pos):
    o = scores.argsort(descending=True); tp = is_pos[o].float().cumsum(0)
    prec = tp / torch.arange(1, len(o) + 1)
    return (prec * is_pos[o].float()).sum().item() / n_pos
dense = F.normalize(torch.randn(1, D), dim=-1)      # a popular kind of image: many look alike
def photos(n):
    tight = torch.rand(n, 1) < 0.3                  # 30% of photos come from the dense region
    x = torch.where(tight, dense + 0.6 * torch.randn(n, D) / D ** 0.5, torch.randn(n, D))
    return F.normalize(x, dim=-1)
ref, bg = photos(2000), photos(2000)                # reference set and a background set
src = torch.arange(200)                             # 200 queries are edited copies of ref[0:200]
copies = F.normalize(ref[src] + 1.0 * torch.randn(200, D) / D ** 0.5, dim=-1)
queries = torch.cat([copies, photos(800)])          # plus 800 distractor queries with no copy
s, j = (queries @ ref.T).max(1)                     # top-1 candidate per query
is_pos = torch.zeros(1000, dtype=torch.bool); is_pos[:200] = j[:200] == src
bias = (queries @ bg.T).topk(10, dim=1).values.mean(1)          # mean of 10 background neighbours
print("micro-AP raw %.3f  normalized %.3f" % (micro_ap(s, is_pos, 200), micro_ap(s - 1.0 * bias, is_pos, 200)))

Output with PyTorch 2.8 on CPU (two runs give identical output): exact search gives recall@1 0.760, recall@10 0.990 and mAP 0.352. Recall@10 is nearly perfect while mAP is low, because each query has 50 relevant images and mAP asks for all of them near the top. αQE raises mAP to 0.397 and recall@1 to 0.800, while recall@10 drops slightly to 0.980: expansion helps on average and hurts a few queries (drift). Zero-shot reaches 0.695. With 1 shot per class: prototypes 0.416, prompt + shots 0.765, linear probe 0.452. With 4 shots: 0.729, 0.832 and 0.816. With 16 shots: 0.930, 0.933 and 0.952. With 64 shots: 0.953, 0.948 and 0.981. GeM prints 4.000, 6.374 and 10.0. Copy detection: micro-AP 0.242 raw and 0.811 with score normalization. Exact values can differ slightly across PyTorch versions. [Likely]

Where this is used

ApplicationWhich job, and how
Text-to-image and image-to-image searchRetrieval with CLIP or SigLIP embeddings in an ANN index; re-rank the top candidates with a cross-encoder or VLM (session 170, session 219)
Product and catalog classification, content taggingZero-shot with written class names on day one; prompt + shots or a linear probe once labels arrive; new categories need only a sentence
Training-data curationDeduplicate with a copy detector, remove near-duplicates of benchmark test sets, and retrieve images close to curated sets (DINOv2’s LVD-142M) (session 187)
Content moderation and copyrightCopy detection against a database of known images, one global threshold, verification before action; ISC21 is the benchmark
Text-to-video retrievalEmbed sampled frames, pool them over time, search with a text embedding; recall@k on MSR-VTT-style benchmarks (session 208)
Video fingerprinting and video copy detectionPer-frame copy descriptors, then temporal alignment of matching frames to find copied segments, even when re-encoded or cropped (session 209)
Zero-shot action recognition and video classificationPrompts such as “a video of a person {action}.” against pooled frame embeddings; a linear probe on clip features when labels exist
Key-moment detectionScore every sampled frame against a text query and keep the peaks; a threshold per video, so calibration matters as in copy detection
Video-RAG and long-video question answeringFrame or clip embeddings form the index; retrieve the moments relevant to the question, then let a video-LLM answer (session 210)
Open-vocabulary detectionThe same zero-shot rule applied to region embeddings instead of whole images (session 168)

Common mistakes

  • One descriptor for all jobs. A semantic embedding (CLIP, DINOv2) flags every cat as a copy of every other cat; a copy descriptor (SSCD) is a poor classifier. Choose the descriptor for the kind of “same” you need.
  • Evaluating a detector with mAP. mAP ignores distractor queries and score scales across queries. If production uses one threshold, report micro-AP and precision at your operating recall.
  • Not saying which recall@k. “Hit” recall and “fraction of relevant” recall can differ a lot when a query has many relevant items. State the definition, and the database size: recall@10 among 1,000 images is not recall@10 among 1 million.
  • Fixing a threshold on a small reference set. False positives grow with the reference set, so precision falls (0.8 becomes 0.29 at ten times the size in the ISC21 example). Re-validate thresholds as the database grows.
  • Forgetting normalization and preprocessing. Database and query must use the same resize, crop, whitening and L2 normalization. A missing normalization silently turns cosine search into dot-product search that favours long vectors.
  • Training a few-shot linear probe from scratch. With 1 to 4 labels per class it can be worse than zero-shot. Keep the text prior (169.9, Tip-Adapter) or regularize strongly, and always compare against zero-shot.
  • Query expansion everywhere. QE and DBA help when neighbours share the label; they cause drift for hard queries and can hurt copy detection. Measure per-query, not only on average.
  • Leaking test images. Web-scale training sets contain copies of benchmark images. Deduplicate training data against every test set with a copy detector before you trust a number, as DINOv2 did.

Interview questions

Design image search over 1 billion photos with text queries.core
Use a dual encoder (CLIP or SigLIP). Offline: embed all images, L2-normalize, optionally reduce dimension, and build an approximate index, typically IVF-PQ or HNSW, sharded across machines. At \(d = 768\) in float16 the raw vectors are 1.5 TB, so compression matters: PQ at 64 bytes per vector is 64 GB. Online: embed the text, retrieve the top 1,000 by approximate inner product, re-score them exactly with full vectors, then re-rank the top 50 to 100 with a stronger cross-encoder or VLM if quality needs it (counting, word order, small text). Measure recall@k of the ANN stage against exact search, and end-to-end recall@k and nDCG on labeled queries. Deduplicate the gallery with a copy detector so results are not ten copies of one image.
What is the difference between mAP and micro-AP? When do you use each?core
mAP (macro-AP) computes AP per query and averages over queries; only the order inside each query matters, and queries with no relevant item are skipped. Micro-AP pools all (query, candidate, confidence) pairs from all queries, sorts them globally and computes one precision–recall curve; a high-scoring wrong pair from any query, including a distractor query with no true match, lowers it. Use mAP for search, where each user sees one ranked list. Use micro-AP for detection with one global threshold, such as copy detection (the ISC21 metric). In the worked example, the same per-query rankings give mAP 0.503 and µAP 0.513, and removing a distractor query raises µAP to 0.568 while mAP does not change.
Zero-shot, few-shot or linear probe: which do you choose?core
With no labels: zero-shot with good class names and prompt ensembles. With a few labels per class: keep the text prior and adapt it (prompt + shots, Tip-Adapter, CoOp), because a probe trained from scratch on 1 to 4 shots is noisy; CLIP’s own measurement is that zero-shot matches a 4-shot linear probe on average over 20 datasets. With many labels: a linear probe with a regularization sweep, which on ImageNet takes CLIP ViT-L/14@336px from 76.2% to 85.4%. Then check robustness: the CLIP paper found the probe slightly lowered average accuracy on ImageNet distribution-shift sets. If the domain is far from web data (medical, satellite), zero-shot can be poor and labels matter more.
Why can zero-shot beat a 1-shot classifier on the same features?basic
A prompt names the concept directly; one example image contains the concept plus its background, lighting, pose and other objects, and a classifier built from it cannot know which part matters. In embedding terms, the single image is the class center plus a large nuisance offset, while the prompt embedding is offset by the text–image mismatch only. Averaging more shots cancels the nuisance, and a linear probe can learn to ignore nuisance directions, so with enough labels the image-based classifier wins. In Figure 4 (default settings): zero-shot 55.8%, 1-shot prototypes 25.6%, 1-shot probe 26.0%; the probe passes zero-shot at 4 shots.
How would you build an image copy detector? What does SSCD change compared with SimCLR?advanced
Two stages: a global descriptor plus ANN shortlist, then verification (local feature matching) for high precision. SSCD is the descriptor: SimCLR-style self-supervised contrastive training where the augmentations mimic real copies (strong blur, text, emoji, JPEG, rotations, mixup and cutmix for partial copies); GeM pooling with \(p = 3\) instead of average pooling; a linear projection to 512-d kept at inference instead of a discarded MLP; a lower temperature (0.05); and a KoLeo entropy term with weight 30 that spreads descriptors uniformly. At inference: PCA-whitening learned on an unlabeled training set, L2 normalization, and optional score normalization against a background set. On DISC21 this moves µAP from 13.1 (SimCLR trunk) to 61.5 (72.5 with score normalization).
What does the KoLeo (entropy) regularizer do, and why does it help copy detection but hurt classification?advanced
\(\mathcal{L}_{\text{KoLeo}} = -\frac1N \sum_i \log \min_{j \notin \hat P_i} \lVert z_i - z_j \rVert\) pushes each descriptor away from its nearest non-copy neighbour in the batch, strongly when they are close, weakly when far. The result is a near-uniform distribution on the sphere. For copy detection that is ideal: copies cluster tightly, and everything else, even same-class images, stays far, so one distance threshold means the same thing everywhere and all dimensions are used (SimCLR had collapsed to about 256 of 512). For classification you want same-class images to form dense clusters, and the uniformity prior destroys that; SSCD’s ImageNet linear-probe accuracy drops as the weight grows.
Precision is 0.8 at recall 0.6 on 1 million references. What happens at 10 million?core
Recall stays 0.6 if no new copies are added. False positives grow roughly linearly with the number of references. With 10,000 true copies among the queries: 6,000 true positives and 1,500 false positives at 1 million; about 15,000 false positives at 10 million; precision \(6{,}000/21{,}000 = 0.29\). So thresholds must be re-tuned for scale, systems operate at high precision and low recall, and a verification stage is needed. This is the ISC21 paper’s own example.
Explain query expansion. When does it hurt?core
After a first search, combine the query with its top results and search again. AQE averages the top \(n_{\text{QE}}\) equally; αQE weights each by its similarity to the power \(\alpha\), so far neighbours count little and the choice of \(n_{\text{QE}}\) matters less. It helps when the top results are mostly relevant: the new query moves towards the center of the relevant group and finds relevant images that looked different. It hurts when the first results are mostly wrong (query drift), when the relevant set is a single item (copy detection, where a look-alike neighbour must not match), and it doubles the search cost.
What is score normalization, and why does it raise micro-AP but not much mAP?advanced
Subtract from each query’s scores a bias equal to (\(\beta\) times) its average similarity to its nearest neighbours in a background set known to have no copies: \(\hat s(q, r) = s(q, r) - \beta\,\text{bias}(q)\). Queries in crowded regions of the space, which score high against everything, are pulled down; unique queries barely move. The subtraction is the same for all candidates of one query, so the order within a query, and therefore mAP, does not change at all. Micro-AP compares scores across queries, so it improves a lot: 11 to 22 points for every model in the SSCD table. SSCD stores the bias as an extra dimension so a normal inner-product index still works.
Why GeM pooling for retrieval instead of average pooling?basic
GeM computes \((\text{mean}\,x^p)^{1/p}\) per channel; \(p = 1\) is average pooling, large \(p\) approaches max pooling. Average pooling dilutes a distinctive local feature with all the background; max pooling keeps only one location and is noisy. With \(p = 3\) the strong local activations dominate but several locations contribute: for (1, 2, 3, 10) average gives 4, GeM gives 6.37, max gives 10. For retrieval and copy detection, where a crop or overlay changes much of the image but the distinctive part remains, this works better. SSCD found that learning \(p\) diverged in contrastive training, so it fixes \(p = 3\).
Your recall@10 is 0.99 but mAP is 0.35 on the same queries. Is something wrong?basic
Not necessarily. Hit recall@10 asks whether at least one relevant item is in the top 10. mAP asks whether all relevant items are ranked near the top. If each query has 50 relevant images (a whole class), finding one is easy, and ranking all 50 above every other class is hard. This is exactly the output of the PyTorch script. Report the metric that matches the product: a user who needs one good result cares about recall@k; a system that must collect all matches cares about AP.

Exercises

  1. A query has \(R = 4\) relevant items. The top 6 results are (relevant, no, no, relevant, relevant, no), and the fourth relevant item is not returned. Compute P@5, the fraction recall@5 and AP.
  2. Show that adding a constant \(c\) to every score of one query does not change its AP. Does it change micro-AP?
  3. How many bytes does a flat float32 index of 1 million SSCD descriptors take? And of the 1,024-d SSCDlarge descriptors in float16?
  4. Compute GeM with \(p = 2\) for the values (1, 2, 3, 10).
  5. With formula (169.9), \(\kappa = 4\) and 4 labeled shots per class, what fraction of the un-normalized class vector comes from the prompt? Which way should you move \(\kappa\) if your prompts are poor?
  6. A copy detector returns one candidate per query for 1,000 queries; 200 are true copies, all found, and they have the top 200 scores; the 800 distractor candidates have the lower scores. What is µAP? Now one distractor scores above all true copies: what is µAP?
Answerscheck
1. \(\text{P@}5 = 3/5 = 0.6\). \(\text{Recall@}5 = 3/4 = 0.75\). AP = \(\frac14\big(\frac11 + \frac24 + \frac35\big) = \frac14(1 + 0.5 + 0.6) = 0.525\). 2. Adding \(c\) to every score of one query keeps the order inside that query, so the ranked list, every P@i and the AP are unchanged. Micro-AP can change, because the shifted pairs move up or down relative to the pairs of other queries in the pooled list. 3. \(10^6 \times 512 \times 4 = 2.048\) GB; \(10^6 \times 1{,}024 \times 2 = 2.048\) GB as well. 4. \(((1 + 4 + 9 + 100)/4)^{1/2} = 28.5^{1/2} = 5.339\). 5. The prompt weight is \(\kappa = 4\) unit vectors against the sum of 4 unit image vectors, so about half of the “votes” (exactly half if all vectors are aligned; the sum of the shots is shorter when they disagree, so the prompt share is then larger). Poor prompts: lower \(\kappa\). 6. All 200 true pairs come first: \(p(i) = 1\) at each, \(\mu\text{AP} = 1.000\). With one distractor on top, the true pairs sit at positions 2 to 201, \(p = i/(i+1)\) for the \(i\)-th true pair: \(\mu\text{AP} = \frac{1}{200}\sum_{i=1}^{200} \frac{i}{i+1} = 0.976\).

Key takeaways

  • One embedding serves three jobs with three meanings of “same”: same class (search, classification), same object (instance retrieval), same source image (copy detection). Pick the descriptor for the job: CLIP, SigLIP or DINOv2 for meaning, SSCD for copies.
  • Retrieval is embed once, index, search (169.1). Memory and cost grow as \(N d b\) and \(2Nd\) (169.2); IVF, PQ and HNSW trade a little recall for orders of magnitude in speed and memory. A second stage re-ranks a few hundred candidates with something better: exact scores, αQE (169.7), verification or a cross-encoder.
  • Recall@k (two definitions), AP and mAP (169.3 to 169.5) judge rankings per query. Micro-AP (169.6) pools all pairs and judges one global threshold; distractor queries and score calibration matter.
  • On frozen features: zero-shot needs no labels, a linear probe (169.8) wins with many labels, and with a few labels you should keep the text prior (169.9, Tip-Adapter 169.10). Zero-shot CLIP matches a 4-shot linear probe on average; the full ImageNet probe adds 9.2 points (76.2% to 85.4%).
  • SSCD turns SimCLR into a copy detector: copy-like augmentations, GeM pooling (169.11), a KoLeo entropy term with weight 30 (169.12 to 169.13), whitening and score normalization (169.14). On DISC21 it reaches 61.5 µAP (72.5 with normalization), against 32.2 for DINO features.
  • At scale, false positives grow with the reference set, so precision falls (0.8 to 0.29 for ten times more references). Thresholds must be re-checked, and verification follows the shortlist.

Key formulas

\[ s(\mathbf{q}, \mathbf{x}) = \mathbf{q}^\top \mathbf{x}, \qquad \text{AP}(q) = \frac{1}{R_q} \sum_{i} \text{P@}i \cdot \text{rel}_q(i), \qquad \mu\text{AP} = \sum_{i} p(i)\, \Delta r(i) \]
\[ \mathbf{q}' = \text{normalize}\Big(\mathbf{q} + \sum_{i \le n_{\text{QE}}} (\mathbf{q}^\top \mathbf{x}_{(i)})^{\alpha}\, \mathbf{x}_{(i)}\Big), \qquad \mathbf{w}_k \propto \kappa\, \mathbf{t}_k + \sum_{i:\,y_i = k} \mathbf{u}_i \]
\[ g_c = \Big(\tfrac{1}{|\Omega|} \textstyle\sum_{\omega} x_{c,\omega}^{\,p}\Big)^{1/p}, \qquad \mathcal{L}_{\text{SSCD}} = \mathcal{L}_{\text{InfoNCE}} - \frac{\lambda}{N} \sum_i \log \min_{j \notin \hat P_i} \lVert \mathbf{z}_i - \mathbf{z}_j \rVert, \qquad \hat s = s - \beta\, \text{bias}(q) \]

References for this session

  1. Pizzi et al. (2022). A Self-Supervised Descriptor for Image Copy Detection
  2. Douze et al. (2021). The 2021 Image Similarity Dataset and Challenge
  3. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  4. Pizzi et al. (2022). A Self-Supervised Descriptor for Image Copy Detection
  5. Douze et al. (2021). The 2021 Image Similarity Dataset and Challenge
  6. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  7. Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision
  8. Johnson et al. (2017). Billion-scale similarity search with GPUs
  9. Malkov & Yashunin (2016). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
  10. Jégou et al. (2011). Product Quantization for Nearest Neighbor Search
  11. Radenović, Tolias & Chum (2018). Fine-tuning CNN Image Retrieval with No Human Annotation
  12. Zhong, Zheng, Cao & Li (2017). Re-ranking Person Re-identification with k-reciprocal Encoding
  13. Zhou, Yang, Loy & Liu (2022). Learning to Prompt for Vision-Language Models
  14. Zhang et al. (2022). Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
  15. Wortsman et al. (2022). Robust fine-tuning of zero-shot models
  16. Meta (code). FAISS library