Training embedding and retrieval models
An embedding model turns a text, an image or a clip into one vector, so that similar meaning gives nearby vectors. This session shows how such models are trained with contrastive learning: the InfoNCE loss and its temperature, in-batch and hard negatives, where positive pairs come from (SimCSE, Contriever, E5), how to evaluate a retriever, and why CLIP is the same recipe with an image encoder.
What you will learn
- Why a bi-encoder needs a contrastive loss: pull positive pairs together, push negatives apart, and avoid collapse
- The InfoNCE loss, its gradient (each negative is pushed in proportion to its softmax probability), and the mutual-information bound log N
- What the temperature does, with real values from SimCSE, E5, Contriever and CLIP
- In-batch negatives, hard negatives, false negatives, MoCo queues and GradCache, with the measured effect of batch size
- Where positive pairs come from (dropout, crops, web pairs, labels) and the two-stage recipe with a cross-encoder teacher
- How to evaluate a retriever: Recall@k, MRR, nDCG@10, BEIR, MTEB, alignment and uniformity
- How CLIP uses the same loss in both directions, and how this carries over to video retrieval
Why this matters
A search engine, a RAG system, a duplicate detector and a recommendation feed all need the same thing: a function that turns a text (or an image, or a video clip) into one vector, so that similar meaning gives nearby vectors. Then search becomes “find the nearest vectors”, which is fast even over a billion items. This session is about how such an embedding model is trained.
In session 33 we saw the problem: raw BERT vectors are poor for similarity, and Sentence-BERT fixed it by fine-tuning on sentence pairs. Today we go one step further. Almost every modern embedding model (SimCSE, Contriever, E5, BGE, GTE, and the image-text model CLIP) is trained with one family of losses: contrastive learning. The idea is simple: show the model a pair that belongs together (a positive pair) and many items that do not (the negatives), and train it so that the positive scores higher than every negative.
The details decide the quality: how many negatives, how hard they are, the temperature, where the positive pairs come from, and how you measure the result. In the previous session, session 59, a small model learned from a teacher’s soft probabilities. We will meet that idea again here: the best retrievers are often trained with a cross-encoder teacher.
The task: one vector per text, and similarity as a dot product
An embedding model is an encoder \(f_\theta\) (often a BERT-style transformer, now also decoder LLMs) followed by pooling: the token vectors are turned into one vector, usually by averaging them (mean pooling) or by taking the [CLS] or the last token (session 12). We then divide by the length so the vector has norm 1. The similarity of a query \(q\) and a document \(d\) is the cosine:
This design is called a bi-encoder (or dual encoder): the query and the document are encoded separately. That is the whole point. All documents can be encoded once, offline, and stored in a vector index (session 151). At query time we encode only the query and do a nearest-neighbor search. A cross-encoder reads the query and the document together and outputs one score; it is more accurate, but it needs one forward pass per (query, document) pair, so it is used only to rerank the top few dozen results. Figure 7 at the end compares the two, and shows that CLIP is a bi-encoder with an image tower.
The training question is: how do we change \(\theta\) so that \(s(q, d^+)\) is high for a relevant document \(d^+\) and low for everything else?
Contrastive learning: pull positives together, push negatives apart
We cannot just maximize \(s(q, d^+)\). The easiest way to do that is to map every input to the same vector: then every cosine is 1, and the model is useless. This failure is called collapse. So the loss must also have a term that pushes apart things that do not belong together. That is the core of contrastive learning: a pull on positive pairs and a push on negative pairs.
The oldest form is the triplet loss: take an anchor \(a\), a positive \(p\) and a negative \(n\), and ask the positive to win by a margin \(m\):
With \(m = 0.2\), \(s(a, p) = 0.8\) and \(s(a, n) = 0.7\), the loss is \(0.1\): the negative is too close. With \(s(a, n) = 0.5\) the loss is 0 and nothing is learned from that triplet. Sentence-BERT offered a triplet loss as one of its training objectives. [Certain] The weakness: one negative per anchor gives little signal, and most random negatives are already far away, so most triplets give zero gradient. The fix used by every modern model is to compare the positive with many negatives at once.
Figure 1 shows what the pull and the push do. Eight pairs of points live on a circle (2-D unit vectors). At the start they are crowded in a narrow arc, like raw BERT vectors that all point into one narrow cone: every cosine is high, so cosine says almost nothing. Train with the full contrastive loss, then with “positives only”.
How to read Figure 1
With InfoNCE, each pair first snaps together (pull), then the pairs spread around the whole circle (push), and the mean cosine between non-matching points drops from 0.919 to −0.143. That is the lowest possible value for 8 evenly spaced points: \(-1/7\). With “positives only”, the pairs also snap together, but they stay crowded in the arc: the mean negative cosine stays at 0.938. In a real network with shared weights, the pull alone would go further and collapse everything to one point. Move \(\tau\): at \(\tau = 1\) the loss cannot go below about 1.315 even with perfect geometry, at \(\tau = 0.1\) it can go down to about 0.10.
InfoNCE: a softmax over one positive and many negatives
Take one query \(q\), its positive \(d^+\), and \(N - 1\) negatives \(d_1^-, \dots, d_{N-1}^-\). Treat the problem as classification: among \(N\) candidates, which one is the positive? Turn the similarities into probabilities with a softmax, and use cross-entropy (session 3). This is the InfoNCE loss (van den Oord et al., 2018, from Contrastive Predictive Coding): [Certain]
- Read the symbols
- \(s(q, d)\)
- the cosine similarity (60.1), between −1 and 1
- \(\tau\)
- the temperature, a small positive number (0.01 to 0.1 in practice); dividing by it stretches the scores before the softmax
- \(N\)
- the number of candidates: one positive plus \(N - 1\) negatives
- the fraction
- \(p^+\), the probability the model gives to the positive; the loss is \(-\log p^+\)
Contrast this with the triplet loss. Here every negative contributes, and the softmax decides how much each one matters. How much? Look at the gradient with respect to each score. Write \(p^+\) for the positive’s probability and \(p_j\) for negative \(j\)’s probability:
Read it as forces. The positive is pulled with strength \((1 - p^+)/\tau\). Each negative is pushed with strength \(p_j/\tau\), proportional to its own softmax probability. So a negative that is already far away (small \(p_j\)) gets almost no push, and a negative that looks like the positive (a hard negative, large \(p_j\)) gets most of it. The total push equals the pull: \(\sum_j p_j = 1 - p^+\).
A query has \(s(q, d^+) = 0.8\) and two negatives with scores 0.5 and 0.3.
With \(\tau = 1\): the logits are 0.8, 0.5, 0.3; the softmax gives \(p = (0.426, 0.316, 0.258)\); the loss is \(-\log 0.426 = 0.853\). Push on the negatives: 0.316 and 0.258, so the closer negative gets 55% of the total push.
With \(\tau = 0.1\): the logits are 8, 5, 3; \(p = (0.946, 0.047, 0.006)\); the loss is \(0.055\). Gradients (60.4): pull \(-0.535\), push \(0.471\) and \(0.064\). Now the closer negative gets 88% of the push.
A lower temperature makes the loss focus on the hardest negatives. That is the main job of \(\tau\).
What the temperature does
Cosines live in \([-1, 1]\). With \(\tau = 1\), the largest possible gap between two logits is 2, which is too small for a confident softmax. Example: with 64 candidates, even if the positive had cosine 1 and all 63 negatives had cosine −1, \(p^+\) would be only \(e/(e + 63/e) = 0.105\), so the loss could never go below 2.254. Dividing by a small \(\tau\) removes this limit. On the other side, a very small \(\tau\) makes the softmax almost a hard max: only the single hardest negative gets a gradient, training becomes noisy, and false negatives (see below) hurt more.
Real values: SimCSE uses \(\tau = 0.05\); its ablation on the STS-B dev set (Spearman × 100, supervised model) gave 84.9 at \(\tau = 0.001\), 85.4 at 0.01, 86.2 at 0.05, 82.0 at 0.1 and 64.0 at 1. [Certain] E5 uses \(\tau = 0.01\), Contriever 0.05. [Certain] CLIP learns \(\tau\): it starts at 0.07 and the logit scale \(1/\tau\) is capped at 100, so \(\tau\) never goes below 0.01. [Certain] The widely used MultipleNegativesRankingLoss in the sentence-transformers library multiplies cosines by a scale of 20, which is \(\tau = 0.05\). [Likely]
How to read Figure 2
Start at the default (\(\tau = 0.05\)): the hard negative gets 99.7% of all the push, and the four easy ones almost nothing. Raise \(\tau\) to 1: the push is spread almost evenly, and the loss stays high even when the positive is clearly the best. Move the hard negative above the positive: the loss jumps, which is exactly what should happen. Then imagine that this “hard negative” is in fact a relevant document that nobody labeled (a false negative): at low \(\tau\), the model is pushed hard in the wrong direction.
Why it is called InfoNCE: a bound on mutual information
The CPC paper showed that the loss gives a lower bound on the mutual information between the two views (how much knowing one tells you about the other): [Certain]
Two useful facts follow. First, a model that knows nothing gives every candidate the same probability \(1/N\), so its loss is \(\log N\): for 64 candidates that is 4.159 nats. Use this as a sanity check at step 0 (our PyTorch code below prints 4.172 for random vectors). Second, the bound can never exceed \(\log N\): with few negatives, the loss cannot tell a good model from a great one. This is one reason why more negatives help.
Where the negatives come from
In-batch negatives
Encoding extra negatives costs forward and backward passes. The classic trick reuses the batch itself. Take \(B\) pairs \((q_i, d_i^+)\). Stack the query vectors into \(Q \in \R^{B \times k}\) and the passage vectors into \(P \in \R^{B \times k}\). One matrix multiplication gives all \(B^2\) scores, \(S = QP^\top/\tau\). Row \(i\) is a classification problem whose correct answer is column \(i\); the other \(B - 1\) passages in the row are the negatives (DPR, Karpukhin et al., 2020): [Certain]
This is just cross-entropy over the rows of \(S\), with labels \(0, 1, \dots, B - 1\). It is free: the negatives were encoded anyway as other examples’ positives. And it scales: the number of negatives per query grows with the batch size. The evidence is strong:
- DPR, Natural Questions dev set, top-20 retrieval accuracy: 7 in-batch negatives 69.1%, 31 gave 70.8%, 127 gave 73.0%. [Certain]
- E5 contrastive pre-training, average over 6 BEIR datasets (nDCG@10): batch 1K gave 45.8, 8K gave 50.2, 32K gave 51.6. [Certain]
Figure 3 runs formula (60.6) by hand on a batch of four pairs, one row at a time, so you can see every number go through the loop. The vectors are hand-made 6-number unit vectors that stand in for an encoder’s output (rounded to 2 decimals).
| query \(q_i\) | its passage \(d_i^+\) | hard negative \(d_i^-\) | |
|---|---|---|---|
| 1 | capital of France? | Paris is the capital of France. | Berlin is the capital of Germany. |
| 2 | how long to boil an egg | Boil eggs for 9 to 12 minutes. | Poach an egg for 3 minutes. |
| 3 | sort a list in python | Use sorted(xs) or xs.sort(). | Sort a dict by its values in python. |
| 4 | shoes for a marathon | Light road shoes for long races. | Boots for long mountain hikes. |
- \(\mathbf{q}_i, \mathbf{p}_j\) (and \(\mathbf{h}_j\)) \(\leftarrow\) unit vectors from the encoder
- for \(i = 1, \dots, B\): // one row = one classification problem
- \(S_{ij} \leftarrow \mathbf{q}_i^\top \mathbf{c}_j / \tau\) for every candidate \(j\) // (60.6)
- \(P_{ij} \leftarrow e^{S_{ij}} / \sum_k e^{S_{ik}}\) // softmax of the row
- \(\ell_i \leftarrow -\log P_{ii}\); push on \(j\): \(P_{ij}/\tau\) // (60.3), (60.4)
- \(\mathcal{L} \leftarrow \frac{1}{B} \sum_i \ell_i\) // cross_entropy(S, arange(B))
How to read Figure 3
Press Play, or Step through it. Line 3 fills a row with dot products (the panel writes out \(\mathbf{q}_i^\top \mathbf{p}_i\) term by term), line 4 turns the row into probabilities, line 5 reads off the loss. With the default \(\tau = 0.1\) and in-batch negatives only, every positive gets \(P_{ii} \approx 0.9995\) and the batch loss is 0.0005: the other topics are easy negatives, and this batch teaches almost nothing. Now tick “add the hard negatives” and play again. Row 2 changes the most: “boil an egg” has cosine 0.986 with its passage but 0.952 with “Poach an egg”, so \(P_{22}\) falls to 0.583, \(\ell_2 = 0.539\), and the hard negative takes 99.7% of the push. The batch loss rises to 0.154. Finally set \(\tau = 1\): every positive still has the highest score, but the softmax is flat and the loss stays at 0.785 (1.452 with hard negatives). That is the high floor from the temperature section.
The price is memory. For \(B = 32{,}768\) (the batch of E5 and of CLIP), the score matrix alone has \(32{,}768^2 \approx 1.07\) billion entries: 4.3 GB in FP32. In practice the batch is spread over many GPUs, each GPU gathers the vectors of all others (an all-gather, session 43), and computes its own rows. [Likely] When even the encoder activations do not fit, GradCache first computes all embeddings without storing activations, computes the loss and its gradient with respect to the embeddings, and then re-runs the encoder in small chunks to backpropagate. Memory then stays almost constant as \(B\) grows. [Certain]
Hard negatives
Most in-batch negatives are easy: a question about tax law against a passage about football. Formula (60.4) says easy negatives give almost no gradient. A hard negative is a document that looks relevant but is not: it shares words or topic with the query. Common sources:
- BM25 negatives. The top results of a classic word-matching search that do not contain the answer. DPR adds one per question. [Certain]
- Mined negatives. Search with the current (or a previous) embedding model and take high-ranked non-positives. This is what our PyTorch code does in its second stage. ANCE refreshes these negatives during training with an asynchronously updated index. [Likely]
- Dataset structure. In NLI data, every premise has an “entailment” hypothesis (positive) and a “contradiction” hypothesis (hard negative). SimCSE and E5 use this. [Certain]
Hard negatives join the denominator. A common form (SimCSE, DPR) shares them across the batch: every query sees all \(B\) positives and all \(B\) hard negatives, so the score matrix becomes \(B \times 2B\): [Certain]
The gains are large. In DPR, adding one BM25 negative per question (127 in-batch + 128 BM25 negatives) raised top-20 accuracy on the NQ dev set from 73.0% to 78.0%; adding two per question did not help further. [Certain] In supervised SimCSE, contradiction hard negatives raised STS-B dev from 84.9 to 86.2. [Certain] Figure 3 showed it on four pairs; Figure 4 lets you see a larger score matrix and what a hard negative column does to every row.
How to read Figure 4
Without hard negatives, the off-diagonal cells are random passages: their cosines are around 0 and the loss is tiny, so the batch teaches little. Turn on hard negatives: a second block appears with one near-duplicate per query. Its cells are much darker, and the loss grows many times. That larger loss is the useful signal. The symmetric loss adds the column direction (“for this passage, which query?”), which CLIP uses for images and captions.
False negatives: the danger of hard negatives
The harder you mine, the more often the “negative” is in fact relevant: MS MARCO, for example, labels only about one relevant passage per query, and many good passages are unlabeled. [Likely] Pushing those away teaches the model the wrong thing. Common defenses: skip mined candidates that rank very close to the positive, sample negatives from a middle band of ranks (for example ranks 30 to 100) instead of the very top, or let a cross-encoder filter out candidates that it judges relevant. RocketQA made this denoising step popular. [Likely] In-batch negatives have the same risk at a smaller scale: two queries in a batch can share an answer, so remove duplicates when you build batches.
Even more negatives: a queue and a momentum encoder
MoCo (He et al.) keeps a queue of passage vectors from previous batches and uses them as extra negatives. Old vectors were produced by older weights, so to keep them consistent, the passages are encoded by a slowly moving copy of the encoder, the momentum encoder, whose weights \(\theta_k\) follow the trained weights \(\theta_q\) by an exponential moving average: [Certain]
No gradient flows through the queue. Contriever uses MoCo with a queue of 131,072 and \(m = 0.9995\). [Certain] E5 compared the options at the same compute and found plain in-batch negatives with a 32K batch best: 51.6 average, versus 45.5 for MoCo with 130K negatives. [Certain] So with enough GPUs, a big batch is the simplest and strongest choice; the queue is for when you cannot afford one.
- input: batch of \(B\) pairs \((q_i, d_i^+)\), optional hard negatives \(d_i^-\), temperature \(\tau\), encoder \(f_\theta\)
- \(\mathbf{q}_i \leftarrow \operatorname{normalize}(\operatorname{pool}(f_\theta(\text{“query: ”} + q_i)))\) // prefix tells the shared encoder which side it is
- \(\mathbf{p}_i \leftarrow \operatorname{normalize}(\operatorname{pool}(f_\theta(\text{“passage: ”} + d_i^+)))\); same for \(d_i^-\) if present
- gather \(\mathbf{q}, \mathbf{p}\) (and hard negatives) from all GPUs // more negatives for free
- \(S \leftarrow Q\,[P;\, P^-]^\top / \tau\) // shape \(B \times B\) or \(B \times 2B\)
- \(\mathcal{L} \leftarrow \operatorname{CrossEntropy}(S, \text{labels} = [0, 1, \dots, B-1])\) // formulas (60.6), (60.7)
- optional: \(\mathcal{L} \leftarrow \tfrac12 \big(\mathcal{L} + \operatorname{CrossEntropy}(S_{:, :B}^\top, \text{labels})\big)\) // symmetric (60.14)
- backward, clip, AdamW step // as in session 4
Where the positives come from
Negatives are cheap; good positive pairs are the real data problem. Four families, from no labels to many labels:
| Source of positive pairs | Example method | Details |
|---|---|---|
| The same text twice, with different dropout | Unsupervised SimCSE | Encode a sentence twice; the two dropout masks give two slightly different vectors. 1 million Wikipedia sentences, batch 64, lr \(3\times10^{-5}\), \(\tau = 0.05\). [Certain] |
| Two random crops of one document | Contriever | Two spans cut independently from the same document, with 10% of tokens deleted. [Certain] |
| Naturally paired web text | E5 (CCPairs) | (title, passage), (question, upvoted answer), (post, comment), (title, abstract), citation pairs. About 1.3 billion pairs, filtered to about 270 million. [Certain] |
| Human-labeled pairs | Sentence-BERT, supervised SimCSE, DPR, E5 fine-tuning | NLI entailment pairs, MS MARCO (Bing query, relevant passage), Natural Questions. [Certain] |
The SimCSE dropout trick is surprisingly strong. Its ablation on STS-B dev: dropout \(p = 0.1\) gives 82.5; no dropout gives 71.1; and using the same dropout mask for both copies gives 43.6, because then the two views are identical and the task is trivial. [Certain] The noise is the augmentation. On seven STS tasks, unsupervised SimCSE with BERT-base reached an average Spearman of 76.25, and supervised SimCSE 81.57, versus 74.89 for SBERT-base. [Certain]
E5’s data filter is worth knowing for interviews. First train a model on all 1.3 billion noisy pairs. Then, for each pair, rank its passage against 1 million random passages; keep the pair only if its own passage is in the model’s top 2. [Certain] The idea: a network learns the clean, consistent pairs first, so pairs it cannot fit are probably noise. About 21% of the pairs survive.
The two-stage recipe
Most strong text embedding models since 2022 follow the same plan. [Likely] Stage 1, contrastive pre-training: hundreds of millions of weak pairs, in-batch negatives only, a very large batch. Stage 2, fine-tuning: a few hundred thousand to a few million labeled pairs, smaller batch, mined hard negatives, and often a cross-encoder teacher. E5’s fine-tuning loss mixes distillation from the teacher (the KL idea from session 59) with the contrastive loss: [Certain]
- Read the symbols
- \(p_{\text{ce}}\)
- the cross-encoder teacher’s softmax over the candidate passages of one query (positive and hard negatives)
- \(p_{\text{stu}}\)
- the bi-encoder student’s softmax over the same candidates
- \(\mathcal{L}_{\text{cont}}\)
- the InfoNCE loss (60.6) with the hard labels
- \(\alpha\)
- a weight; E5 uses 0.2
The teacher gives graded scores (“this negative is almost relevant”) instead of a hard 0 or 1, which also softens the false-negative problem.
- input: a pretrained encoder (BERT-like or a small LLM), web pairs, labeled pairs, a cross-encoder teacher
- stage 1: clean the web pairs: heuristic filters, then a consistency filter (keep a pair if its passage ranks in the top-\(k\))
- train with in-batch InfoNCE (60.6), batch 32K, \(\tau = 0.01\), 20K steps, lr \(10^{-4}\) to \(3\times10^{-4}\) [Certain for E5]
- stage 2: for each labeled query, mine hard negatives with the stage-1 model; drop likely false negatives
- score (query, candidates) with the teacher \(\rightarrow p_{\text{ce}}\)
- train with (60.9): batch 256, 7 hard negatives per query, 3 epochs, lr \(10^{-5}\) to \(3\times10^{-5}\) [Certain for E5]
- evaluate on held-out in-domain data and on BEIR / MTEB; check every task type, not only the average
Details that matter in practice
- Prefixes. E5 uses one shared encoder and breaks the symmetry with the prefixes “query: ” and “passage: ”. [Certain] Queries are short and passages long; the prefix lets one network treat them differently. Newer models use full task instructions (“Given a web search query, retrieve relevant passages”). [Likely] Forgetting the prefix at inference time is a classic silent bug.
- Shared or separate encoders. DPR uses two separate BERTs (one for questions, one for passages) and a plain dot product. [Certain] Most later models share one encoder. [Likely]
- Normalize. If you train with cosine, normalize at inference too, and use the same similarity in your index.
- Symmetric tasks. For “are these two sentences similar” (no query/passage roles), use the same prefix on both sides and often the symmetric loss.
- Matryoshka embeddings. Add the contrastive loss on the first 64, 128, 256, … dimensions too, so a prefix of the vector is itself a usable embedding. You can then store short vectors for a first pass and long ones for reranking. [Certain]
- LLM-based embedders. Recent top models start from a decoder LLM and use the last token (or a learned attention pooling) as the embedding, trained with the same InfoNCE loss and synthetic data generated by LLMs. [Likely]
Now the runnable version. A toy world: 300 topics, each with 8 typical words. A passage has 3 topic words among 16 tokens; a query has 3 topic words among 6. The encoder is a bag of embeddings with mean pooling and a projection. We train with in-batch InfoNCE, then add one mined hard negative per query for the second half.
import math, torch, torch.nn.functional as F torch.manual_seed(0) # ---- toy data: 300 topics, each topic owns 8 words out of a 2,000-word vocabulary ---- V, T, D = 2000, 300, 64 topic_words = torch.randint(0, V, (T, 8)) def sample(topic, n): # n words: 3 from the topic, the rest random noise own = topic_words[topic, torch.randint(0, 8, (3,))] return torch.cat([own, torch.randint(0, V, (n - 3,))]) docs = torch.stack([sample(t, 16) for t in range(T)]) # one passage per topic q_test = torch.stack([sample(t, 6) for t in range(T)]) # new queries, same topics class BiEncoder(torch.nn.Module): # shared encoder: token vectors -> mean pool -> project def __init__(self): super().__init__() self.emb = torch.nn.EmbeddingBag(V, D, mode="mean") self.proj = torch.nn.Linear(D, D) def forward(self, ids): return F.normalize(self.proj(self.emb(ids)), dim=-1) # unit length: dot = cosine def info_nce(q, p, tau=0.05, hard=None): # formulas (60.6) and (60.7) cand = p if hard is None else torch.cat([p, hard]) # B or 2B candidates logits = q @ cand.T / tau # (B, B) or (B, 2B) labels = torch.arange(q.shape[0]) # positive of row i is column i return F.cross_entropy(logits, labels) def symmetric_info_nce(a, b, tau=0.05): # formula (60.14): rows and columns, as in CLIP logits = a @ b.T / tau labels = torch.arange(a.shape[0]) return 0.5 * (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) @torch.no_grad() def evaluate(model, queries): # Recall@1 and MRR@10 over the whole corpus S = model(queries) @ model(docs).T # (300, 300) cosines rank = (S > S.diag()[:, None]).sum(1) + 1 # rank of the true passage mrr = torch.where(rank <= 10, 1.0 / rank, torch.zeros_like(rank, dtype=torch.float)).mean() return (rank == 1).float().mean().item(), mrr.item() model = BiEncoder() print("before: R@1 %.3f MRR@10 %.3f" % evaluate(model, q_test)) # sanity check: with random unit vectors the loss is close to log B B = 64 u, v = F.normalize(torch.randn(B, D), dim=-1), F.normalize(torch.randn(B, D), dim=-1) print("random vectors: loss %.3f, log B = %.3f" % (info_nce(u, v, tau=1.0).item(), math.log(B))) opt = torch.optim.AdamW(model.parameters(), lr=1e-2, weight_decay=0.01) for step in range(600): idx = torch.randperm(T)[:B] # a batch of B (query, passage) pairs q_ids = torch.stack([sample(t, 6) for t in idx.tolist()]) # fresh training queries each step q, p = model(q_ids), model(docs[idx]) hard = None if step >= 300: # stage 2: mine one hard negative per query with torch.no_grad(): S = q @ model(docs).T S[torch.arange(B), idx] = -1e4 # never pick the true passage hard_ids = S.argmax(1) hard = model(docs[hard_ids]) loss = info_nce(q, p, tau=0.05, hard=hard) opt.zero_grad(); loss.backward(); opt.step() if step % 100 == 99: print(f"step {step+1}: loss {loss.item():.3f} test R@1 %.3f MRR@10 %.3f" % evaluate(model, q_test)) with torch.no_grad(): # both directions on 64 test pairs: query->passage and back print("symmetric test loss %.3f" % symmetric_info_nce(model(q_test[:B]), model(docs[:B])).item())
On CPU with PyTorch 2.8 this prints: before training, Recall@1 0.013 and MRR@10 0.035 (300 passages, so chance is about 0.003); random vectors give a loss of 4.172, close to \(\log 64 = 4.159\); after 600 steps, Recall@1 0.930 and MRR@10 0.957. The loss goes up after step 300 (0.563 to 0.880 at step 400): the hard negatives make the task harder, which is the point. Running the same script with the hard-negative stage turned off ends at Recall@1 0.893. Exact digits can change with the PyTorch version; the pattern does not.
Watch a small retriever learn
Figure 5 runs the same recipe in your browser, on a world small enough to see every number. There are 12 topics in 6 domains, two topics per domain: France and Japan share the words “capital, city”; boiling and poaching share “egg, cook”; sorting and regex share “python, code”; and so on. Each passage has 5 words: its 2 domain words, its 2 topic words and “the”. A training query has 3 words, drawn fresh at every step: one domain word, one topic word and one filler (“how”, “what”, “best”, …). So the other topic of the same domain is the natural hard negative. The model has two towers, as in DPR: a query table \(E_q\) and a passage table \(E_p\), with 8 numbers per word (44 words). A text vector is the mean of its word vectors, normalized to length 1. Training is plain SGD with \(\eta = 0.2\) for 200 steps. The backward pass needs only three rules:
- Read the symbols
- \(P_{ij}\)
- the softmax of row \(i\) of the score matrix \(S\) (60.6)
- \(\delta_{ij}\)
- 1 on the diagonal (the positive) and 0 elsewhere; so the positive is pulled with \((P_{ii} - 1)/B \lt 0\) and every negative is pushed with \(P_{ij}/B \gt 0\), as in (60.4)
- \(\mathbf{c}_j\)
- the unit vector of candidate \(j\) (a positive or a hard negative); the passage side gets the mirror rule \(\tfrac{1}{\tau} \sum_i \tfrac{\partial \mathcal{L}}{\partial S_{ij}} \mathbf{q}_i\)
- \(\mathbf{v}, \mathbf{u}, \mathbf{g}\)
- \(\mathbf{v}\) is the mean of the word vectors, \(\mathbf{u} = \mathbf{v}/\lVert \mathbf{v} \rVert\) the normalized vector, \(\mathbf{g}\) the gradient with respect to \(\mathbf{u}\); normalization removes the part of \(\mathbf{g}\) along \(\mathbf{u}\). Each of the \(n\) words of the text then receives \(\tfrac{1}{n}\, \partial \mathcal{L}/\partial \mathbf{v}\)
- pick \(B\) topics; one fresh query each, and its passage \(d_i^+\)
- \(\mathbf{q}_i \leftarrow \operatorname{normalize}(\text{mean } E_q[\text{words}])\), same for passages with \(E_p\)
- if hard: \(h_i \leftarrow \arg\max_{k \ne i} \mathbf{q}_i^\top \mathbf{p}_k\) over all 12 passages // mined with the current model
- \(S \leftarrow Q\,[P; H]^\top / \tau\), \(\mathcal{L} \leftarrow \operatorname{CE}(S, \text{diagonal})\) // (60.6), (60.7)
- \(\partial \mathcal{L}/\partial S_{ij} = (P_{ij} - \delta_{ij})/B\), back through normalize and mean // (60.10)
- \(E_q, E_p \leftarrow E - \eta\, \nabla E\) // plain SGD, \(\eta = 0.2\)
- Recall@1 of 24 fixed test queries against all 12 passages // (60.11)
How to read Figure 5
Press Step a few times: each training step runs the seven lines in order, and the panel shows the real numbers of the current line. In line 3, look at which passage is mined. In the first 20 steps it is the other topic of the same domain (France for a Japan query) only 41% of the time, because the model still knows little; after step 20 it is that sibling about 80% of the time. Then pick “fast” and press Play. With the defaults (\(\tau = 0.1\), \(B = 4\), no hard negatives), Recall@1 starts at 0.042 (1 of 24 queries) and first reaches 1 at step 149; the last wrong answers are orange dots, the sibling topic. Tick “mine one hard negative per query”: the same run reaches 1 at step 64, and the dashed curve is the run without them. The loss is higher with hard negatives (5.714 instead of 3.794 at step 1), because the task is harder; a higher loss here is a sign of a more useful batch. Try \(\tau = 1\): learning is much slower (Recall@1 0.667 at step 100 without hard negatives). Try \(B = 2\) and \(B = 6\): more in-batch negatives also speed up learning. Why is the step-1 loss above \(\log 4 = 1.386\)? That value is for a model that gives every candidate the same score. Random 8-number unit vectors have cosines spread by about \(1/\sqrt{8} = 0.35\), which at \(\tau = 0.1\) are logits spread by about 3.5, so a random model is often confidently wrong.
The numbers in Figure 5 are a small copy of the PyTorch result above: with hard negatives the loss goes up and retrieval gets better faster. With \(\tau = 0.1\) and \(B = 4\), Recall@1 after 20 steps is 0.458 without hard negatives and 0.667 with them; after 50 steps it is 0.875 and 0.917. A copy of the simulation in Python, with the same random generator, gives the same numbers to 4 decimals. The hard negatives in this world are never false negatives, because every topic has exactly one passage. In real data, mining that aggressively would also pick unlabeled positives, which is why the false-negative defenses above matter. (Recall@1, defined formally in (60.11) below, is the fraction of test queries whose correct passage is ranked first.)
How to evaluate an embedding model
The training loss is not the goal; retrieval quality is. Embed a corpus, run a set of test queries with known relevant documents, and score each ranked list. Three metrics cover most cases. For one query with \(R\) relevant documents in the corpus, a ranked list, and \(\operatorname{rank}_1\) the position of the first relevant result:
Recall@k asks “did we get them into the top \(k\)?”. It is the right metric for the first stage of RAG or a reranking pipeline: whatever is missed here can never be found later. MRR (mean reciprocal rank) asks “how high is the first good result?”; it suits question answering with one answer. nDCG@k (normalized discounted cumulative gain) rewards relevant results more when they are higher, and supports graded relevance:
- Read the symbols
- \(\text{rel}_i\)
- the relevance grade of the result at position \(i\) (0 or 1 for binary labels)
- \(\log_2(i+1)\)
- the discount: position 1 counts fully, position 3 counts half, position 7 counts one third
- \(\text{IDCG@}k\)
- the DCG of the best possible ordering, so a perfect list scores 1
Click a result to mark it relevant or not relevant.
How to read Figure 6
At the default, Recall@10 is 0.75 (3 of 4 found), MRR is 0.5 (first hit at rank 2), and nDCG@10 is 0.532. Move the hit at rank 9 to rank 1 (click 9 off, 1 on): Recall@10 does not change, but MRR jumps to 1 and nDCG rises. Raise \(R\): recall and nDCG drop, MRR does not care. This is why you must report several metrics, and why missing labels (unknown relevant documents) make every number look worse than it is.
Relevant results at ranks 2, 4 and 9, and \(R = 4\). Recall@5 \(= 2/4 = 0.5\); Recall@10 \(= 3/4 = 0.75\); MRR \(= 1/2\).
DCG@10 \(= 1/\log_2 3 + 1/\log_2 5 + 1/\log_2 10 = 0.631 + 0.431 + 0.301 = 1.363\). The ideal list puts all 4 relevant documents first: IDCG@10 \(= 1 + 0.631 + 0.5 + 0.431 = 2.562\). So nDCG@10 \(= 1.363/2.562 = 0.532\).
Benchmarks
- BEIR: 18 public retrieval datasets from different domains (biomedical, finance, argument, fact checking, …), scored with nDCG@10, used zero-shot (the model never trained on them). Its main lesson: BM25 is a strong baseline out of domain. [Certain] E5 was the first model reported to beat BM25 on BEIR without labeled data: an average of 42.9 for E5-PT-base versus 41.7 for BM25 over 15 datasets. [Certain]
- MTEB: 8 task types (retrieval, reranking, clustering, classification, pair classification, STS, summarization, bitext mining), 58 datasets, 112 languages. Its finding: no single model wins every task. [Certain]
- STS (semantic textual similarity): Spearman correlation between the model’s cosines and human similarity scores. Good for paraphrase-style similarity, a weak predictor of retrieval quality. [Likely]
Public leaderboards are a starting point, not an answer. Some top models have trained on the training splits of benchmark datasets, and your domain (legal text, sports commentary, code) may behave very differently. [Likely] Always build a small test set from your own queries, a few hundred labeled examples, and compare models on it (session 150).
Looking inside the space: alignment and uniformity
Wang and Isola showed that the contrastive loss, with many negatives, optimizes two properties of normalized embeddings: positives should be close (alignment), and all embeddings should spread evenly over the sphere (uniformity). They are cheap to measure on a sample: [Certain]
Both should be low. A collapsed model has perfect alignment and terrible uniformity; raw BERT (the narrow cone) is similar. Figure 1 prints the uniformity of its 16 points: from −0.260 at the start to −1.867 when the pairs spread evenly. SimCSE used these two numbers to explain why its dropout trick works: the training keeps alignment and improves uniformity. [Certain]
The same idea as CLIP: our bridge to vision
Replace the query encoder by an image encoder (a ViT) and the passage encoder by a text encoder, take image-caption pairs from the web, and train with in-batch InfoNCE. That is CLIP (Radford et al., 2021): 400 million (image, text) pairs, a batch of 32,768, a linear projection from each encoder into the shared space, and a learned temperature. [Certain] Because both directions matter (find the caption for an image, find the image for a caption), CLIP uses the symmetric loss: cross-entropy over the rows and over the columns of the same score matrix: [Certain]
- Read the symbols
- \(I, T\)
- the \(B \times k\) matrices of normalized image and text embeddings
- first term
- image \(i\) must pick its caption among all \(B\) captions (a row)
- second term
- caption \(i\) must pick its image among all \(B\) images (a column)
Everything in this session carries over. Batch size matters for the same reason (more negatives). Temperature matters (CLIP learns it, with a cap). Hard negatives and false negatives exist (two web images can both match “a dog on a beach”). And evaluation is the same Recall@k, now for text-to-image and image-to-text retrieval. SigLIP replaces the softmax by an independent sigmoid on every cell of the matrix, which removes the need to normalize over the whole batch (session 167). [Certain] The full story is in session 166.
For video, the step is small again. A video-text retriever embeds a clip (for example by pooling the CLIP embeddings of sampled frames, or with a video encoder) and trains with the same symmetric InfoNCE on (clip, caption) pairs (session 208). [Likely] A video fingerprinting model uses two augmented copies of the same clip (crop, re-encode, overlay text) as the positive pair, exactly like SimCSE’s two dropout copies, and other clips as negatives (session 209). [Likely]
Where this is used
| Application | Positive pairs | Negatives and notes |
|---|---|---|
| First-stage retrieval for RAG and search | (query, relevant passage): MS MARCO, NQ, mined web pairs | In-batch plus mined hard negatives; measure Recall@k on your own queries (session 152) |
| Semantic similarity, deduplication, clustering | Paraphrases, NLI entailment, dropout views | Symmetric loss; tune a cosine threshold on labeled pairs |
| Reranking | Same labeled pairs | Cross-encoder trained with a classification or listwise loss; often the teacher of the bi-encoder (60.9) |
| Image-text search and zero-shot classification | (image, caption) from the web | CLIP / SigLIP; huge batches; class names written as text prompts |
| Text-to-video retrieval and key-moment search | (clip, caption), (moment, commentary line) | Neighboring clips from the same video are hard negatives, and often false negatives [Likely] |
| Video fingerprinting and near-duplicate detection | Two augmented copies of one clip | Other clips as negatives; evaluate with precision at a fixed false-positive rate |
| Video-RAG | (question, clip or transcript chunk) | Multimodal bi-encoder over frames and speech text (session 210) |
| Recommendation (two-tower models) | (user history, item clicked) | In-batch negatives over-sample popular items; a log-frequency correction is common [Likely] |
Common mistakes
- Temperature 1 with cosine similarity. The logits then live in \([-1, 1]\), the softmax can never become confident, and training stalls (with 64 candidates the loss cannot go below 2.254). Use \(\tau\) around 0.01 to 0.05, or learn it.
- Forgetting to normalize, or normalizing only one side. Train and serve with the same similarity: cosine (normalized dot product) or raw dot product, never a mix.
- Duplicates in a batch. If two rows share the same positive, one of them is labeled a negative of itself. Deduplicate, or mask those cells.
- Mining the very top results as negatives. Many of them are unlabeled positives. Skip the top few, sample from a band, or filter with a cross-encoder.
- Missing or wrong prefixes and instructions. A model trained with “query: ” / “passage: ” silently loses quality without them. Read the model card.
- Batch split across GPUs without gathering. Each GPU then sees only its local \(B/n\) negatives. Gather the embeddings (with gradients) across GPUs.
- Choosing a model by the leaderboard average. The average mixes 8 task types; check the retrieval column, and test on your own data.
- Changing the embedding model without re-indexing. Vectors from two different models (or two checkpoints) are not comparable. Re-embed the whole corpus.
Interview questions
Write the InfoNCE loss and explain each part.basic
cross_entropy(Q @ P.T / tau, arange(B)).Why can’t we just maximize the similarity of positive pairs?basic
What does the temperature do? What happens if it is too high or too low?core
Why does batch size matter so much in contrastive learning, and how do you get large batches on limited memory?core
What is a hard negative, how do you mine one, and what is the risk?core
Bi-encoder vs cross-encoder: when do you use each?basic
How would you evaluate a new embedding model for your company’s search?core
Explain how SimCSE learns sentence embeddings without labels. Why does it work?advanced
How is CLIP’s loss related to a text retriever’s loss? What changes?core
What does the bound \(I \ge \log N - \mathcal{L}\) tell you in practice?advanced
Your retriever has great training loss but poor Recall@10 on production queries. What do you check?advanced
Exercises
- A model is trained with in-batch negatives, batch 1,024. How many negatives does each query see? What loss do you expect at step 0?
- Compute the InfoNCE loss for \(s(q, d^+) = 0.9\), two negatives with \(s = 0.6\), and \(\tau = 0.1\).
- How much memory does the FP32 score matrix need for one global batch of 65,536 pairs, if it were computed on one GPU?
- A ranked list of 5 results has relevance (0, 0, 1, 0, 1), and the corpus has exactly 2 relevant documents. Compute Recall@5, MRR and nDCG@5.
- CLIP caps \(1/\tau\) at 100 and starts at \(\tau = 0.07\). What are the smallest \(\tau\) and the starting logit scale?
- In Figure 4, turn on hard negatives with \(B = 8\). Which row has the largest loss, and why? Then explain in one sentence why the symmetric loss does not use the hard-negative columns in the column direction.
Answerscheck
Key takeaways
- An embedding model is a bi-encoder: encode each text alone, normalize, compare with a dot product (60.1). Search becomes nearest neighbors; a cross-encoder reranks the top results.
- Contrastive learning pulls positives together and pushes negatives apart; without negatives the space collapses. InfoNCE (60.3) is a softmax over one positive and many negatives; its gradient (60.4) pushes each negative in proportion to its probability.
- The temperature sets the focus: low \(\tau\) (0.01 to 0.05) concentrates on hard negatives; \(\tau = 1\) with cosine leaves a high loss floor.
- In-batch negatives (60.6) are free and grow with the batch (E5: batch 1K → 32K raised BEIR scores from 45.8 to 51.6); hard negatives (60.7) add strong signal but bring false negatives.
- Positives come from dropout (SimCSE), crops (Contriever), web pairs (E5) or labels (NLI, MS MARCO); the standard recipe is weak-pair pre-training, then hard-negative fine-tuning with a cross-encoder teacher (60.9).
- Evaluate with Recall@k, MRR and nDCG@10 (60.11, 60.12) on your own data; BEIR and MTEB are for shortlisting.
- CLIP is the same recipe with an image tower and the symmetric loss (60.14): the bridge from text retrieval to vision and video.
Key formulas
References for this session
- van den Oord et al. (2018). Representation Learning with Contrastive Predictive Coding
- Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Gao et al. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings
- Wang et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Izacard et al. (2021). Unsupervised Dense Information Retrieval with Contrastive Learning
- van den Oord et al. (2018). Representation Learning with Contrastive Predictive Coding
- Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Gao et al. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings
- Wang et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Izacard et al. (2021). Unsupervised Dense Information Retrieval with Contrastive Learning
- Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- He et al. (2019). Momentum Contrast for Unsupervised Visual Representation Learning
- Gao et al. (2021). Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
- Wang & Isola (2020). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- Thakur et al. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Muennighoff et al. (2022). MTEB: Massive Text Embedding Benchmark
- Kusupati et al. (2022). Matryoshka Representation Learning
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training