LLM & VLM courseBlog
60Track 4: Post-training and RL · session 8 of 25

Training embedding and retrieval models

An embedding model turns a text, an image or a clip into one vector, so that similar meaning gives nearby vectors. This session shows how such models are trained with contrastive learning: the InfoNCE loss and its temperature, in-batch and hard negatives, where positive pairs come from (SimCSE, Contriever, E5), how to evaluate a retriever, and why CLIP is the same recipe with an image encoder.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: BeginnerBefore this: Sessions 3, 12 and 33

What you will learn

  • Why a bi-encoder needs a contrastive loss: pull positive pairs together, push negatives apart, and avoid collapse
  • The InfoNCE loss, its gradient (each negative is pushed in proportion to its softmax probability), and the mutual-information bound log N
  • What the temperature does, with real values from SimCSE, E5, Contriever and CLIP
  • In-batch negatives, hard negatives, false negatives, MoCo queues and GradCache, with the measured effect of batch size
  • Where positive pairs come from (dropout, crops, web pairs, labels) and the two-stage recipe with a cross-encoder teacher
  • How to evaluate a retriever: Recall@k, MRR, nDCG@10, BEIR, MTEB, alignment and uniformity
  • How CLIP uses the same loss in both directions, and how this carries over to video retrieval

Why this matters

A search engine, a RAG system, a duplicate detector and a recommendation feed all need the same thing: a function that turns a text (or an image, or a video clip) into one vector, so that similar meaning gives nearby vectors. Then search becomes “find the nearest vectors”, which is fast even over a billion items. This session is about how such an embedding model is trained.

In session 33 we saw the problem: raw BERT vectors are poor for similarity, and Sentence-BERT fixed it by fine-tuning on sentence pairs. Today we go one step further. Almost every modern embedding model (SimCSE, Contriever, E5, BGE, GTE, and the image-text model CLIP) is trained with one family of losses: contrastive learning. The idea is simple: show the model a pair that belongs together (a positive pair) and many items that do not (the negatives), and train it so that the positive scores higher than every negative.

The details decide the quality: how many negatives, how hard they are, the temperature, where the positive pairs come from, and how you measure the result. In the previous session, session 59, a small model learned from a teacher’s soft probabilities. We will meet that idea again here: the best retrievers are often trained with a cross-encoder teacher.

The task: one vector per text, and similarity as a dot product

An embedding model is an encoder \(f_\theta\) (often a BERT-style transformer, now also decoder LLMs) followed by pooling: the token vectors are turned into one vector, usually by averaging them (mean pooling) or by taking the [CLS] or the last token (session 12). We then divide by the length so the vector has norm 1. The similarity of a query \(q\) and a document \(d\) is the cosine:

\[ s(q, d) = \frac{f_\theta(q)^\top f_\theta(d)}{\lVert f_\theta(q) \rVert \, \lVert f_\theta(d) \rVert} = \mathbf{q}^\top \mathbf{d} \quad \text{when } \lVert\mathbf{q}\rVert = \lVert\mathbf{d}\rVert = 1 \tag{60.1} \]

This design is called a bi-encoder (or dual encoder): the query and the document are encoded separately. That is the whole point. All documents can be encoded once, offline, and stored in a vector index (session 151). At query time we encode only the query and do a nearest-neighbor search. A cross-encoder reads the query and the document together and outputs one score; it is more accurate, but it needs one forward pass per (query, document) pair, so it is used only to rerank the top few dozen results. Figure 7 at the end compares the two, and shows that CLIP is a bi-encoder with an image tower.

The training question is: how do we change \(\theta\) so that \(s(q, d^+)\) is high for a relevant document \(d^+\) and low for everything else?

Contrastive learning: pull positives together, push negatives apart

We cannot just maximize \(s(q, d^+)\). The easiest way to do that is to map every input to the same vector: then every cosine is 1, and the model is useless. This failure is called collapse. So the loss must also have a term that pushes apart things that do not belong together. That is the core of contrastive learning: a pull on positive pairs and a push on negative pairs.

The oldest form is the triplet loss: take an anchor \(a\), a positive \(p\) and a negative \(n\), and ask the positive to win by a margin \(m\):

\[ \mathcal{L}_{\text{triplet}} = \max\!\big(0,\; m - s(a, p) + s(a, n)\big) \tag{60.2} \]

With \(m = 0.2\), \(s(a, p) = 0.8\) and \(s(a, n) = 0.7\), the loss is \(0.1\): the negative is too close. With \(s(a, n) = 0.5\) the loss is 0 and nothing is learned from that triplet. Sentence-BERT offered a triplet loss as one of its training objectives. [Certain] The weakness: one negative per anchor gives little signal, and most random negatives are already far away, so most triplets give zero gradient. The fix used by every modern model is to compare the positive with many negatives at once.

Figure 1 shows what the pull and the push do. Eight pairs of points live on a circle (2-D unit vectors). At the start they are crowded in a narrow arc, like raw BERT vectors that all point into one narrow cone: every cosine is high, so cosine says almost nothing. Train with the full contrastive loss, then with “positives only”.

anchor (a query), small dotits positive (a matching document), large dotline: the pair
Figure 1. Contrastive training on the unit circle, 8 pairs. Each pair has its own number. The loss is the symmetric InfoNCE loss (60.14) over the 8 × 8 similarity matrix, trained by gradient descent on the angles. “Uniformity” is formula (60.13): lower means more spread out.
How to read Figure 1

With InfoNCE, each pair first snaps together (pull), then the pairs spread around the whole circle (push), and the mean cosine between non-matching points drops from 0.919 to −0.143. That is the lowest possible value for 8 evenly spaced points: \(-1/7\). With “positives only”, the pairs also snap together, but they stay crowded in the arc: the mean negative cosine stays at 0.938. In a real network with shared weights, the pull alone would go further and collapse everything to one point. Move \(\tau\): at \(\tau = 1\) the loss cannot go below about 1.315 even with perfect geometry, at \(\tau = 0.1\) it can go down to about 0.10.

InfoNCE: a softmax over one positive and many negatives

Take one query \(q\), its positive \(d^+\), and \(N - 1\) negatives \(d_1^-, \dots, d_{N-1}^-\). Treat the problem as classification: among \(N\) candidates, which one is the positive? Turn the similarities into probabilities with a softmax, and use cross-entropy (session 3). This is the InfoNCE loss (van den Oord et al., 2018, from Contrastive Predictive Coding): [Certain]

\[ \mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp\!\big(s(q, d^+)/\tau\big)}{\exp\!\big(s(q, d^+)/\tau\big) + \sum_{j=1}^{N-1} \exp\!\big(s(q, d_j^-)/\tau\big)} \tag{60.3} \]
Read the symbols
\(s(q, d)\)
the cosine similarity (60.1), between −1 and 1
\(\tau\)
the temperature, a small positive number (0.01 to 0.1 in practice); dividing by it stretches the scores before the softmax
\(N\)
the number of candidates: one positive plus \(N - 1\) negatives
the fraction
\(p^+\), the probability the model gives to the positive; the loss is \(-\log p^+\)

Contrast this with the triplet loss. Here every negative contributes, and the softmax decides how much each one matters. How much? Look at the gradient with respect to each score. Write \(p^+\) for the positive’s probability and \(p_j\) for negative \(j\)’s probability:

\[ \frac{\partial \mathcal{L}}{\partial s(q, d^+)} = -\frac{1 - p^+}{\tau}, \qquad \frac{\partial \mathcal{L}}{\partial s(q, d_j^-)} = \frac{p_j}{\tau} \tag{60.4} \]

Read it as forces. The positive is pulled with strength \((1 - p^+)/\tau\). Each negative is pushed with strength \(p_j/\tau\), proportional to its own softmax probability. So a negative that is already far away (small \(p_j\)) gets almost no push, and a negative that looks like the positive (a hard negative, large \(p_j\)) gets most of it. The total push equals the pull: \(\sum_j p_j = 1 - p^+\).

Worked example: the same scores at two temperatures

A query has \(s(q, d^+) = 0.8\) and two negatives with scores 0.5 and 0.3.

With \(\tau = 1\): the logits are 0.8, 0.5, 0.3; the softmax gives \(p = (0.426, 0.316, 0.258)\); the loss is \(-\log 0.426 = 0.853\). Push on the negatives: 0.316 and 0.258, so the closer negative gets 55% of the total push.

With \(\tau = 0.1\): the logits are 8, 5, 3; \(p = (0.946, 0.047, 0.006)\); the loss is \(0.055\). Gradients (60.4): pull \(-0.535\), push \(0.471\) and \(0.064\). Now the closer negative gets 88% of the push.

A lower temperature makes the loss focus on the hardest negatives. That is the main job of \(\tau\).

What the temperature does

Cosines live in \([-1, 1]\). With \(\tau = 1\), the largest possible gap between two logits is 2, which is too small for a confident softmax. Example: with 64 candidates, even if the positive had cosine 1 and all 63 negatives had cosine −1, \(p^+\) would be only \(e/(e + 63/e) = 0.105\), so the loss could never go below 2.254. Dividing by a small \(\tau\) removes this limit. On the other side, a very small \(\tau\) makes the softmax almost a hard max: only the single hardest negative gets a gradient, training becomes noisy, and false negatives (see below) hurt more.

Real values: SimCSE uses \(\tau = 0.05\); its ablation on the STS-B dev set (Spearman × 100, supervised model) gave 84.9 at \(\tau = 0.001\), 85.4 at 0.01, 86.2 at 0.05, 82.0 at 0.1 and 64.0 at 1. [Certain] E5 uses \(\tau = 0.01\), Contriever 0.05. [Certain] CLIP learns \(\tau\): it starts at 0.07 and the logit scale \(1/\tau\) is capped at 100, so \(\tau\) never goes below 0.01. [Certain] The widely used MultipleNegativesRankingLoss in the sentence-transformers library multiplies cosines by a scale of 20, which is \(\tau = 0.05\). [Likely]

positive: probability \(p^+\)negatives: probability \(p_j\)push on each negative \(p_j/\tau\) (scaled)
Figure 2. InfoNCE (60.3) for one query with one positive, one hard negative, and four easy negatives with fixed similarities 0.4, 0.3, 0.2 and 0.1. The top bars are softmax probabilities; the lower bars show how the push of formula (60.4) is shared among the negatives.
How to read Figure 2

Start at the default (\(\tau = 0.05\)): the hard negative gets 99.7% of all the push, and the four easy ones almost nothing. Raise \(\tau\) to 1: the push is spread almost evenly, and the loss stays high even when the positive is clearly the best. Move the hard negative above the positive: the loss jumps, which is exactly what should happen. Then imagine that this “hard negative” is in fact a relevant document that nobody labeled (a false negative): at low \(\tau\), the model is pushed hard in the wrong direction.

Why it is called InfoNCE: a bound on mutual information

The CPC paper showed that the loss gives a lower bound on the mutual information between the two views (how much knowing one tells you about the other): [Certain]

\[ I(q; d^+) \;\ge\; \log N - \mathcal{L}_N \tag{60.5} \]

Two useful facts follow. First, a model that knows nothing gives every candidate the same probability \(1/N\), so its loss is \(\log N\): for 64 candidates that is 4.159 nats. Use this as a sanity check at step 0 (our PyTorch code below prints 4.172 for random vectors). Second, the bound can never exceed \(\log N\): with few negatives, the loss cannot tell a good model from a great one. This is one reason why more negatives help.

Where the negatives come from

In-batch negatives

Encoding extra negatives costs forward and backward passes. The classic trick reuses the batch itself. Take \(B\) pairs \((q_i, d_i^+)\). Stack the query vectors into \(Q \in \R^{B \times k}\) and the passage vectors into \(P \in \R^{B \times k}\). One matrix multiplication gives all \(B^2\) scores, \(S = QP^\top/\tau\). Row \(i\) is a classification problem whose correct answer is column \(i\); the other \(B - 1\) passages in the row are the negatives (DPR, Karpukhin et al., 2020): [Certain]

\[ \mathcal{L}_{\text{batch}} = \frac{1}{B} \sum_{i=1}^{B} -\log \frac{\exp(S_{ii})}{\sum_{j=1}^{B} \exp(S_{ij})}, \qquad S = \frac{Q P^\top}{\tau} \tag{60.6} \]

This is just cross-entropy over the rows of \(S\), with labels \(0, 1, \dots, B - 1\). It is free: the negatives were encoded anyway as other examples’ positives. And it scales: the number of negatives per query grows with the batch size. The evidence is strong:

  • DPR, Natural Questions dev set, top-20 retrieval accuracy: 7 in-batch negatives 69.1%, 31 gave 70.8%, 127 gave 73.0%. [Certain]
  • E5 contrastive pre-training, average over 6 BEIR datasets (nDCG@10): batch 1K gave 45.8, 8K gave 50.2, 32K gave 51.6. [Certain]

Figure 3 runs formula (60.6) by hand on a batch of four pairs, one row at a time, so you can see every number go through the loop. The vectors are hand-made 6-number unit vectors that stand in for an encoder’s output (rounded to 2 decimals).

query \(q_i\)its passage \(d_i^+\)hard negative \(d_i^-\)
1capital of France?Paris is the capital of France.Berlin is the capital of Germany.
2how long to boil an eggBoil eggs for 9 to 12 minutes.Poach an egg for 3 minutes.
3sort a list in pythonUse sorted(xs) or xs.sort().Sort a dict by its values in python.
4shoes for a marathonLight road shoes for long races.Boots for long mountain hikes.
\(\tau\)
queries (rows)positive passages; the diagonal is the right answerhard negativesrow loss and current row
  1. \(\mathbf{q}_i, \mathbf{p}_j\) (and \(\mathbf{h}_j\)) \(\leftarrow\) unit vectors from the encoder
  2. for \(i = 1, \dots, B\): // one row = one classification problem
  3.   \(S_{ij} \leftarrow \mathbf{q}_i^\top \mathbf{c}_j / \tau\) for every candidate \(j\) // (60.6)
  4.   \(P_{ij} \leftarrow e^{S_{ij}} / \sum_k e^{S_{ik}}\) // softmax of the row
  5.   \(\ell_i \leftarrow -\log P_{ii}\); push on \(j\): \(P_{ij}/\tau\) // (60.3), (60.4)
  6. \(\mathcal{L} \leftarrow \frac{1}{B} \sum_i \ell_i\) // cross_entropy(S, arange(B))
Figure 3. One in-batch InfoNCE loss, computed in your browser row by row. Each cell shows the cosine \(s(q_i, c_j)\) and, after the softmax of its row, the probability \(P_{ij}\). The bars on the right are the row losses \(\ell_i\). The algorithm box highlights the line being run, and the panel next to it shows that line with the real numbers.
How to read Figure 3

Press Play, or Step through it. Line 3 fills a row with dot products (the panel writes out \(\mathbf{q}_i^\top \mathbf{p}_i\) term by term), line 4 turns the row into probabilities, line 5 reads off the loss. With the default \(\tau = 0.1\) and in-batch negatives only, every positive gets \(P_{ii} \approx 0.9995\) and the batch loss is 0.0005: the other topics are easy negatives, and this batch teaches almost nothing. Now tick “add the hard negatives” and play again. Row 2 changes the most: “boil an egg” has cosine 0.986 with its passage but 0.952 with “Poach an egg”, so \(P_{22}\) falls to 0.583, \(\ell_2 = 0.539\), and the hard negative takes 99.7% of the push. The batch loss rises to 0.154. Finally set \(\tau = 1\): every positive still has the highest score, but the softmax is flat and the loss stays at 0.785 (1.452 with hard negatives). That is the high floor from the temperature section.

The price is memory. For \(B = 32{,}768\) (the batch of E5 and of CLIP), the score matrix alone has \(32{,}768^2 \approx 1.07\) billion entries: 4.3 GB in FP32. In practice the batch is spread over many GPUs, each GPU gathers the vectors of all others (an all-gather, session 43), and computes its own rows. [Likely] When even the encoder activations do not fit, GradCache first computes all embeddings without storing activations, computes the loss and its gradient with respect to the embeddings, and then re-runs the encoder in small chunks to backpropagate. Memory then stays almost constant as \(B\) grows. [Certain]

Hard negatives

Most in-batch negatives are easy: a question about tax law against a passage about football. Formula (60.4) says easy negatives give almost no gradient. A hard negative is a document that looks relevant but is not: it shares words or topic with the query. Common sources:

  • BM25 negatives. The top results of a classic word-matching search that do not contain the answer. DPR adds one per question. [Certain]
  • Mined negatives. Search with the current (or a previous) embedding model and take high-ranked non-positives. This is what our PyTorch code does in its second stage. ANCE refreshes these negatives during training with an asynchronously updated index. [Likely]
  • Dataset structure. In NLI data, every premise has an “entailment” hypothesis (positive) and a “contradiction” hypothesis (hard negative). SimCSE and E5 use this. [Certain]

Hard negatives join the denominator. A common form (SimCSE, DPR) shares them across the batch: every query sees all \(B\) positives and all \(B\) hard negatives, so the score matrix becomes \(B \times 2B\): [Certain]

\[ \mathcal{L}_i = -\log \frac{e^{s(q_i, d_i^+)/\tau}}{\sum_{j=1}^{B} \Big( e^{s(q_i, d_j^+)/\tau} + e^{s(q_i, d_j^-)/\tau} \Big)} \tag{60.7} \]

The gains are large. In DPR, adding one BM25 negative per question (127 in-batch + 128 BM25 negatives) raised top-20 accuracy on the NQ dev set from 73.0% to 78.0%; adding two per question did not help further. [Certain] In supervised SimCSE, contradiction hard negatives raised STS-B dev from 84.9 to 86.2. [Certain] Figure 3 showed it on four pairs; Figure 4 lets you see a larger score matrix and what a hard negative column does to every row.

Figure 4. The in-batch score matrix of formula (60.6) for a toy batch of 16-dimensional embeddings, \(\tau = 0.05\). Rows are queries, columns are passages. The diagonal (outlined) holds the positives. Darker means more similar. Click any cell to select its row.
How to read Figure 4

Without hard negatives, the off-diagonal cells are random passages: their cosines are around 0 and the loss is tiny, so the batch teaches little. Turn on hard negatives: a second block appears with one near-duplicate per query. Its cells are much darker, and the loss grows many times. That larger loss is the useful signal. The symmetric loss adds the column direction (“for this passage, which query?”), which CLIP uses for images and captions.

False negatives: the danger of hard negatives

The harder you mine, the more often the “negative” is in fact relevant: MS MARCO, for example, labels only about one relevant passage per query, and many good passages are unlabeled. [Likely] Pushing those away teaches the model the wrong thing. Common defenses: skip mined candidates that rank very close to the positive, sample negatives from a middle band of ranks (for example ranks 30 to 100) instead of the very top, or let a cross-encoder filter out candidates that it judges relevant. RocketQA made this denoising step popular. [Likely] In-batch negatives have the same risk at a smaller scale: two queries in a batch can share an answer, so remove duplicates when you build batches.

Even more negatives: a queue and a momentum encoder

MoCo (He et al.) keeps a queue of passage vectors from previous batches and uses them as extra negatives. Old vectors were produced by older weights, so to keep them consistent, the passages are encoded by a slowly moving copy of the encoder, the momentum encoder, whose weights \(\theta_k\) follow the trained weights \(\theta_q\) by an exponential moving average: [Certain]

\[ \theta_k \leftarrow m\, \theta_k + (1 - m)\, \theta_q \tag{60.8} \]

No gradient flows through the queue. Contriever uses MoCo with a queue of 131,072 and \(m = 0.9995\). [Certain] E5 compared the options at the same compute and found plain in-batch negatives with a 32K batch best: 51.6 average, versus 45.5 for MoCo with 130K negatives. [Certain] So with enough GPUs, a big batch is the simplest and strongest choice; the queue is for when you cannot afford one.

Algorithm 1: one training step of a bi-encoder with in-batch and hard negatives
  1. input: batch of \(B\) pairs \((q_i, d_i^+)\), optional hard negatives \(d_i^-\), temperature \(\tau\), encoder \(f_\theta\)
  2. \(\mathbf{q}_i \leftarrow \operatorname{normalize}(\operatorname{pool}(f_\theta(\text{“query: ”} + q_i)))\) // prefix tells the shared encoder which side it is
  3. \(\mathbf{p}_i \leftarrow \operatorname{normalize}(\operatorname{pool}(f_\theta(\text{“passage: ”} + d_i^+)))\); same for \(d_i^-\) if present
  4. gather \(\mathbf{q}, \mathbf{p}\) (and hard negatives) from all GPUs // more negatives for free
  5. \(S \leftarrow Q\,[P;\, P^-]^\top / \tau\) // shape \(B \times B\) or \(B \times 2B\)
  6. \(\mathcal{L} \leftarrow \operatorname{CrossEntropy}(S, \text{labels} = [0, 1, \dots, B-1])\) // formulas (60.6), (60.7)
  7. optional: \(\mathcal{L} \leftarrow \tfrac12 \big(\mathcal{L} + \operatorname{CrossEntropy}(S_{:, :B}^\top, \text{labels})\big)\) // symmetric (60.14)
  8. backward, clip, AdamW step // as in session 4

Where the positives come from

Negatives are cheap; good positive pairs are the real data problem. Four families, from no labels to many labels:

Source of positive pairsExample methodDetails
The same text twice, with different dropoutUnsupervised SimCSEEncode a sentence twice; the two dropout masks give two slightly different vectors. 1 million Wikipedia sentences, batch 64, lr \(3\times10^{-5}\), \(\tau = 0.05\). [Certain]
Two random crops of one documentContrieverTwo spans cut independently from the same document, with 10% of tokens deleted. [Certain]
Naturally paired web textE5 (CCPairs)(title, passage), (question, upvoted answer), (post, comment), (title, abstract), citation pairs. About 1.3 billion pairs, filtered to about 270 million. [Certain]
Human-labeled pairsSentence-BERT, supervised SimCSE, DPR, E5 fine-tuningNLI entailment pairs, MS MARCO (Bing query, relevant passage), Natural Questions. [Certain]

The SimCSE dropout trick is surprisingly strong. Its ablation on STS-B dev: dropout \(p = 0.1\) gives 82.5; no dropout gives 71.1; and using the same dropout mask for both copies gives 43.6, because then the two views are identical and the task is trivial. [Certain] The noise is the augmentation. On seven STS tasks, unsupervised SimCSE with BERT-base reached an average Spearman of 76.25, and supervised SimCSE 81.57, versus 74.89 for SBERT-base. [Certain]

E5’s data filter is worth knowing for interviews. First train a model on all 1.3 billion noisy pairs. Then, for each pair, rank its passage against 1 million random passages; keep the pair only if its own passage is in the model’s top 2. [Certain] The idea: a network learns the clean, consistent pairs first, so pairs it cannot fit are probably noise. About 21% of the pairs survive.

The two-stage recipe

Most strong text embedding models since 2022 follow the same plan. [Likely] Stage 1, contrastive pre-training: hundreds of millions of weak pairs, in-batch negatives only, a very large batch. Stage 2, fine-tuning: a few hundred thousand to a few million labeled pairs, smaller batch, mined hard negatives, and often a cross-encoder teacher. E5’s fine-tuning loss mixes distillation from the teacher (the KL idea from session 59) with the contrastive loss: [Certain]

\[ \mathcal{L} = \KL\!\big(p_{\text{ce}} \,\Vert\, p_{\text{stu}}\big) + \alpha\, \mathcal{L}_{\text{cont}} \tag{60.9} \]
Read the symbols
\(p_{\text{ce}}\)
the cross-encoder teacher’s softmax over the candidate passages of one query (positive and hard negatives)
\(p_{\text{stu}}\)
the bi-encoder student’s softmax over the same candidates
\(\mathcal{L}_{\text{cont}}\)
the InfoNCE loss (60.6) with the hard labels
\(\alpha\)
a weight; E5 uses 0.2

The teacher gives graded scores (“this negative is almost relevant”) instead of a hard 0 or 1, which also softens the false-negative problem.

Algorithm 2: the two-stage recipe (in the style of E5)
  1. input: a pretrained encoder (BERT-like or a small LLM), web pairs, labeled pairs, a cross-encoder teacher
  2. stage 1: clean the web pairs: heuristic filters, then a consistency filter (keep a pair if its passage ranks in the top-\(k\))
  3.   train with in-batch InfoNCE (60.6), batch 32K, \(\tau = 0.01\), 20K steps, lr \(10^{-4}\) to \(3\times10^{-4}\) [Certain for E5]
  4. stage 2: for each labeled query, mine hard negatives with the stage-1 model; drop likely false negatives
  5.   score (query, candidates) with the teacher \(\rightarrow p_{\text{ce}}\)
  6.   train with (60.9): batch 256, 7 hard negatives per query, 3 epochs, lr \(10^{-5}\) to \(3\times10^{-5}\) [Certain for E5]
  7. evaluate on held-out in-domain data and on BEIR / MTEB; check every task type, not only the average

Details that matter in practice

  • Prefixes. E5 uses one shared encoder and breaks the symmetry with the prefixes “query: ” and “passage: ”. [Certain] Queries are short and passages long; the prefix lets one network treat them differently. Newer models use full task instructions (“Given a web search query, retrieve relevant passages”). [Likely] Forgetting the prefix at inference time is a classic silent bug.
  • Shared or separate encoders. DPR uses two separate BERTs (one for questions, one for passages) and a plain dot product. [Certain] Most later models share one encoder. [Likely]
  • Normalize. If you train with cosine, normalize at inference too, and use the same similarity in your index.
  • Symmetric tasks. For “are these two sentences similar” (no query/passage roles), use the same prefix on both sides and often the symmetric loss.
  • Matryoshka embeddings. Add the contrastive loss on the first 64, 128, 256, … dimensions too, so a prefix of the vector is itself a usable embedding. You can then store short vectors for a first pass and long ones for reranking. [Certain]
  • LLM-based embedders. Recent top models start from a decoder LLM and use the last token (or a learned attention pooling) as the embedding, trained with the same InfoNCE loss and synthetic data generated by LLMs. [Likely]

Now the runnable version. A toy world: 300 topics, each with 8 typical words. A passage has 3 topic words among 16 tokens; a query has 3 topic words among 6. The encoder is a bag of embeddings with mean pooling and a projection. We train with in-batch InfoNCE, then add one mined hard negative per query for the second half.

import math, torch, torch.nn.functional as F
torch.manual_seed(0)

# ---- toy data: 300 topics, each topic owns 8 words out of a 2,000-word vocabulary ----
V, T, D = 2000, 300, 64
topic_words = torch.randint(0, V, (T, 8))
def sample(topic, n):                      # n words: 3 from the topic, the rest random noise
    own = topic_words[topic, torch.randint(0, 8, (3,))]
    return torch.cat([own, torch.randint(0, V, (n - 3,))])
docs    = torch.stack([sample(t, 16) for t in range(T)])   # one passage per topic
q_test  = torch.stack([sample(t, 6) for t in range(T)])    # new queries, same topics

class BiEncoder(torch.nn.Module):          # shared encoder: token vectors -> mean pool -> project
    def __init__(self):
        super().__init__()
        self.emb = torch.nn.EmbeddingBag(V, D, mode="mean")
        self.proj = torch.nn.Linear(D, D)
    def forward(self, ids):
        return F.normalize(self.proj(self.emb(ids)), dim=-1)   # unit length: dot = cosine

def info_nce(q, p, tau=0.05, hard=None):   # formulas (60.6) and (60.7)
    cand = p if hard is None else torch.cat([p, hard])          # B or 2B candidates
    logits = q @ cand.T / tau                                   # (B, B) or (B, 2B)
    labels = torch.arange(q.shape[0])                           # positive of row i is column i
    return F.cross_entropy(logits, labels)

def symmetric_info_nce(a, b, tau=0.05):    # formula (60.14): rows and columns, as in CLIP
    logits = a @ b.T / tau
    labels = torch.arange(a.shape[0])
    return 0.5 * (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels))

@torch.no_grad()
def evaluate(model, queries):              # Recall@1 and MRR@10 over the whole corpus
    S = model(queries) @ model(docs).T                          # (300, 300) cosines
    rank = (S > S.diag()[:, None]).sum(1) + 1                   # rank of the true passage
    mrr = torch.where(rank <= 10, 1.0 / rank, torch.zeros_like(rank, dtype=torch.float)).mean()
    return (rank == 1).float().mean().item(), mrr.item()

model = BiEncoder()
print("before: R@1 %.3f  MRR@10 %.3f" % evaluate(model, q_test))
# sanity check: with random unit vectors the loss is close to log B
B = 64
u, v = F.normalize(torch.randn(B, D), dim=-1), F.normalize(torch.randn(B, D), dim=-1)
print("random vectors: loss %.3f, log B = %.3f" % (info_nce(u, v, tau=1.0).item(), math.log(B)))

opt = torch.optim.AdamW(model.parameters(), lr=1e-2, weight_decay=0.01)
for step in range(600):
    idx = torch.randperm(T)[:B]                                 # a batch of B (query, passage) pairs
    q_ids = torch.stack([sample(t, 6) for t in idx.tolist()])  # fresh training queries each step
    q, p = model(q_ids), model(docs[idx])
    hard = None
    if step >= 300:                                             # stage 2: mine one hard negative per query
        with torch.no_grad():
            S = q @ model(docs).T
            S[torch.arange(B), idx] = -1e4                      # never pick the true passage
            hard_ids = S.argmax(1)
        hard = model(docs[hard_ids])
    loss = info_nce(q, p, tau=0.05, hard=hard)
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 100 == 99:
        print(f"step {step+1}: loss {loss.item():.3f}  test R@1 %.3f  MRR@10 %.3f" % evaluate(model, q_test))

with torch.no_grad():                      # both directions on 64 test pairs: query->passage and back
    print("symmetric test loss %.3f" % symmetric_info_nce(model(q_test[:B]), model(docs[:B])).item())

On CPU with PyTorch 2.8 this prints: before training, Recall@1 0.013 and MRR@10 0.035 (300 passages, so chance is about 0.003); random vectors give a loss of 4.172, close to \(\log 64 = 4.159\); after 600 steps, Recall@1 0.930 and MRR@10 0.957. The loss goes up after step 300 (0.563 to 0.880 at step 400): the hard negatives make the task harder, which is the point. Running the same script with the hard-negative stage turned off ends at Recall@1 0.893. Exact digits can change with the PyTorch version; the pattern does not.

Watch a small retriever learn

Figure 5 runs the same recipe in your browser, on a world small enough to see every number. There are 12 topics in 6 domains, two topics per domain: France and Japan share the words “capital, city”; boiling and poaching share “egg, cook”; sorting and regex share “python, code”; and so on. Each passage has 5 words: its 2 domain words, its 2 topic words and “the”. A training query has 3 words, drawn fresh at every step: one domain word, one topic word and one filler (“how”, “what”, “best”, …). So the other topic of the same domain is the natural hard negative. The model has two towers, as in DPR: a query table \(E_q\) and a passage table \(E_p\), with 8 numbers per word (44 words). A text vector is the mean of its word vectors, normalized to length 1. Training is plain SGD with \(\eta = 0.2\) for 200 steps. The backward pass needs only three rules:

\[ \frac{\partial \mathcal{L}}{\partial S_{ij}} = \frac{P_{ij} - \delta_{ij}}{B}, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{q}_i} = \frac{1}{\tau} \sum_{j} \frac{\partial \mathcal{L}}{\partial S_{ij}}\, \mathbf{c}_j, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{v}} = \frac{\mathbf{g} - (\mathbf{g}^\top \mathbf{u})\, \mathbf{u}}{\lVert \mathbf{v} \rVert} \tag{60.10} \]
Read the symbols
\(P_{ij}\)
the softmax of row \(i\) of the score matrix \(S\) (60.6)
\(\delta_{ij}\)
1 on the diagonal (the positive) and 0 elsewhere; so the positive is pulled with \((P_{ii} - 1)/B \lt 0\) and every negative is pushed with \(P_{ij}/B \gt 0\), as in (60.4)
\(\mathbf{c}_j\)
the unit vector of candidate \(j\) (a positive or a hard negative); the passage side gets the mirror rule \(\tfrac{1}{\tau} \sum_i \tfrac{\partial \mathcal{L}}{\partial S_{ij}} \mathbf{q}_i\)
\(\mathbf{v}, \mathbf{u}, \mathbf{g}\)
\(\mathbf{v}\) is the mean of the word vectors, \(\mathbf{u} = \mathbf{v}/\lVert \mathbf{v} \rVert\) the normalized vector, \(\mathbf{g}\) the gradient with respect to \(\mathbf{u}\); normalization removes the part of \(\mathbf{g}\) along \(\mathbf{u}\). Each of the \(n\) words of the text then receives \(\tfrac{1}{n}\, \partial \mathcal{L}/\partial \mathbf{v}\)
\(\tau\) \(B\)
test query: top-1 is correctwrong, but the other topic of the same domainwrong, another domaindashed: the same run with the other hard-negative setting
  1. pick \(B\) topics; one fresh query each, and its passage \(d_i^+\)
  2. \(\mathbf{q}_i \leftarrow \operatorname{normalize}(\text{mean } E_q[\text{words}])\), same for passages with \(E_p\)
  3. if hard: \(h_i \leftarrow \arg\max_{k \ne i} \mathbf{q}_i^\top \mathbf{p}_k\) over all 12 passages // mined with the current model
  4. \(S \leftarrow Q\,[P; H]^\top / \tau\), \(\mathcal{L} \leftarrow \operatorname{CE}(S, \text{diagonal})\) // (60.6), (60.7)
  5. \(\partial \mathcal{L}/\partial S_{ij} = (P_{ij} - \delta_{ij})/B\), back through normalize and mean // (60.10)
  6. \(E_q, E_p \leftarrow E - \eta\, \nabla E\) // plain SGD, \(\eta = 0.2\)
  7. Recall@1 of 24 fixed test queries against all 12 passages // (60.11)
Figure 5. A two-tower retriever trained in your browser with in-batch InfoNCE, optionally with one mined hard negative per query. Top: the current batch (rows are queries, columns are candidate passages; the outlined diagonal holds the positives), first as softmax probabilities, then as the gradient of formula (60.10). Middle: Recall@1 on 24 fixed test queries (two per topic), and the batch loss. Bottom: the top-1 passage of every test query. Everything is computed live from a seeded random start; the same initial weights and the same batches are used with and without hard negatives, so the two curves are a fair comparison.
How to read Figure 5

Press Step a few times: each training step runs the seven lines in order, and the panel shows the real numbers of the current line. In line 3, look at which passage is mined. In the first 20 steps it is the other topic of the same domain (France for a Japan query) only 41% of the time, because the model still knows little; after step 20 it is that sibling about 80% of the time. Then pick “fast” and press Play. With the defaults (\(\tau = 0.1\), \(B = 4\), no hard negatives), Recall@1 starts at 0.042 (1 of 24 queries) and first reaches 1 at step 149; the last wrong answers are orange dots, the sibling topic. Tick “mine one hard negative per query”: the same run reaches 1 at step 64, and the dashed curve is the run without them. The loss is higher with hard negatives (5.714 instead of 3.794 at step 1), because the task is harder; a higher loss here is a sign of a more useful batch. Try \(\tau = 1\): learning is much slower (Recall@1 0.667 at step 100 without hard negatives). Try \(B = 2\) and \(B = 6\): more in-batch negatives also speed up learning. Why is the step-1 loss above \(\log 4 = 1.386\)? That value is for a model that gives every candidate the same score. Random 8-number unit vectors have cosines spread by about \(1/\sqrt{8} = 0.35\), which at \(\tau = 0.1\) are logits spread by about 3.5, so a random model is often confidently wrong.

The numbers in Figure 5 are a small copy of the PyTorch result above: with hard negatives the loss goes up and retrieval gets better faster. With \(\tau = 0.1\) and \(B = 4\), Recall@1 after 20 steps is 0.458 without hard negatives and 0.667 with them; after 50 steps it is 0.875 and 0.917. A copy of the simulation in Python, with the same random generator, gives the same numbers to 4 decimals. The hard negatives in this world are never false negatives, because every topic has exactly one passage. In real data, mining that aggressively would also pick unlabeled positives, which is why the false-negative defenses above matter. (Recall@1, defined formally in (60.11) below, is the fraction of test queries whose correct passage is ranked first.)

How to evaluate an embedding model

The training loss is not the goal; retrieval quality is. Embed a corpus, run a set of test queries with known relevant documents, and score each ranked list. Three metrics cover most cases. For one query with \(R\) relevant documents in the corpus, a ranked list, and \(\operatorname{rank}_1\) the position of the first relevant result:

\[ \text{Recall@}k = \frac{\#\{\text{relevant in the top } k\}}{R}, \qquad \text{MRR} = \frac{1}{|Q|}\sum_{q \in Q} \frac{1}{\operatorname{rank}_1(q)} \tag{60.11} \]

Recall@k asks “did we get them into the top \(k\)?”. It is the right metric for the first stage of RAG or a reranking pipeline: whatever is missed here can never be found later. MRR (mean reciprocal rank) asks “how high is the first good result?”; it suits question answering with one answer. nDCG@k (normalized discounted cumulative gain) rewards relevant results more when they are higher, and supports graded relevance:

\[ \text{DCG@}k = \sum_{i=1}^{k} \frac{2^{\text{rel}_i} - 1}{\log_2(i + 1)}, \qquad \text{nDCG@}k = \frac{\text{DCG@}k}{\text{IDCG@}k} \tag{60.12} \]
Read the symbols
\(\text{rel}_i\)
the relevance grade of the result at position \(i\) (0 or 1 for binary labels)
\(\log_2(i+1)\)
the discount: position 1 counts fully, position 3 counts half, position 7 counts one third
\(\text{IDCG@}k\)
the DCG of the best possible ordering, so a perfect list scores 1

Click a result to mark it relevant or not relevant.

Figure 6. Retrieval metrics for one query and its top-10 results, formulas (60.11) and (60.12). By default, results 2, 4 and 9 are relevant and one more relevant document exists that was not retrieved.
How to read Figure 6

At the default, Recall@10 is 0.75 (3 of 4 found), MRR is 0.5 (first hit at rank 2), and nDCG@10 is 0.532. Move the hit at rank 9 to rank 1 (click 9 off, 1 on): Recall@10 does not change, but MRR jumps to 1 and nDCG rises. Raise \(R\): recall and nDCG drop, MRR does not care. This is why you must report several metrics, and why missing labels (unknown relevant documents) make every number look worse than it is.

Worked example: the default list of Figure 6

Relevant results at ranks 2, 4 and 9, and \(R = 4\). Recall@5 \(= 2/4 = 0.5\); Recall@10 \(= 3/4 = 0.75\); MRR \(= 1/2\).

DCG@10 \(= 1/\log_2 3 + 1/\log_2 5 + 1/\log_2 10 = 0.631 + 0.431 + 0.301 = 1.363\). The ideal list puts all 4 relevant documents first: IDCG@10 \(= 1 + 0.631 + 0.5 + 0.431 = 2.562\). So nDCG@10 \(= 1.363/2.562 = 0.532\).

Benchmarks

  • BEIR: 18 public retrieval datasets from different domains (biomedical, finance, argument, fact checking, …), scored with nDCG@10, used zero-shot (the model never trained on them). Its main lesson: BM25 is a strong baseline out of domain. [Certain] E5 was the first model reported to beat BM25 on BEIR without labeled data: an average of 42.9 for E5-PT-base versus 41.7 for BM25 over 15 datasets. [Certain]
  • MTEB: 8 task types (retrieval, reranking, clustering, classification, pair classification, STS, summarization, bitext mining), 58 datasets, 112 languages. Its finding: no single model wins every task. [Certain]
  • STS (semantic textual similarity): Spearman correlation between the model’s cosines and human similarity scores. Good for paraphrase-style similarity, a weak predictor of retrieval quality. [Likely]

Public leaderboards are a starting point, not an answer. Some top models have trained on the training splits of benchmark datasets, and your domain (legal text, sports commentary, code) may behave very differently. [Likely] Always build a small test set from your own queries, a few hundred labeled examples, and compare models on it (session 150).

Looking inside the space: alignment and uniformity

Wang and Isola showed that the contrastive loss, with many negatives, optimizes two properties of normalized embeddings: positives should be close (alignment), and all embeddings should spread evenly over the sphere (uniformity). They are cheap to measure on a sample: [Certain]

\[ \mathcal{L}_{\text{align}} = \E_{(x, y) \sim \text{pos}} \lVert f(x) - f(y) \rVert^2, \qquad \mathcal{L}_{\text{uniform}} = \log \E_{x, y \sim \text{data}} \, e^{-2 \lVert f(x) - f(y) \rVert^2} \tag{60.13} \]

Both should be low. A collapsed model has perfect alignment and terrible uniformity; raw BERT (the narrow cone) is similar. Figure 1 prints the uniformity of its 16 points: from −0.260 at the start to −1.867 when the pairs spread evenly. SimCSE used these two numbers to explain why its dropout trick works: the training keeps alignment and improves uniformity. [Certain]

The same idea as CLIP: our bridge to vision

Replace the query encoder by an image encoder (a ViT) and the passage encoder by a text encoder, take image-caption pairs from the web, and train with in-batch InfoNCE. That is CLIP (Radford et al., 2021): 400 million (image, text) pairs, a batch of 32,768, a linear projection from each encoder into the shared space, and a learned temperature. [Certain] Because both directions matter (find the caption for an image, find the image for a caption), CLIP uses the symmetric loss: cross-entropy over the rows and over the columns of the same score matrix: [Certain]

\[ \mathcal{L}_{\text{sym}} = \frac{1}{2B} \sum_{i=1}^{B} \left( -\log \frac{e^{S_{ii}}}{\sum_{j} e^{S_{ij}}} \;-\; \log \frac{e^{S_{ii}}}{\sum_{j} e^{S_{ji}}} \right), \qquad S = \frac{I\,T^\top}{\tau} \tag{60.14} \]
Read the symbols
\(I, T\)
the \(B \times k\) matrices of normalized image and text embeddings
first term
image \(i\) must pick its caption among all \(B\) captions (a row)
second term
caption \(i\) must pick its image among all \(B\) images (a column)

Everything in this session carries over. Batch size matters for the same reason (more negatives). Temperature matters (CLIP learns it, with a cap). Hard negatives and false negatives exist (two web images can both match “a dog on a beach”). And evaluation is the same Recall@k, now for text-to-image and image-to-text retrieval. SigLIP replaces the softmax by an independent sigmoid on every cell of the matrix, which removes the need to normalize over the whole batch (session 167). [Certain] The full story is in session 166.

Cross-encoder (reranker) [CLS] query [SEP] doc one transformer query and doc attend to each other score 1 forward pass per pair: accurate, slow; top-k only Bi-encoder (text retriever) query passage encoder+ pool encoder+ pool shared weights qd q · d docs encoded once, offline; search = nearest neighbors CLIP (image and text) img “a dog on sand” imageencoder (ViT) textencoder B × Bscores symmetric InfoNCE (60.14)
Figure 7. Three model shapes. A cross-encoder reads the pair together and is used to rerank. A bi-encoder encodes each side alone, so the corpus can be indexed once; this is what this session trains. CLIP is a bi-encoder with two different towers and the same in-batch contrastive loss; the diagonal of its \(B \times B\) score matrix holds the true pairs.

For video, the step is small again. A video-text retriever embeds a clip (for example by pooling the CLIP embeddings of sampled frames, or with a video encoder) and trains with the same symmetric InfoNCE on (clip, caption) pairs (session 208). [Likely] A video fingerprinting model uses two augmented copies of the same clip (crop, re-encode, overlay text) as the positive pair, exactly like SimCSE’s two dropout copies, and other clips as negatives (session 209). [Likely]

Where this is used

ApplicationPositive pairsNegatives and notes
First-stage retrieval for RAG and search(query, relevant passage): MS MARCO, NQ, mined web pairsIn-batch plus mined hard negatives; measure Recall@k on your own queries (session 152)
Semantic similarity, deduplication, clusteringParaphrases, NLI entailment, dropout viewsSymmetric loss; tune a cosine threshold on labeled pairs
RerankingSame labeled pairsCross-encoder trained with a classification or listwise loss; often the teacher of the bi-encoder (60.9)
Image-text search and zero-shot classification(image, caption) from the webCLIP / SigLIP; huge batches; class names written as text prompts
Text-to-video retrieval and key-moment search(clip, caption), (moment, commentary line)Neighboring clips from the same video are hard negatives, and often false negatives [Likely]
Video fingerprinting and near-duplicate detectionTwo augmented copies of one clipOther clips as negatives; evaluate with precision at a fixed false-positive rate
Video-RAG(question, clip or transcript chunk)Multimodal bi-encoder over frames and speech text (session 210)
Recommendation (two-tower models)(user history, item clicked)In-batch negatives over-sample popular items; a log-frequency correction is common [Likely]

Common mistakes

  • Temperature 1 with cosine similarity. The logits then live in \([-1, 1]\), the softmax can never become confident, and training stalls (with 64 candidates the loss cannot go below 2.254). Use \(\tau\) around 0.01 to 0.05, or learn it.
  • Forgetting to normalize, or normalizing only one side. Train and serve with the same similarity: cosine (normalized dot product) or raw dot product, never a mix.
  • Duplicates in a batch. If two rows share the same positive, one of them is labeled a negative of itself. Deduplicate, or mask those cells.
  • Mining the very top results as negatives. Many of them are unlabeled positives. Skip the top few, sample from a band, or filter with a cross-encoder.
  • Missing or wrong prefixes and instructions. A model trained with “query: ” / “passage: ” silently loses quality without them. Read the model card.
  • Batch split across GPUs without gathering. Each GPU then sees only its local \(B/n\) negatives. Gather the embeddings (with gradients) across GPUs.
  • Choosing a model by the leaderboard average. The average mixes 8 task types; check the retrieval column, and test on your own data.
  • Changing the embedding model without re-indexing. Vectors from two different models (or two checkpoints) are not comparable. Re-embed the whole corpus.

Interview questions

Write the InfoNCE loss and explain each part.basic
For a query \(q\), positive \(d^+\) and negatives \(d_j^-\): \(\mathcal{L} = -\log \frac{e^{s(q,d^+)/\tau}}{e^{s(q,d^+)/\tau} + \sum_j e^{s(q,d_j^-)/\tau}}\). It is a softmax cross-entropy where the “class” is the index of the positive among the candidates. \(s\) is usually the cosine; \(\tau\) sharpens the softmax. Minimizing it raises the positive’s score relative to all negatives at once. With in-batch negatives it is simply cross_entropy(Q @ P.T / tau, arange(B)).
Why can’t we just maximize the similarity of positive pairs?basic
Because the trivial solution maps everything to one vector: every similarity becomes 1, the positive loss is perfect, and the embeddings carry no information (collapse). Contrastive losses add negatives that must score lower, which forces the space to spread out (uniformity). Non-contrastive methods (BYOL, DINO) avoid collapse with other tricks such as a momentum teacher, a predictor, or centering, but text retrievers almost always use negatives.
What does the temperature do? What happens if it is too high or too low?core
It scales the logits. The gradient on each negative is \(p_j/\tau\), so a lower \(\tau\) concentrates the push on the hardest negatives. Too high (for example 1 with cosine): the logit range is only \([-1, 1]\), the softmax stays flat, the loss has a high floor (2.254 for 64 candidates) and the model learns slowly. Too low: almost a hard max, so only the top negative gets gradient; training is noisier, sensitive to false negatives, and can be unstable. Typical: 0.01 to 0.05; SimCSE found 0.05 best on STS-B dev; CLIP learns it from 0.07 with a cap of 100 on \(1/\tau\).
Why does batch size matter so much in contrastive learning, and how do you get large batches on limited memory?core
With in-batch negatives, each query sees \(B - 1\) negatives, so a bigger batch gives more and harder negatives (the hardest of more random samples is harder), and the bound \(\log N - \mathcal{L}\) can be larger. E5 measured 45.8 → 50.2 → 51.6 average nDCG@10 for batch 1K → 8K → 32K. Ways to scale: gather embeddings across GPUs, GradCache (embed without activations, compute the loss gradient on embeddings, then backprop in chunks), a MoCo queue with a momentum encoder, mixed precision, and adding hard negatives so a smaller batch carries more signal. Note: gradient accumulation does not enlarge the set of negatives, because each micro-batch builds its own softmax.
What is a hard negative, how do you mine one, and what is the risk?core
A negative that scores high but is not relevant: same topic, shared words. Sources: BM25 top results without the answer, top results of the current embedding model, contradiction hypotheses in NLI. They carry most of the gradient, so they add a lot (DPR: 73.0% → 78.0% top-20 on NQ dev with one BM25 negative per query). The risk is false negatives: unlabeled relevant documents that the model is told to push away. Mitigate by skipping the top ranks, sampling from a band, cross-encoder filtering, or distilling soft scores from a cross-encoder teacher.
Bi-encoder vs cross-encoder: when do you use each?basic
A bi-encoder embeds query and document separately; documents are embedded once and searched with ANN in milliseconds, so it handles millions to billions of items. A cross-encoder reads both together and lets them attend to each other, so it is more accurate but needs one forward pass per pair. Production systems use both: bi-encoder (often plus BM25) to get the top 100 to 1000, then a cross-encoder to rerank the top few dozen. The cross-encoder is also a good teacher for training the bi-encoder.
How would you evaluate a new embedding model for your company’s search?core
Build a labeled test set from real queries (a few hundred, with relevant documents marked, ideally graded), embed your real corpus, and measure Recall@k for the k your reranker or RAG prompt uses, plus nDCG@10 and MRR. Compare against BM25 and the current model. Use MTEB / BEIR only to shortlist; look at the retrieval column, not the average, and beware of training-set contamination. Also check latency, vector size (Matryoshka truncation), and failure slices (long queries, rare entities, other languages). Missing labels lower every metric, so inspect some “false positives” by hand.
Explain how SimCSE learns sentence embeddings without labels. Why does it work?advanced
It feeds the same sentence twice through the encoder with dropout on; the two different dropout masks give two slightly different vectors, which form the positive pair. Other sentences in the batch are negatives; \(\tau = 0.05\). Dropout is a minimal augmentation that keeps the meaning exactly; discrete augmentations (deleting words) changed meaning and did worse. With the same mask on both copies, the task is trivial and STS-B dev falls to 43.6 (vs 82.5). In terms of alignment and uniformity, the loss keeps positives aligned while spreading out the anisotropic BERT space.
How is CLIP’s loss related to a text retriever’s loss? What changes?core
It is the same in-batch InfoNCE on a \(B \times B\) score matrix, with two different encoders (image ViT and text transformer) instead of one shared text encoder, and a symmetric loss (rows: image → text; columns: text → image). CLIP uses 400M web pairs, batch 32,768, and a learned temperature initialized at 0.07 with \(1/\tau \le 100\). The same issues apply: batch size, temperature, false negatives among web captions, and Recall@k evaluation. SigLIP replaces the softmax with per-cell sigmoids so the loss does not need the full row normalization.
What does the bound \(I \ge \log N - \mathcal{L}\) tell you in practice?advanced
Two things. A random model has loss \(\log N\) (all candidates equal), so the initial loss is a quick bug check: for 64 candidates expect about 4.16. And the estimated information is capped at \(\log N\): with few negatives the loss saturates near 0 and stops distinguishing good from great models, so more negatives give a tighter, more useful objective. It is a bound, not an exact value; low loss does not prove high mutual information, and the bound is loose for strong dependence.
Your retriever has great training loss but poor Recall@10 on production queries. What do you check?advanced
(1) Train/serve mismatch: prefixes or instructions, normalization, pooling, max length truncating long documents, a different tokenizer. (2) The index: built with an older checkpoint, or ANN settings with low recall. (3) Data: training queries do not look like production queries (length, language, domain); add in-domain pairs or synthetic queries generated by an LLM. (4) The loss was too easy: only easy in-batch negatives, so the model never learned fine distinctions; add mined hard negatives with false-negative filtering. (5) Labels: check whether “misses” are actually relevant documents without labels.

Exercises

  1. A model is trained with in-batch negatives, batch 1,024. How many negatives does each query see? What loss do you expect at step 0?
  2. Compute the InfoNCE loss for \(s(q, d^+) = 0.9\), two negatives with \(s = 0.6\), and \(\tau = 0.1\).
  3. How much memory does the FP32 score matrix need for one global batch of 65,536 pairs, if it were computed on one GPU?
  4. A ranked list of 5 results has relevance (0, 0, 1, 0, 1), and the corpus has exactly 2 relevant documents. Compute Recall@5, MRR and nDCG@5.
  5. CLIP caps \(1/\tau\) at 100 and starts at \(\tau = 0.07\). What are the smallest \(\tau\) and the starting logit scale?
  6. In Figure 4, turn on hard negatives with \(B = 8\). Which row has the largest loss, and why? Then explain in one sentence why the symmetric loss does not use the hard-negative columns in the column direction.
Answerscheck
1. \(B - 1 = 1{,}023\) negatives; the loss of a random model is \(\log 1024 = 6.93\) nats. 2. Logits 9, 6, 6: \(p^+ = e^9/(e^9 + 2e^6) = 0.909\), loss \(= -\log 0.909 = 0.095\). 3. \(65{,}536^2 \times 4\) bytes \(= 17.2\) GB, which is why the rows are split across GPUs. 4. Recall@5 = 2/2 = 1; MRR = 1/3; DCG@5 \(= 1/\log_2 4 + 1/\log_2 6 = 0.5 + 0.387 = 0.887\); IDCG \(= 1 + 1/\log_2 3 = 1.631\); nDCG@5 = 0.544. 5. Smallest \(\tau = 1/100 = 0.01\); the starting scale is \(1/0.07 = 14.3\). 6. Row q5, with loss 0.297: its positive has cosine 0.898 and its hard negative 0.845, the smallest gap in the batch (0.053), so the hard negative takes the largest share of the softmax. The loss depends on the gap divided by \(\tau\), not on the raw scores. In the column direction the question is “which query matches this passage?”, and the hard negatives are passages, not queries, so they have no column of their own to classify.

Key takeaways

  • An embedding model is a bi-encoder: encode each text alone, normalize, compare with a dot product (60.1). Search becomes nearest neighbors; a cross-encoder reranks the top results.
  • Contrastive learning pulls positives together and pushes negatives apart; without negatives the space collapses. InfoNCE (60.3) is a softmax over one positive and many negatives; its gradient (60.4) pushes each negative in proportion to its probability.
  • The temperature sets the focus: low \(\tau\) (0.01 to 0.05) concentrates on hard negatives; \(\tau = 1\) with cosine leaves a high loss floor.
  • In-batch negatives (60.6) are free and grow with the batch (E5: batch 1K → 32K raised BEIR scores from 45.8 to 51.6); hard negatives (60.7) add strong signal but bring false negatives.
  • Positives come from dropout (SimCSE), crops (Contriever), web pairs (E5) or labels (NLI, MS MARCO); the standard recipe is weak-pair pre-training, then hard-negative fine-tuning with a cross-encoder teacher (60.9).
  • Evaluate with Recall@k, MRR and nDCG@10 (60.11, 60.12) on your own data; BEIR and MTEB are for shortlisting.
  • CLIP is the same recipe with an image tower and the symmetric loss (60.14): the bridge from text retrieval to vision and video.

Key formulas

\[ \mathcal{L} = -\log \frac{e^{s(q,d^+)/\tau}}{\sum_{d \in \{d^+\} \cup \mathcal{N}} e^{s(q,d)/\tau}}, \qquad \frac{\partial \mathcal{L}}{\partial s(q, d_j^-)} = \frac{p_j}{\tau}, \qquad I \ge \log N - \mathcal{L} \]
\[ \mathcal{L}_{\text{batch}} = \operatorname{CE}\!\left(\frac{QP^\top}{\tau}, [0, \dots, B{-}1]\right), \qquad \text{nDCG@}k = \frac{\sum_{i \le k} (2^{\text{rel}_i} - 1)/\log_2(i+1)}{\text{IDCG@}k} \]

References for this session

  1. van den Oord et al. (2018). Representation Learning with Contrastive Predictive Coding
  2. Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
  3. Gao et al. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings
  4. Wang et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training
  5. Izacard et al. (2021). Unsupervised Dense Information Retrieval with Contrastive Learning
  6. van den Oord et al. (2018). Representation Learning with Contrastive Predictive Coding
  7. Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
  8. Gao et al. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings
  9. Wang et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training
  10. Izacard et al. (2021). Unsupervised Dense Information Retrieval with Contrastive Learning
  11. Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering
  12. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  13. He et al. (2019). Momentum Contrast for Unsupervised Visual Representation Learning
  14. Gao et al. (2021). Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
  15. Wang & Isola (2020). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
  16. Thakur et al. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
  17. Muennighoff et al. (2022). MTEB: Massive Text Embedding Benchmark
  18. Kusupati et al. (2022). Matryoshka Representation Learning
  19. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training