Text embeddings and semantic search
Semantic search finds texts by meaning, not by shared words: embed the query, find the nearest document vectors. This session covers how to do it well: bi-encoders for speed and cross-encoders for precision, the pooling and normalization a model expects, Matryoshka vectors that can be cut short, and how to measure search with recall@k, MRR and nDCG on a public benchmark and on your own data.
What you will learn
- Bi-encoder, cross-encoder and late interaction (ColBERT): how each scores a query and a document, and what each costs per query and in index memory
- The two-stage design: retrieve with a bi-encoder, rerank with a cross-encoder, and why reranking can never raise recall beyond the first stage
- Pooling (mean with mask, CLS, last token), normalization, cosine vs dot product vs distance, and query prefixes, with the measured cost of getting each one wrong
- Matryoshka embeddings: the nested loss, truncation with renormalization, and the memory formula, with real truncation results for an MRL and a non-MRL model
- Recall@k, MRR and nDCG@k with graded labels, worked by hand, and how to read their standard errors
- What MTEB measures, why a leaderboard rank is not enough, and how to build your own retrieval test set
Why this matters
A user types “my laptop gets hot when I play games”. The best help page is titled “Fan noise and thermal throttling under load”. The two texts share no important word, so keyword search misses it. Semantic search finds it, because it compares meanings, not words.
The tool is a text embedding: a model turns any text into one vector of a few hundred numbers, so that texts with similar meaning get vectors that point in similar directions. Search then becomes geometry: embed the query, find the document vectors closest to it. This one idea runs inside retrieval-augmented generation (RAG), product and code search, duplicate detection, recommendation, clustering, and the memory of LLM agents. It is also how video search works: CLIP-style models put frames, clips and sentences into the same vector space.
Session 60 showed how such models are trained (contrastive loss, negatives). This session is about using them well. Four practical questions decide whether a search system is good or useless:
- Which architecture: a bi-encoder (fast, one vector per text) or a cross-encoder (slow, reads query and document together)? Usually both, in two stages.
- How do we turn token vectors into one vector (pooling), and how do we compare vectors (normalization, cosine or dot product)? A wrong choice here silently loses half of the quality.
- How many numbers do we store? Matryoshka embeddings let us cut a vector to a fraction of its length.
- How do we know it works? Recall@k, MRR, nDCG, measured on a public benchmark like MTEB and, more important, on our own data.
All real numbers in this session come from one run we did on SciFact, a public scientific claim-checking dataset from the BEIR collection (5,183 paper abstracts, 300 test queries, on average 1.13 relevant abstracts per query). It is small enough to run on a laptop, and it is one of the MTEB retrieval tasks. [Certain: our run]
Semantic search in one picture
We have a corpus of \(N\) documents (passages, help pages, product descriptions). Offline, we embed each one and store the vectors in an index. Online, a query arrives; we embed it with the same model and return the \(k\) documents with the highest similarity score. That is all a dense retriever does.
Figure 1 runs this whole loop on a toy you can check by hand. Each word has a fixed vector of 5 numbers, one per “meaning axis” (heat, computer, games, food, other). A toy encoder with one attention layer and no learned weights mixes the words of a text: \(\mathbf{h}_t = \mathbf{x}_t + \sum_j \alpha_{tj} \mathbf{x}_j\), with \(\alpha_{tj} = \softmax_j(\mathbf{x}_t^\top \mathbf{x}_j / \sqrt{5})\) over the real tokens. Then pooling turns the token vectors into one vector, we scale it to length 1, and the score is the cosine. The pooling choices and the padding trap are explained in the pooling section; you can already switch them here and watch what they do.
- for each document \(d_i\) in the corpus (offline):
- \(H \leftarrow\) encoder(tokens of \(d_i\), padded to batch length \(T\)); mask \(m\)
- \(\mathbf{e} \leftarrow\) pool(\(H, m\)) (mean with mask, mean without mask, or CLS)
- \(\hat{\mathbf{d}}_i \leftarrow \mathbf{e} / \lVert \mathbf{e} \rVert\); add \(\hat{\mathbf{d}}_i\) to the index
- when a query \(q\) arrives (online):
- \(\hat{\mathbf{q}} \leftarrow\) normalize(pool(encoder(\(q\)))) (same model, same pooling)
- for each \(i\): \(s_i \leftarrow \hat{\mathbf{q}}^\top \hat{\mathbf{d}}_i\) (the cosine, 150.1)
- return the \(k = 3\) documents with the largest \(s_i\)
How to read Figure 1
Press Play. Steps 2 to 13 build the index offline: each document is encoded (padded to 6 positions), pooled and scaled to length 1. Step 14 encodes the query. Steps 15 to 20 score one document each, and you can watch the ranking change. With the default query and mean pooling, the final order is d5 (0.923), d1 (0.894), d6 (0.834): the two answers about heat come first, and d1 shares no word with the query. d4 (“pasta in a hot oven”) shares the word “hot” but comes last (0.685). Now pick “laptop battery” and “Mean, no mask (bug)”: the exact answer d2 falls from first (0.994) to third (0.827), only because it is short and got 3 padding rows. Pick CLS: with the default query all six scores sit between 0.935 and 0.985, because the [CLS] row of a model that was not built for CLS pooling looks almost the same for every text.
Two words we will use all the time. Symmetric search compares texts of the same kind (a question with other questions, to find duplicates). Asymmetric search compares a short query with a long document that answers it. Most real search is asymmetric, and some models need to be told which side is which (the query prefixes below).
Embedding search does not replace keyword search such as BM25. Keywords are better for rare names, product codes and exact phrases; embeddings are better for paraphrases and questions. Production systems often run both and merge the lists (hybrid search, session 152). How to search millions of vectors fast is session 151.
Bi-encoder vs cross-encoder
There are two ways to let a transformer judge whether document \(d\) answers query \(q\).
The bi-encoder: encode separately, compare with a dot product
A bi-encoder runs the encoder \(f\) once on the query and once on the document, independently, and compares the two vectors:
- Read the symbols
- \(f(\cdot)\)
- the embedding model: text in, one vector in \(\R^{d}\) out (\(d\) = 384 for all-MiniLM-L6-v2, 768 for BERT-base-size models)
- \(f(q)^\top f(d)\)
- the dot product of the two vectors: multiply coordinate by coordinate and add
- \(\lVert \cdot \rVert\)
- the length (L2 norm) of a vector; dividing by both lengths gives the cosine of the angle between them, a number in \([-1, 1]\)
The key property: \(f(d)\) does not depend on the query. We compute all document vectors once, offline. At query time we run the model once (on the query) and do \(N\) cheap dot products, or use an approximate index that looks at only a small part of them. DPR (Karpukhin et al., 2020) and Sentence-BERT (Reimers and Gurevych, 2019) are the classic bi-encoders; DPR used two separate encoders for questions and passages, SBERT one shared encoder. [Certain]
The cross-encoder: read both texts together
A cross-encoder puts query and document into one input and lets full attention mix them. A small head reads the [CLS] vector and outputs one relevance score:
- Read the symbols
- \(\mathbf{h}_{\texttt{[CLS]}}(\cdot)\)
- the final hidden state at the [CLS] position after the encoder has read the pair
- \(\mathbf{w}, b\)
- a linear head trained so that relevant pairs get high scores (often with a sigmoid and binary cross-entropy)
Every query word can attend to every document word, so the model can check details: does the passage say “not”, does it talk about the same year, does it answer the question or only mention its words? This makes cross-encoders clearly more accurate rankers. But nothing can be computed before the query arrives: to rank \(N\) documents we need \(N\) full forward passes, per query.
Late interaction: ColBERT, in between
ColBERT (Khattab and Zaharia, 2020) keeps one vector per token instead of one per text. Documents are still encoded offline. The score lets each query token find its best match in the document (MaxSim) and adds these up:
- Read the symbols
- \(\mathbf{q}_i, \mathbf{d}_j\)
- the normalized output vectors of query token \(i\) and document token \(j\) (128 numbers each in ColBERT)
- \(\max_j\)
- for each query token, only the single best-matching document token counts
This keeps some word-level matching of a cross-encoder at close to bi-encoder cost. The paper reports quality competitive with BERT rerankers, “two orders-of-magnitude faster” and with “four orders-of-magnitude fewer FLOPs per query”: 61 ms and 7 billion FLOPs against 10,700 ms and 97 trillion FLOPs for BERT-base reranking 1,000 passages. [Certain] The price is storage: one vector per token. Its MS MARCO index (8.8 million passages, 128 dimensions, 2 bytes each) takes 154 GiB, against about 25 GiB for one 768-dim fp32 vector per passage. [Certain for 154 GiB; the 25 GiB is our arithmetic]
The cost, in numbers
Let \(C_{\text{enc}}\) be the cost of one encoder pass, \(C_{\text{pair}}\) the cost of one cross-encoder pass on a (query, document) pair, and \(k\) the number of candidates we rerank. Per query:
Here \(N d\) counts multiply-adds for exact search; an approximate index reduces it a lot. The cross-encoder term grows with \(N\) and is paid for every query; the bi-encoder pays \(N\) encoder passes only once, offline.
Find the most similar pair among \(n = 10{,}000\) sentences. A cross-encoder must score every pair: \(n(n-1)/2 = 49{,}995{,}000\) forward passes. The SBERT paper puts this at about 50 million computations and about 65 hours with BERT; with SBERT (10,000 encoder passes, then cheap cosines) it takes about 5 seconds. [Certain]
Search a corpus of \(N = 1\) million passages with the small reranker cross-encoder/ms-marco-MiniLM-L6-v2, which its model card lists at 1,800 pairs per second on a V100 GPU. [Certain] Scoring everything: \(10^6 / 1800 = 556\) seconds, about 9 minutes, per query. Reranking only the bi-encoder’s top 100: \(100/1800 = 0.056\) s, 56 ms. On our laptop (Apple GPU, batches of 100) the same model did 30,000 pairs in 193 s, about 156 pairs per second: always measure on your own hardware. [Certain: our run]
How to read Figure 2
Click the four designs and follow the arrows: only the bi-encoder and late interaction compute document vectors before the query exists. Then read the table. At the default \(N\) = 1 million, the cross-encoder needs 556 s per query, while reranking the top 100 needs 56 ms and the bi-encoder index needs 3.07 GB. Push \(N\) to 1 billion: scoring every document with a cross-encoder would take about 6.4 days per query, but the rerank cost does not change, because it depends only on \(k\). Late interaction needs about 6 times more index memory than the bi-encoder (18.7 GB against 3.07 GB at 1 million).
Retrieve, then rerank
The standard design uses each model where it is strong: the bi-encoder (often together with BM25) finds a few hundred candidates with high recall; the cross-encoder puts the best ones on top, with high precision.
- offline: for each document \(d\) in the corpus: \(\mathbf{e}_d \leftarrow \text{normalize}(f(\text{prefix}_{\text{doc}} + d))\); add \(\mathbf{e}_d\) to the index
- input: query \(q\), first-stage depth \(K\) (for example 100), final size \(k\) (for example 10)
- \(\mathbf{e}_q \leftarrow \text{normalize}(f(\text{prefix}_{\text{query}} + q))\) // same model, same pooling, the query prefix
- \(C \leftarrow\) the \(K\) documents with the largest \(\mathbf{e}_q^\top \mathbf{e}_d\) // exact or approximate search, formula (150.1)
- optionally merge \(C\) with the BM25 top \(K\) // hybrid search
- for each \(d \in C\): \(r_d \leftarrow s_{\text{cross}}(q, d)\) // one batch of \(|C|\) pairs, formula (150.2)
- return the \(k\) documents of \(C\) with the largest \(r_d\)
The reranker can only reorder what the first stage found. If a relevant document is not in the top \(K\), it is lost forever:
So the two stages are tuned for different metrics: the first stage for Recall@\(K\) (did we catch it at all?), the second stage for nDCG@10 or MRR (is it on top?). This is also why a bigger \(K\) helps only until the first stage’s recall stops growing; after that it only costs reranker time.
What reranking did on SciFact
We retrieved the top 100 abstracts with all-MiniLM-L6-v2 (a 22 M-parameter bi-encoder, 384 dimensions) and reranked them with cross-encoder/ms-marco-MiniLM-L6-v2. Note that the reranker was trained on web search queries (MS MARCO), not on science. [Certain: our run]
| System on SciFact (300 queries) | nDCG@10 | MRR@10 | Recall@10 | Recall@100 |
|---|---|---|---|---|
| Bi-encoder only (all-MiniLM-L6-v2) | 0.645 | 0.605 | 0.783 | 0.925 |
| + rerank the top 20 | 0.687 | 0.656 | 0.811 | — |
| + rerank the top 100 | 0.689 | 0.659 | 0.814 | 0.925 |
Three lessons. Reranking raised nDCG@10 by 4.4 points even with a reranker from another domain. Recall@100 did not move at all, exactly as (150.5) says. And reranking 20 instead of 100 candidates gave almost the same nDCG@10 (0.687 against 0.689) at one fifth of the cost: measure the depth \(K\) instead of guessing it.
Pooling and normalization
A transformer gives one hidden vector per token. A bi-encoder needs one vector per text. Pooling is the rule that turns \(T\) token vectors into one (session 12 introduced the three choices). The rule is part of the model: use the pooling the model was trained with.
- Mean pooling: average the token vectors (Sentence-BERT, all-MiniLM-L6-v2, E5). [Certain]
- CLS pooling: take the vector of the first token (BGE, Snowflake Arctic-embed). [Certain]
- Last-token pooling: take the vector of the last token; this is the natural choice for decoder-only LLM embedders, because with causal attention only the last token has seen the whole text (E5-Mistral, for example). [Certain]
Mean pooling has one trap. In a batch, short texts are padded to the length of the longest one. The padding positions still produce hidden vectors, and they must not enter the average. With the attention mask \(m_t\) (1 for a real token, 0 for padding):
- Read the symbols
- \(\mathbf{h}_t\)
- the last-layer hidden vector at position \(t\)
- \(m_t\)
- the attention mask: 1 for real tokens, 0 for padding; the denominator is the number of real tokens
- \(T\)
- the padded length of the batch, which depends on the other texts in the batch
Without the mask, the vector of a text depends on which other texts happen to share its batch. In our toy code below, the same 5-token sentence alone and padded to 12 tokens gives cosine 1.000 with the mask and 0.432 without it. [Certain: our run]
Cosine, dot product or distance?
Three scores are common: the dot product \(\mathbf{q}^\top \mathbf{d}\), the cosine (150.1), and the (negative) Euclidean distance. They are tied by one identity:
So if all vectors have length 1, the three scores give the same ranking. Normalize once when you store the vectors, and then the fast dot product computes the cosine. If you do not normalize, the dot product rewards long vectors, and the length of a vector often reflects things like text length or word frequency, not relevance. Some models are trained with the raw dot product on purpose (DPR was); then you must use the dot product. The rule again: use the score the model was trained with. Figure 3 shows the effect in two dimensions.
How to read Figure 3
At the default query direction (25°) and the dot product, the long page C wins (1.157) over the on-topic answer A (0.996), although its angle to the query is ten times larger. Switch to cosine: A is first, C falls to third. Switch to Euclidean distance: short vectors such as E move up. Now tick “Normalize”: all three scores give exactly the same order, formula (150.7).
Query and document prefixes
Many retrieval models are trained with a short instruction in front of the text, and expect it at inference time. E5 wants query: before queries and passage: before documents; its model card says that without them “you will see a performance degradation”. [Certain] BGE and Arctic-embed add Represent this sentence for searching relevant passages: to queries only. [Certain] Instruction-tuned embedders (E5-Mistral and later) put a task description in front of each query, so one model can serve retrieval, classification and clustering. Forgetting the prefix is a silent bug: everything still runs, but the query vectors land in the wrong place. The table below measures how much.
The real cost of small pooling bugs
Same model, same data (SciFact), different pooling and scoring. The first rows use all-MiniLM-L6-v2, trained with masked mean pooling and normalized vectors. [Certain: our run]
| Variant | nDCG@10 | Recall@100 |
|---|---|---|
| all-MiniLM-L6-v2, masked mean pooling, cosine (as trained) | 0.645 | 0.925 |
| same, but dot product on unnormalized vectors (lengths 1.59 to 3.34) | 0.607 | 0.918 |
| same, but CLS pooling instead of mean | 0.606 | 0.916 |
| same, but mean pooling without the mask (padding averaged in) | 0.296 | 0.626 |
| Arctic-embed-m-v1.5, CLS pooling, with its query prefix (as trained) | 0.716 | 0.943 |
| Arctic-embed-m-v1.5, query prefix forgotten | 0.365 | 0.793 |
Two bugs cut nDCG@10 by about half: the missing mask (0.645 to 0.296) and, for Arctic, the forgotten query prefix (0.716 to 0.365). CLS pooling on a mean-trained model and the dot product on unnormalized vectors cost about 4 points each (0.606 and 0.607): about the gap between two neighboring models on a leaderboard. None of them raises an error. Note also that Arctic (with the right prefix) beats MiniLM by 7.1 points on this dataset; with the prefix forgotten, the better model becomes much the worse one.
You can watch the mechanism of two of these bugs in Figure 1. Choose “Mean, no mask (bug)”: the padding rows (red) enter the average, so a short document moves toward the padding vector, and its score falls for reasons that have nothing to do with its meaning. Choose CLS: the toy encoder was built for mean pooling, so its [CLS] row is almost the same for every text, and all scores crowd together. A real model that was trained with mean pooling shows the milder version of the same effect: the 4-point drop in the table. [Certain for the toy; the size of the real effect is our run above]
Matryoshka embeddings: one vector, many sizes
Storage and search time grow with the dimension \(d\). For \(N\) vectors of \(m\) numbers with \(b\) bits each:
Ten million passages with 768 fp32 numbers need \(10^7 \times 768 \times 4 = 30.7\) GB. Can we simply keep the first 256 numbers? For an ordinary model, not safely: its training has no reason to put the important information in the first coordinates. Any rotation of the space gives exactly the same cosines, so in the worst case the first 256 coordinates are just a random part of the information.
Matryoshka Representation Learning (MRL, Kusupati et al., 2022) fixes this during training. It applies the loss not only to the full vector but also to its first \(m\) coordinates, for several nested sizes \(m\), like Russian dolls:
- Read the symbols
- \(\mathbf{q}_{1:m}\)
- the first \(m\) coordinates of the query vector (the same for the document)
- \(\mathcal{L}\)
- the usual training loss, for embeddings the contrastive InfoNCE loss of session 60
- \(c_m\)
- a weight for each size, often all 1
Now the first coordinates must be useful on their own, so the model learns to put the most important information first. The paper reports, on ImageNet, “up to 14x smaller embedding size” at the same classification accuracy and “up to 14x real-world speed-ups” for retrieval. [Certain] OpenAI’s text-embedding-3 models support shortening through a dimensions parameter, and the docs say that if you cut vectors yourself you must renormalize them. [Certain] The method to use an MRL vector is three lines: keep the first \(m\) numbers, renormalize to length 1, search.
- for \(m\) in (2, 4, 8, 16, 32, 64):
- keep the first \(m\) numbers: \(\mathbf{q}_{1:m}, \mathbf{d}_{1:m}\) (memory \(N m b/8\), 150.8)
- renormalize: \(\mathbf{q}_{1:m} / \lVert \mathbf{q}_{1:m} \rVert\), same for every \(\mathbf{d}\) (if ticked)
- score all 1,000 documents; find the rank of the true document
- Recall@10 \(\leftarrow\) share of the 200 queries whose document is in the top 10
How to read Figure 4
Look first at \(m = 64\): both curves end at the same point, because a rotation does not change any cosine. All the difference is in where the information sits. At the default \(m = 8\) (one eighth of the storage), the Matryoshka model keeps Recall@10 of 0.730, the ordinary one only 0.145. Switch to “Ordinary model” and see why on the right: its energy is spread evenly, so the first 8 coordinates hold only about one eighth of it. Untick “Renormalize”: the raw dot product of truncated vectors is no longer a cosine, and recall changes. Now press Reset and Play, and follow the example query (query 2 of 200). In the Matryoshka model its true document climbs from rank 82 at \(m = 2\) to rank 31, then 5 (a hit, cosine 0.812 at \(m = 8\)), then 1. In the ordinary model it stays far down (rank 165 at \(m = 8\), cosine 0.400) and only reaches the top 10 at \(m = 32\). At \(m = 64\) both give the same cosine, 0.582.
Truncation on real models
We cut the vectors of two real models on SciFact and renormalized them. all-MiniLM-L6-v2 was not trained with MRL. Snowflake Arctic-embed-m-v1.5 (768 dims) was trained with MRL; its card says it keeps most of its quality at 256 dimensions. [Certain for the card claim; numbers: our run]
| Model | Kept dimensions | nDCG@10 | Share of full |
|---|---|---|---|
| all-MiniLM-L6-v2 (no MRL) | 384 (all) | 0.645 | 100% |
| 192 (1/2) | 0.614 | 95.2% | |
| 96 (1/4) | 0.553 | 85.8% | |
| 48 (1/8) | 0.441 | 68.3% | |
| 24 (1/16) | 0.274 | 42.5% | |
| Arctic-embed-m-v1.5 (MRL) | 768 (all) | 0.716 | 100% |
| 384 (1/2) | 0.703 | 98.1% | |
| 256 (1/3) | 0.698 | 97.5% | |
| 128 (1/6) | 0.630 | 88.0% | |
| 64 (1/12) | 0.483 | 67.5% | |
| 32 (1/24) | 0.290 | 40.5% |
Three honest lessons. (1) The MRL model kept 97.5% of its quality at one third of the size (256 dims), as its card promises; the non-MRL model kept 95.2% at one half. (2) Below the smallest size the model was trained for (256 for this model), quality falls about as fast as for an ordinary model: 67.5% at 64 dims. (3) Real ordinary models lose less than the random rotation of Figure 4 suggests, because their information is not spread perfectly evenly: real embeddings have a few strong directions. [Likely] Renormalizing mattered only for short prefixes: at 256 dims the raw dot product gave 0.699 (same as the cosine), at 64 dims 0.422 instead of 0.483. [Certain: our run]
Fewer bits per number
The other factor in (150.8) is \(b\). Scalar quantization stores each number in 8 or 4 bits instead of 32; binary quantization keeps only the sign (1 bit) and compares vectors with the Hamming distance, often followed by rescoring the best candidates with the full vectors. The Arctic-embed-m-v1.5 card reports 256 dimensions at int8 (256 bytes) keeping 99% of its MTEB retrieval score, and at int4 (128 bytes) keeping 98%. [Certain] Ten million 768-dim fp32 vectors take 30.7 GB; 256 dims in int8 take 2.56 GB; 768 bits per vector take 0.96 GB. The details of compressed indexes are in session 151.
Measuring retrieval: recall@k, MRR, nDCG
An evaluation set for retrieval has three parts: a corpus, a set of queries \(Q\), and relevance judgments (in BEIR and MTEB called qrels): for each query, which documents are relevant, with a grade such as 0 (not relevant) to 3 (perfect). session 60 defined the three standard metrics; here we use them with graded labels and with a reranker. For one query with \(R\) relevant documents in the corpus and a ranked list:
- Read the symbols
- \(\operatorname{rank}_1(q)\)
- the position of the first relevant result for query \(q\); if none is in the top \(k\), the term is 0
- \(g_i\)
- the relevance grade of the result at position \(i\); the gain \(2^{g_i} - 1\) is 0, 1, 3, 7 for grades 0 to 3, so a perfect answer is worth 7 times a partly relevant one
- \(\log_2(i+1)\)
- the position discount: 1 at the top, 2 at position 3, about 3.3 at position 10
- \(\text{IDCG@}k\)
- the DCG of the best possible list, built from all relevant documents of the query, also those the system did not retrieve; so nDCG is between 0 and 1
Each metric answers a different question. Recall@k: did we catch the relevant documents at all? (The right metric for a first stage, and for RAG, where the LLM reads the whole top \(k\).) MRR: how far down is the first good answer? (Good when one answer is enough, such as question answering.) nDCG@k: is the whole order right, with the best documents first? (The MTEB main metric for retrieval. [Certain]) Some tools report the share of queries with at least one hit in the top \(k\) as “hit rate”, “success@k” or “top-k accuracy”, and DPR’s “top-20 accuracy” is this number; it is not the same as recall when a query has several relevant documents.
The first stage returns 10 documents with grades 1, 0, 0, 3, 0, 2, 0, 0, 1, 0; one more relevant document (grade 2) exists but was not retrieved. So \(R = 5\), with grades 3, 2, 2, 1, 1.
Recall@10 \(= 4/5 = 0.8\). MRR \(= 1/1 = 1\) (position 1 is partly relevant).
DCG@10 \(= \frac{1}{\log_2 2} + \frac{7}{\log_2 5} + \frac{3}{\log_2 7} + \frac{1}{\log_2 10} = 1 + 3.015 + 1.069 + 0.301 = 5.384\) (rounded parts; the exact sum is 5.3843).
IDCG@10 \(= \frac{7}{1} + \frac{3}{\log_2 3} + \frac{3}{2} + \frac{1}{\log_2 5} + \frac{1}{\log_2 6} = 7 + 1.893 + 1.5 + 0.431 + 0.387 = 11.210\). So nDCG@10 \(= 5.384 / 11.210 = 0.480\).
The reranker puts the order d4, d1, d6, d2, d9, …: grades 3, 1, 2, 0, 1. DCG@10 \(= 7 + 0.631 + 1.5 + 0 + 0.387 = 9.518\), nDCG@10 \(= 0.849\). Recall@10 is still 0.8 and MRR still 1: only nDCG sees that the perfect answer moved to the top.
How to read Figure 5
The default shows the worked example: nDCG@10 0.480 in the first stage and 0.849 after reranking, with Recall@10 0.800 and MRR 1.000 in both. Move \(k\) to 3: now recall differs too (0.200 against 0.600), because the reranker brought two relevant documents into the top 3. Click a grade box to change a label. Make d1 “not relevant”: the first-stage MRR drops to 0.250 (first hit at position 4) while the reranked MRR stays 1.000.
Two traps of retrieval metrics
Unjudged documents count as not relevant. Labels are usually collected only for documents that some system once retrieved (pooling of candidates). A new model that finds a good document nobody labeled gets no credit. MS MARCO has on average about one labeled relevant passage per query, so many good results are “wrong” by the labels. [Likely] When a new model looks worse, read its top results before you believe the number.
Small test sets are noisy. A metric is a mean over queries, so it has a standard error. For \(n\) queries with per-query scores \(x_i\) and, when comparing two systems on the same queries, per-query differences \(\Delta_i\):
On SciFact the per-query nDCG@10 of all-MiniLM-L6-v2 has standard deviation 0.399, so with 300 queries its mean 0.645 has a standard error of 0.023. Arctic’s 0.716 is 0.071 higher. Compared query by query, the differences have standard deviation 0.250, so the paired standard error is 0.014 and the gap is 4.9 standard errors: real. Arctic was better on 26.7% of the queries, worse on 11.7%, and equal on 61.7%. The unpaired standard error (0.031) is more than twice the paired one: always compare two systems on the same queries. With only 50 queries, the paired standard error would be about 0.035, and a 3-point gain could be noise. [Certain: our run]
Public benchmarks (MTEB) vs your own data
MTEB (Massive Text Embedding Benchmark, Muennighoff et al., 2022) tests one embedding model on many jobs: 8 task types (retrieval, reranking, clustering, classification, pair classification, semantic similarity, summarization, bitext mining), 58 datasets and 112 languages in the first version, with 33 models compared. Its main finding: “no particular text embedding method dominates across all tasks.” [Certain] The retrieval part reuses BEIR datasets such as SciFact, scored with nDCG@10, so our number 0.645 for all-MiniLM-L6-v2 is the kind of number the leaderboard shows. MMTEB (Enevoldsen et al., 2025) extends it to over 500 tasks in more than 250 languages; there the best public model was multilingual-e5-large-instruct, with only 560 million parameters. [Certain]
A leaderboard is a good way to choose a few candidates. It is a poor way to choose the final model, for five reasons:
- The average hides the task. The headline number averages retrieval with classification, clustering and similarity. For search, read only the retrieval column, and better the datasets closest to your domain.
- Training data overlap. Many models train on the training splits of datasets that appear in the benchmark (MS MARCO, NQ, HotpotQA, FEVER), so part of the score is in-domain, not zero-shot. [Likely]
- Your domain is different. Legal contracts, support tickets in three languages, code, sports commentary: none of these looks like Wikipedia or scientific abstracts.
- Your queries are different. Real users type two words, typos and product codes; benchmark queries are often clean full questions.
- Cost is not in the score. A 7 B-parameter embedder with 4,096 dimensions may beat a 100 M one by two points and cost 50 times more to run and 5 times more to store.
- collect 100 to 500 real queries (search logs, support tickets, questions users asked the chatbot); remove duplicates; keep the hard and the rare ones
- pool candidates: for each query take the top 10 to 20 from several different systems (BM25, two or three embedding models, with and without reranker)
- judge every pooled (query, document) pair with a grade 0 to 3 // humans, or an LLM judge checked against humans on a sample
- freeze the corpus, queries and labels as version 1; never tune on the test queries // keep a separate dev split
- for each candidate system: compute Recall@\(K\) of the first stage and nDCG@10 of the final list, formulas (150.10), (150.11)
- report the mean, the standard error (150.12) and a paired comparison with the current system; read 20 failures by eye
- when a new system wins, judge its unlabeled top results too and add them to the pool // the labels grow over time
This set is the most valuable asset of a search team. It is also the input of the next sessions: session 153 evaluates retrieval and generation separately with it, and session 160 builds one.
Code
First, the ideas of this session in plain PyTorch with no downloads: masked mean pooling (150.6), Matryoshka truncation on the toy data of Figure 4 (with PyTorch’s own random numbers, so the values differ a little from the figure), and the metrics of Figure 5.
import math, torch import torch.nn.functional as F torch.manual_seed(0) # 1) mean pooling: average only the real tokens (formula 150.6) def mean_pool(H, mask): # H: (B, T, h), mask: (B, T) with 1 = real token m = mask.unsqueeze(-1).float() return (H * m).sum(1) / m.sum(1).clamp(min=1e-9) h = 8 tokens = torch.randn(5, h) # hidden states of one 5-token sentence pad = torch.randn(1, h) # what the model outputs at a padding position alone = tokens.unsqueeze(0) # batch of 1: no padding padded = torch.cat([tokens, pad.expand(7, h)]).unsqueeze(0) # same sentence padded to 12 mask_a = torch.ones(1, 5); mask_p = torch.tensor([[1.] * 5 + [0.] * 7]) good = F.cosine_similarity(mean_pool(alone, mask_a), mean_pool(padded, mask_p)).item() bad = F.cosine_similarity(alone.mean(1), padded.mean(1)).item() # the bug: no mask print(f"same sentence, with mask: cos {good:.4f}; without mask: cos {bad:.4f}") # 2) a toy corpus: information sorted by importance (Matryoshka) or spread out (ordinary) N, d, Q, sig = 1000, 64, 200, 1.5 s = torch.exp(-torch.arange(d) / 20.0); s = s / s.norm() * math.sqrt(d) D = torch.randn(N, d) * s # document vectors Qv = D[:Q] + sig * torch.randn(Q, d) # query i is a noisy copy of document i Rot, _ = torch.linalg.qr(torch.randn(d, d)) # a random rotation: same information, spread out target = torch.arange(Q) def recall_at_k(Qx, Dx, k=10): top = (Qx @ Dx.T).topk(k, dim=1).indices return (top == target[:, None]).any(1).float().mean().item() for m in [4, 8, 16, 64]: # truncate, then renormalize (cosine) mrl = recall_at_k(F.normalize(Qv[:, :m], dim=-1), F.normalize(D[:, :m], dim=-1)) ordi = recall_at_k(F.normalize((Qv @ Rot)[:, :m], dim=-1), F.normalize((D @ Rot)[:, :m], dim=-1)) print(f"m={m:2d} Matryoshka-like R@10 {mrl:.3f} ordinary R@10 {ordi:.3f} bytes/vector fp32 {4 * m}") # 3) metrics for one ranked list (formulas 150.10 and 150.11); grades 0..3 def metrics(grades, all_relevant, k): dcg = sum((2 ** g - 1) / math.log2(p + 2) for p, g in enumerate(grades[:k])) ideal = sorted(all_relevant, reverse=True) idcg = sum((2 ** g - 1) / math.log2(p + 2) for p, g in enumerate(ideal[:k])) recall = sum(g > 0 for g in grades[:k]) / len(all_relevant) mrr = next((1 / (p + 1) for p, g in enumerate(grades[:k]) if g > 0), 0.0) return recall, mrr, dcg / idcg first = [1, 0, 0, 3, 0, 2, 0, 0, 1, 0] # grades in bi-encoder order (Figure 5) rerank = [first[i] for i in [3, 0, 5, 1, 8, 2, 4, 6, 7, 9]] all_rel = [g for g in first if g > 0] + [2] # one relevant document was never retrieved for name, lst in [("bi-encoder", first), ("reranked", rerank)]: print(name, "R@10 %.3f MRR %.3f nDCG@10 %.3f" % metrics(lst, all_rel, 10)) # Output (PyTorch 2.8, CPU): # same sentence, with mask: cos 1.0000; without mask: cos 0.4319 # m= 4 Matryoshka-like R@10 0.295 ordinary R@10 0.065 bytes/vector fp32 16 # m= 8 Matryoshka-like R@10 0.765 ordinary R@10 0.190 bytes/vector fp32 32 # m=16 Matryoshka-like R@10 0.935 ordinary R@10 0.430 bytes/vector fp32 64 # m=64 Matryoshka-like R@10 0.985 ordinary R@10 0.985 bytes/vector fp32 256 # bi-encoder R@10 0.800 MRR 1.000 nDCG@10 0.480 # reranked R@10 0.800 MRR 1.000 nDCG@10 0.849
Second, the real pipeline we ran for the tables above (it needs sentence-transformers and pyarrow, and downloads SciFact and two small models the first time). Encoding is deterministic up to tiny floating-point differences between devices.
import csv, math, torch, pyarrow.parquet as pq import torch.nn.functional as F from huggingface_hub import hf_hub_download from sentence_transformers import SentenceTransformer, CrossEncoder get = lambda f, repo="BeIR/scifact": hf_hub_download(repo, f, repo_type="dataset") corpus = pq.read_table(get("corpus/corpus-00000-of-00001.parquet")).to_pylist() queries = {q["_id"]: q["text"] for q in pq.read_table(get("queries/queries-00000-of-00001.parquet")).to_pylist()} qrels = {} with open(get("test.tsv", "BeIR/scifact-qrels")) as f: rows = csv.reader(f, delimiter="\t"); next(rows) for qid, did, score in rows: if int(score) > 0: qrels.setdefault(qid, set()).add(did) qids = sorted(qrels, key=int) dids = [c["_id"] for c in corpus] docs = [(c["title"] + " " + c["text"]).strip() for c in corpus] qs = [queries[q] for q in qids] def evaluate(S, k=10): # S: (queries, docs) scores; nDCG@10 and Recall@100 top = S.topk(100, dim=1).indices.tolist(); nd = rc = 0.0 for i, q in enumerate(qids): rel, lst = qrels[q], [dids[j] for j in top[i]] dcg = sum(1 / math.log2(p + 2) for p, d in enumerate(lst[:k]) if d in rel) nd += dcg / sum(1 / math.log2(p + 2) for p in range(min(k, len(rel)))) rc += len(rel & set(lst)) / len(rel) return f"nDCG@10 {nd / len(qids):.3f} Recall@100 {rc / len(qids):.3f}" def embed(model, texts): # the model's own pooling; we normalize ourselves return F.normalize(model.encode(texts, batch_size=32, convert_to_tensor=True).float().cpu(), dim=-1) # 1) bi-encoder: masked mean pooling, cosine (as trained) mini = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") D, Q = embed(mini, docs), embed(mini, qs) S = Q @ D.T print("MiniLM bi-encoder ", evaluate(S)) # 2) rerank the top 100 with a cross-encoder (Algorithm 1) ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2") S2 = torch.full_like(S, -1e9) for i, top in enumerate(S.topk(100, dim=1).indices): S2[i, top] = torch.tensor(ce.predict([(qs[i], docs[j]) for j in top.tolist()], batch_size=100), dtype=torch.float32) print("+ rerank top 100 ", evaluate(S2)) # 3) a Matryoshka model with CLS pooling and a query prefix arc = SentenceTransformer("Snowflake/snowflake-arctic-embed-m-v1.5") prefix = "Represent this sentence for searching relevant passages: " Da, Qa = embed(arc, docs), embed(arc, [prefix + q for q in qs]) print("Arctic, 768 dims ", evaluate(Qa @ Da.T)) print("Arctic, no prefix ", evaluate(embed(arc, qs) @ Da.T)) for m in [256, 64]: # truncate, renormalize, search print(f"Arctic, first {m:3d} dims", evaluate(F.normalize(Qa[:, :m], dim=-1) @ F.normalize(Da[:, :m], dim=-1).T)) # Output (our run, Apple M-series GPU, sentence-transformers 5.1, PyTorch 2.8): # MiniLM bi-encoder nDCG@10 0.645 Recall@100 0.925 # + rerank top 100 nDCG@10 0.689 Recall@100 0.925 # Arctic, 768 dims nDCG@10 0.716 Recall@100 0.943 # Arctic, no prefix nDCG@10 0.365 Recall@100 0.793 # Arctic, first 256 dims nDCG@10 0.698 Recall@100 0.950 # Arctic, first 64 dims nDCG@10 0.483 Recall@100 0.774
Where this is used
| Application | Typical setup | What to watch |
|---|---|---|
| RAG for chatbots and assistants | Hybrid BM25 + bi-encoder top 50–100, cross-encoder rerank, top 5–10 to the LLM | First-stage Recall@\(K\) limits everything (150.5); chunk size; query prefixes |
| Site, help-center and e-commerce search | Bi-encoder fine-tuned on click logs, reranker on the top 100 | Product codes and names need keyword search too; latency budget for the reranker |
| Duplicate and near-duplicate detection (questions, tickets, bug reports) | Symmetric bi-encoder, cosine above a tuned threshold | The threshold depends on the model; tune it on labeled pairs |
| Clustering and topic discovery | Embed everything, k-means or HDBSCAN on normalized vectors | Normalize first, so that k-means on Euclidean distance means cosine (150.7) |
| Agent memory and semantic caching of LLM answers | Embed past turns or past questions; reuse when cosine is very high | A false cache hit returns a wrong answer: use a strict threshold |
| Code search | Code-trained embedders, natural-language query to function | General text models do poorly on code; evaluate on your repository |
| Text-to-video retrieval and Video-RAG | CLIP-style text and frame or clip embeddings in one space; rerank with a video-language model | One vector per clip, not per video; pool over frames the same way as over tokens |
| Key-moment search in long videos | Embed transcript windows and frames every few seconds; return time stamps | Neighboring windows are near duplicates: merge hits before computing recall |
| Video and audio fingerprinting | Compact (often binary) embeddings of short segments, nearest-neighbor match | Binary codes and Hamming distance; storage per hour of video (150.8) |
Common mistakes
- Mean pooling without the attention mask. Vectors then depend on batch composition. In our run it cut nDCG@10 from 0.645 to 0.296. Use the library’s pooling, or (150.6).
- A different pooling, score or prefix than the model was trained with. CLS instead of mean, dot product on unnormalized vectors, a forgotten
query:prefix: each costs a few points silently. Read the model card. - Truncating a non-Matryoshka model, or forgetting to renormalize. Cutting all-MiniLM-L6-v2 to 48 of 384 dimensions kept only 68% of its nDCG@10. Truncate only MRL models, and renormalize after cutting.
- Using a cross-encoder for the whole corpus, or a bi-encoder alone when precision at the top matters. Use both, in two stages, and measure the depth \(K\).
- Tuning the reranker while the first stage misses the answer. By (150.5) no reranker can fix a document that was not retrieved. Look at first-stage Recall@\(K\) first.
- Choosing a model from the MTEB average. Read the retrieval column and your closest domains, then decide on your own test set.
- Mixing vectors from two models or two model versions in one index. Their spaces are not compatible, even with the same dimension. Re-embed the whole corpus when you change the model.
- Silent truncation of long documents. all-MiniLM-L6-v2 cuts its input at 256 word pieces; everything later is invisible to search. [Certain] Chunk documents to the model’s length.
Interview questions
What is the difference between a bi-encoder and a cross-encoder? When do you use each?basic
Why not run the cross-encoder over the whole corpus?basic
Cosine similarity or dot product?core
What pooling would you use for a BERT-style embedder, and for a decoder-only LLM embedder? Why?core
What is a Matryoshka embedding and how do you use it?core
Define recall@k, MRR and nDCG@k. Which would you report for the retriever of a RAG system?core
Compute nDCG@3 for the grades (0, 3, 1) if the query has exactly these two relevant documents.core
Your new embedding model is number one on MTEB, but in your product search it is worse than the old one. What do you check?advanced
How does ColBERT differ from a bi-encoder and a cross-encoder?advanced
How would you build an evaluation set for a new search product with no labels yet?advanced
Ten million documents, 768-dim fp32 embeddings. How much memory, and how would you cut it?core
Exercises
- A cross-encoder scores 400 pairs per second on your GPU. How long does it take to rerank the top 50 for one query, and to score a corpus of 200,000 documents for one query?
- Two unit vectors have cosine 0.8. What is their Euclidean distance?
- Your batch has texts of 6, 10 and 40 tokens. Without the mask, the 6-token text is averaged over how many positions, and what share of its vector comes from padding?
- Grades in ranked order: (2, 0, 1, 0, 3). The query has exactly these three relevant documents. Compute Recall@3, MRR and nDCG@5.
- A first stage has Recall@100 = 0.85 and Recall@20 = 0.70. You rerank the top 20. What is the highest possible Recall@10 after reranking?
- How much memory do 50 million MRL vectors take if you keep 256 of 1,024 dimensions in int8? And at full size in fp16?
- Two models differ by 0.02 nDCG@10 on 300 queries, and the per-query differences have standard deviation 0.20. Is the difference likely to be real?
Answerscheck
Key takeaways
- A bi-encoder embeds texts separately (150.1): documents offline, one pass per query. A cross-encoder reads the pair together (150.2): more accurate, but \(N\) passes per query (150.4). Late interaction (ColBERT, 150.3) sits in between and pays in index size.
- Retrieve with the bi-encoder (and BM25), rerank the top \(K\) with a cross-encoder. Reranking cannot recover what the first stage missed (150.5). On SciFact, reranking raised nDCG@10 from 0.645 to 0.689 with Recall@100 unchanged.
- Use the pooling, score and prefixes the model was trained with. Masked mean pooling (150.6); normalize once and the dot product is the cosine (150.7). A missing mask halved nDCG@10 in our run.
- Matryoshka training (150.9) puts the important information in the first coordinates, so vectors can be truncated and renormalized; memory is \(N m b/8\) bytes (150.8). Do not truncate models trained without it.
- Recall@k for the first stage, MRR and nDCG@k for the final order (150.10, 150.11); report standard errors (150.12). MTEB chooses candidates; your own labeled queries choose the model.
Key formulas
References for this session
- Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering
- Khattab & Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Kusupati et al. (2022). Matryoshka Representation Learning
- Muennighoff et al. (2022). MTEB: Massive Text Embedding Benchmark
- Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering
- Khattab & Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Kusupati et al. (2022). Matryoshka Representation Learning
- Muennighoff et al. (2022). MTEB: Massive Text Embedding Benchmark
- Thakur et al. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Wang et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Wang, Yang, Huang, Yang, Majumder, Wei (2024). Improving Text Embeddings with Large Language Models
- Enevoldsen, Chung et al. (2025). MMTEB: Massive Multilingual Text Embedding Benchmark
- Snowflake (2024). snowflake-arctic-embed-m-v1.5 model card
- Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond