Natural language inference and question answering
Two classic language-understanding tasks that are still used every day. Natural language inference (NLI) asks whether one sentence follows from another, and an NLI model can label text with classes it never saw in training. Question answering (QA) finds the answer to a question in a passage, or first searches a large collection for the right passage. This session builds both from the model head to the metrics, with the exact rules of the SQuAD scoring script.
What you will learn
- The three NLI labels (entailment, neutral, contradiction), the SNLI and MultiNLI datasets, and why a model that reads only the hypothesis already scores far above chance
- How an NLI model becomes a zero-shot classifier, and the difference between the single-label and multi-label scoring rules
- How extractive QA predicts a start and an end position, how it is trained, and how the best legal span is decoded
- How long passages are split into overlapping windows, and how SQuAD 2.0 models say "no answer"
- The retrieve-then-read design of open-domain QA, and why retrieval recall limits the whole system
- Exact match and token F1: the normalization rules, the max over gold answers, and why both are unfair to chatty LLM answers
Why this matters
Two questions sit under a large part of language understanding. Does this sentence follow from that one? And where, in this text, is the answer to my question? The first is natural language inference (NLI). The second is question answering (QA).
Both tasks are older than chat LLMs, but they did not go away. They became parts of bigger systems:
- A small NLI model is a cheap zero-shot classifier: it can tag support tickets or news with labels it never saw in training, with no labeled data and no LLM call.
- NLI is a common way to check whether a summary or a RAG answer is supported by its sources: “does the source entail this sentence?” (session 156).
- Extractive QA (point to the answer inside a passage) gives answers that are exact quotes, with their location. That matters in legal, medical and enterprise search, where a made-up answer is not acceptable.
- Open-domain QA, “retrieve a few passages, then read them”, is the direct ancestor of RAG (session 152).
- Exact match and token F1 are still the default metrics for short-answer QA, including for LLMs. You must know what they reward and what they punish.
The previous session (session 144) put one label on one text. This session puts a label on a pair of texts (NLI) and then predicts a span (QA). The next session (session 146) labels every token.
Natural language inference: the task
NLI gets two sentences. The premise \(P\) is taken as true. The hypothesis \(H\) is a new claim. The model must say which of three relations holds:
- Entailment: if \(P\) is true, \(H\) must also be true. “A man is playing a guitar on stage.” entails “A man is playing an instrument.”
- Contradiction: if \(P\) is true, \(H\) cannot be true. Same premise, “The man is asleep in bed.”
- Neutral: \(P\) does not decide \(H\). Same premise, “The man is a famous musician.” Maybe; the premise does not say.
“Must be true” here means what a normal reader would conclude, not strict logic. A careful reader would accept “instrument” from “guitar” even though no formal rule is written down. This is why NLI is also called recognizing textual entailment (RTE), and why it tests real language understanding: word meaning, negation, numbers, world knowledge.
The datasets: SNLI and MultiNLI
| Dataset | Size | Where the premises come from | Note |
|---|---|---|---|
| SNLI (Bowman et al., 2015) | 570K pairs [Certain] | Image captions (Flickr30k) | Hypotheses written by crowd workers, who saw only the caption, not the image [Likely] |
| MultiNLI (Williams et al., 2017) | 433K pairs [Certain] | Ten genres of written and spoken English (fiction, government reports, telephone speech, ...) [Certain] | Two test conditions: matched (same genres as training) and mismatched (unseen genres) |
For each premise, a crowd worker wrote three hypotheses: one true, one maybe true, one false. This made data collection fast and cheap. It also left traces, which we look at below.
MultiNLI is part of the GLUE benchmark. BERT-large reached 86.7% accuracy on it [Certain]. Today MultiNLI-trained models, such as facebook/bart-large-mnli and many DeBERTa models, are the standard off-the-shelf NLI models.
The model: a cross-encoder with a 3-way head
The standard NLI model reads both sentences together as one input: [CLS] premise [SEP] hypothesis [SEP]. Every premise token can attend to every hypothesis token in every layer. This is called a cross-encoder. The final vector at the [CLS] position, \(\mathbf{h}_{\text{CLS}}\), goes to a linear layer with 3 outputs (session 33):
- Read the symbols
- \(\mathbf{h}_{\text{CLS}} \in \R^{d}\)
- the final hidden state at the first position; it has seen both sentences
- \(W \in \R^{3 \times d}, \mathbf{b} \in \R^{3}\)
- the new classification head, trained with cross-entropy on the gold label
- C, N, E
- contradiction, neutral, entailment. The order of these three outputs differs between models: always read it from the model’s
config.json
The head outputs logits \((z_C, z_N, z_E) = (-1.2,\ 0.4,\ 2.6)\). The softmax gives \(e^{-1.2} : e^{0.4} : e^{2.6}\), normalized: \(p = (0.020,\ 0.098,\ 0.882)\). The prediction is entailment with probability 0.882.
A decoder LLM can do NLI with a prompt (“Premise: ... Hypothesis: ... Answer with entailment, neutral or contradiction.”), and you can score it by comparing the probabilities of the three label words, as in session 144. A fine-tuned cross-encoder of a few hundred million parameters is still much cheaper per pair, which matters when you must check thousands of pairs.
Annotation artifacts: the hypothesis gives the answer away
Because workers wrote hypotheses quickly, each label got its own style. Contradictions often contain negation words (“nobody”, “never”, “sleeping”). Neutral hypotheses often add purpose or detail (“for a competition”). Entailments often use general words (“a person”, “outdoors”) [Likely]. Gururangan et al. trained a classifier that sees only the hypothesis, never the premise. It scored about 67% on SNLI and 53% on MultiNLI, where chance is 33% [Certain]. A model can therefore look strong without doing inference at all.
NLI as a zero-shot classifier
Here is the useful trick (Yin, Hay and Roth, 2019). To decide whether a text belongs to a class, turn the class name into a hypothesis and ask the NLI model whether the text entails it:
- premise: “The camera is great, but at $1,200 the phone costs too much.”
- hypothesis for label “price”: “This example is about price.”
The NLI model has never seen your labels, but it understands sentences, so it can judge them. You run one premise-hypothesis pair per candidate label. The Hugging Face zero-shot-classification pipeline uses the template "This example is {}." by default [Certain] and offers two scoring rules. Let \(z_E^{(k)}\) and \(z_C^{(k)}\) be the entailment and contradiction logits for label \(k\) of \(K\).
Single-label (exactly one label is right): softmax of the entailment logits across labels.
Multi-label (each label is a separate yes/no question): for each label alone, a softmax over its contradiction and entailment logits; neutral is dropped.
- Read the symbols
- \(z_E^{(k)}, z_C^{(k)}\)
- the entailment and contradiction logits of the pair (text, hypothesis built from label \(k\))
- \(\sigma\)
- the sigmoid \(1/(1+e^{-t})\); a 2-way softmax is a sigmoid of the difference
- \(K\)
- the number of candidate labels; cost grows linearly with \(K\), one forward pass per label
The two rules answer different questions. Single-label scores always sum to 1, so the model is forced to pick something, even for a text that fits no label. Multi-label scores are independent, so a text can fit two labels, or none. Try it in Figure 1. The logits there are illustrative numbers that we chose, not outputs of a real model.
How to read Figure 1
Text A is about one thing (battery): both rules agree. Text B is about two things (price and camera): the single-label rule splits the mass, 0.534 and 0.437, so neither looks confident; the multi-label rule gives 0.984 and 0.976, both clearly yes. Text C is about none of the labels: the single-label rule still names a winner (camera, 0.338), while the multi-label rule, at threshold 0.5, says “no label”. Move the threshold to see how many labels pass.
Entailment logits for (battery, price, camera, screen) are \((-1.8,\ 2.6,\ 2.4,\ -0.6)\). Single-label (145.2): softmax gives \((0.007,\ 0.534,\ 0.437,\ 0.022)\). Multi-label (145.3) for price: \(z_E - z_C = 2.6 - (-1.5) = 4.1\), and \(\sigma(4.1) = 0.984\). For camera: \(\sigma(2.4 + 1.3) = \sigma(3.7) = 0.976\). For battery: \(\sigma(-1.8 - 2.1) = 0.020\).
- input: text \(x\), candidate labels \(\ell_1, \dots, \ell_K\), template “This example is about {}.”, mode, threshold \(t\)
- for \(k = 1\) to \(K\): \(H_k \leftarrow\) template filled with \(\ell_k\)
- \(\quad (z_C^{(k)}, z_N^{(k)}, z_E^{(k)}) \leftarrow \text{NLI}(\text{premise} = x,\ \text{hypothesis} = H_k)\) // batch all K pairs in one call
- if mode is single-label: \(p \leftarrow \softmax_k(z_E^{(k)})\); return \(\argmax_k p_k\) // formula (145.2)
- else: \(p_k \leftarrow \sigma(z_E^{(k)} - z_C^{(k)})\) for each \(k\); return \(\{\ell_k : p_k \gt t\}\) // formula (145.3)
Practical points. The label words and the template matter: “This example is about price.” and “This review complains about the price.” can give quite different scores [Likely]. Tune them on 50 to 100 labeled examples if you have them. The cost is \(K\) forward passes per text, so 50 labels on a million texts is 50 million passes; for many labels, an embedding model that compares the text with label descriptions is cheaper (session 150). And the threshold \(t\) for multi-label needs calibration, as in session 144.
Extractive QA: point to the answer
In extractive QA the input is a question \(q\) and a passage \(c\) (the context), and the answer must be a span: a run of consecutive tokens copied from the passage. SQuAD (Rajpurkar et al., 2016) made this task popular: more than 100,000 questions written by crowd workers on Wikipedia articles, where each answer is a segment of the passage [Certain]. In the paper, a logistic regression model reached 51.0% F1 and humans 86.8% F1 [Certain]. Two years later BERT reached 93.2 F1 on the SQuAD 1.1 test set [Certain].
The model: one start vector, one end vector
Feed [CLS] question [SEP] passage [SEP] to an encoder. Let \(\mathbf{h}_i \in \R^d\) be the final hidden state of token \(i\). BERT adds only two vectors, a start vector \(\mathbf{s}\) and an end vector \(\mathbf{e}\) (in code, one linear layer with 2 outputs). Each token gets a start logit and an end logit, and a softmax over all positions gives two distributions:
- Read the symbols
- \(\mathbf{s}, \mathbf{e} \in \R^{d}\)
- learned start and end vectors (the only new weights: \(2d + 2\) numbers with biases)
- \(\mathbf{s}^\top \mathbf{h}_i\)
- the start logit of token \(i\): how much token \(i\) looks like the first token of the answer
- \(k\)
- runs over all positions of the input, including
[CLS], which SQuAD 2.0 uses for “no answer”
Training uses the gold start \(i^*\) and gold end \(j^*\): the loss is the average of two cross-entropies.
The gold positions come from the dataset as a character offset in the passage. You must map it to token positions after tokenization, using the tokenizer’s offset mapping. Off-by-one errors here silently cap your score. BERT’s SQuAD recipe fine-tuned for 3 epochs with learning rate \(5 \times 10^{-5}\) and batch size 32 [Certain].
Decoding: the best legal span
At test time we want the span \((i, j)\) with the highest score. A span is legal only if it starts before it ends, it lies inside the passage (not in the question), and it is not too long. The score of a span is the sum of its two logits, which is the log of \(p_{\text{start}}(i)\,p_{\text{end}}(j)\) up to a constant:
\(L_{\max}\) is the maximum answer length in tokens; BERT’s code uses 30 [Certain]. Why not take the argmax of start and the argmax of end separately? Because the two are predicted independently, and the best end can lie before the best start. Figure 2 shows exactly this case.
[CLS] holds the no-answer score. Click a token to see its probabilities.How to read Figure 2
For the first question, the highest start logit is on “1889” (4.0) but the highest end logit is on “1887” (4.2). Choose “Separate argmax”: the result starts at position 11 and ends at position 4 (position 0 is [CLS]), which is not a span at all. The legal search compares all pairs with \(i \le j\) and finds “1889” (score 7.9), ahead of “1887” (7.2) and the long span “1887 ... 1889” (6.9). For the second question the passage has no answer; its [CLS] logits are high (null score 5.8) and the best span scores only 1.8, so with τ = 0 the model says “no answer”. Raise τ to 4.0 or more and it is forced to answer “the tower”, which is wrong.
With a softmax over the [CLS] slot and the 12 passage tokens, \(p_{\text{start}}(\text{1889}) = 0.658\) and \(p_{\text{start}}(\text{1887}) = 0.242\), while \(p_{\text{end}}(\text{1887}) = 0.558\) and \(p_{\text{end}}(\text{1889}) = 0.413\). The top legal spans are (1889, 1889) with \(4.0 + 3.9 = 7.9\), (1887, 1887) with \(3.0 + 4.2 = 7.2\) and (1887, 1889) with \(3.0 + 3.9 = 6.9\). A passage of 12 tokens has \(12 \cdot 13 / 2 = 78\) spans with \(i \le j\); with \(L_{\max} = 3\) only 33 remain.
Unanswerable questions and long passages
SQuAD 2.0: knowing when to say nothing
A SQuAD 1.1 model always answers, because every training question has an answer. In real search, often no passage contains one. SQuAD 2.0 (Rajpurkar, Jia and Liang, 2018) added over 50,000 unanswerable questions, written by crowd workers to look like answerable ones [Certain]. In that paper, a strong system with 86% F1 on SQuAD 1.1 got only 66% F1 on SQuAD 2.0 [Certain].
BERT handles this with the [CLS] token: the gold “span” of an unanswerable question is \((0, 0)\), the [CLS] position. At test time it compares the null score with the best real span:
The threshold \(\tau\) is tuned on the dev set to maximize F1 (BERT’s code has null_score_diff_threshold, default 0.0) [Certain]. A higher \(\tau\) answers more often; a lower one abstains more often. This is the same precision-recall trade-off as any classifier threshold.
Passages longer than the model: sliding windows
An encoder has a maximum input length (512 tokens for BERT). BERT’s SQuAD code uses max_seq_length = 384 and splits a longer passage into overlapping windows [Certain]. Each window contains the full question plus a chunk of the passage. With \(M\) the maximum input length and \(|q|\) the question length in tokens, each window holds
- Read the symbols
- \(m\)
- passage tokens per window; the 3 is for
[CLS]and the two[SEP] - \(n\)
- the passage length in tokens
- \(r\)
- the step between window starts (BERT’s
doc_stride, default 128 [Certain]) - \(N_{\text{win}}\)
- the number of windows, each one a full forward pass
Each window is scored on its own. The final answer is the best span over all windows. A token that appears in several windows is judged in the window where it has the most context on both sides (the “max context” rule of BERT’s code: the window that maximizes \(\min(\text{left}, \text{right}) + 0.01 \cdot \text{window length}\)). For SQuAD 2.0 the null score of the example is the smallest null score over its windows, because one window that contains the answer is enough [Certain].
run_squad.py.How to read Figure 3
With the defaults (\(n = 1000\), \(M = 384\), \(|q| = 20\), \(r = 128\)) each window holds \(m = 361\) passage tokens and there are 6 windows, so the model reads the passage about 2.2 times. A smaller step gives more overlap and more compute. A step larger than \(m\) is cut to \(m\) by the loop, so no token is skipped. The orange window is the one whose prediction is used for the answer token.
Open-domain QA: retrieve, then read
So far someone gave us the right passage. In open-domain QA there is only the question, and the answer is somewhere in millions of documents, for example all of Wikipedia. Running a reader on every passage is impossible. Chen et al. (2017, DrQA) proposed the design that is still used: a fast retriever picks the top \(k\) passages, then a slower reader finds the answer in them. Their retriever was TF-IDF over bigram hashes, and their reader a recurrent network [Certain].
Dense retrieval. DPR (Karpukhin et al., 2020) encodes the question and each passage separately with two BERT encoders and scores them by a dot product:
The passage vectors \(E_P(c)\) are computed once, offline, and stored in a vector index, so a query costs one encoder pass plus a nearest-neighbor search (session 151). DPR beat a strong BM25 system by 9 to 19 points of top-20 retrieval accuracy on several open-domain QA datasets [Certain]. Top-\(k\) retrieval accuracy is the fraction of questions where at least one of the \(k\) retrieved passages contains the answer. Note the contrast with NLI: the NLI model and the reader are cross-encoders (question and passage read together, accurate, slow), the retriever is a bi-encoder (encoded apart, fast, less accurate).
The reader. An extractive reader scores spans in each passage with (145.6) and takes the best one over all \(k\) passages; a good system also adds a passage-relevance score so that a high span score in an irrelevant passage does not win. A generative reader writes the answer instead: Fusion-in-Decoder (Izacard and Grave, 2020) encodes each passage with the question separately and lets one T5 decoder attend to all of them [Certain]. RAG (Lewis et al., 2020) trains the retriever and a generator together. An LLM reading retrieved passages in its prompt is the same pattern (session 152).
- offline: split documents into passages (for example 100 words); compute \(E_P(c)\) for all; build an index
- input: question \(q\), number of passages \(k\), max answer length \(L_{\max}\)
- \(\mathbf{u} \leftarrow E_Q(q)\); \(\;C_k \leftarrow\) the \(k\) passages with the largest \(\mathbf{u}^\top E_P(c)\) // formula (145.9)
- for each \(c \in C_k\): split into windows (145.8); run the reader on
[CLS] q [SEP] window [SEP] - \(\quad\) best span in each window with (145.6); keep its score and the null score
- if every window’s null score beats its best span by more than \(\tau\): return “no answer” // formula (145.7)
- return the span with the highest score over all passages, with its passage id as the source
The two stages multiply. If the answer is in the top \(k\) passages for 80% of the questions, and the reader is right 70% of the time when the answer is present, the system cannot do better than about 80%, and you should expect something near \(0.8 \times 0.7 = 0.56\) (if the two errors are independent). So always measure them separately: retrieval recall@\(k\) first, then reader accuracy on questions whose answer was retrieved. Larger \(k\) raises recall but gives the reader more distractors and costs more.
Metrics: exact match and token F1
SQuAD defined the two metrics that most short-answer QA still uses. Both first normalize the strings, exactly as the official script does: lowercase, remove punctuation, remove the articles “a”, “an”, “the”, and collapse white space [Certain].
Exact match (EM) is 1 if the normalized prediction equals a normalized gold answer, else 0. Token F1 treats the prediction and the gold answer as bags of words and measures their overlap:
- Read the symbols
- \(\hat a, a\)
- the predicted answer and one gold answer, as lists of normalized tokens (split on spaces)
- \(|\hat a \cap a|\)
- the number of shared tokens, counting repeats (a multiset intersection); if it is 0, \(F_1 = 0\)
- \(P, R\)
- precision (what fraction of the prediction is right) and recall (what fraction of the gold answer is covered)
Questions in the SQuAD dev and test sets have several gold answers from different annotators. The score of one question is the maximum over its gold answers, and the dataset score is the average over questions:
F1 is always at least as high as EM, because a right answer with one extra word still gets partial credit. Type any answer in Figure 5 to see both.
How to read Figure 5
“in 1889” against “1889”: EM 0, F1 0.667 (precision 1/2, recall 1). “The Eiffel Tower!” against “eiffel tower”: EM 1, because case, punctuation and “the” are removed. The chatty answer “The tower was completed in the year 1889.” contains the right answer but gets F1 0.286 and EM 0. Gustave Eiffel's company against “Gustave Eiffel”: removing the apostrophe makes “eiffels”, a different token, so F1 is only 0.4. (The script removes only ASCII punctuation, so a curly ’ would stay inside the token: another way to lose points.) “1887” against “1889” gets 0 on both: a wrong number is simply wrong.
Prediction “the Eiffel Tower in Paris”, gold “Eiffel Tower”. Normalized: [eiffel, tower, in, paris] and [eiffel, tower]. Shared tokens: 2. \(P = 2/4 = 0.5\), \(R = 2/2 = 1\), \(F_1 = 2 \cdot 0.5 \cdot 1 / 1.5 = 0.667\). EM = 0. With a second gold answer “the Eiffel Tower in Paris”, EM and F1 both become 1 by (145.12).
EM and F1 for LLM answers
EM and F1 were made for extractive models whose answers are short spans. A chat LLM writes full sentences, so its F1 drops even when it is right, as Figure 5 shows (0.286 for a correct answer). Common fixes:
- Ask for a short answer in the prompt (“Answer with a few words only.”) and give a few examples of the format (session 143).
- Report answer containment (“does the normalized gold answer appear inside the normalized prediction?”). It is generous: a long answer that lists many candidates also passes.
- Use an LLM judge or human review for semantic equivalence (“two thousand” vs “2,000”) (session 155). Then check the judge against human labels.
Whatever you choose, report it exactly, because EM, F1 and containment numbers on the same outputs can differ by many points.
Putting it together in PyTorch
The code below has three parts: the SQuAD metrics (145.10 to 145.12), the best-legal-span decoder (145.6), and a tiny extractive reader trained from scratch on a synthetic task with the loss (145.5). The task: the input is [CLS] key length [SEP] passage, and the answer is the length tokens right after the key token in the passage. The model must find the key (question-passage matching) and mark the span. It runs on a CPU in about 10 seconds and is deterministic.
import re, string, collections, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) SQuAD metrics: normalize, exact match, token F1, max over gold answers (145.10-145.12) def normalize(s): s = s.lower() s = "".join(ch for ch in s if ch not in set(string.punctuation)) s = re.sub(r"\b(a|an|the)\b", " ", s) return " ".join(s.split()) def em(pred, gold): return float(normalize(pred) == normalize(gold)) def f1(pred, gold): p, g = normalize(pred).split(), normalize(gold).split() common = sum((collections.Counter(p) & collections.Counter(g)).values()) if common == 0: return 0.0 prec, rec = common / len(p), common / len(g) return 2 * prec * rec / (prec + rec) def score(pred, golds): return max(em(pred, g) for g in golds), max(f1(pred, g) for g in golds) print(score("in 1889", ["1889", "in 1889 for the fair"])) # (0.0, 0.667) print(score("The Eiffel Tower", ["Eiffel tower!"])) # (1.0, 1.0) # 2) best legal span from start and end logits: i <= j < i + max_len (145.6) def best_span(start_logits, end_logits, max_len=30): s = start_logits[:, None] + end_logits[None, :] # score[i, j] n = s.shape[0] i, j = torch.meshgrid(torch.arange(n), torch.arange(n), indexing="ij") legal = (j >= i) & (j - i < max_len) s = s.masked_fill(~legal, float("-inf")) return divmod(s.argmax().item(), n), s.max().item() # 3) a tiny extractive reader on a synthetic task V, P, CLS, SEP = 30, 16, 0, 1 def batch(B): keys = torch.randint(10, V, (B,)) lens = torch.randint(1, 4, (B,)) pas = torch.randint(10, V, (B, P)) pos = torch.randint(0, P - 3, (B,)) for b in range(B): # the key appears exactly once while (pas[b] == keys[b]).any(): pas[b][pas[b] == keys[b]] = torch.randint(10, V, ((pas[b] == keys[b]).sum(),)) pas[b, pos[b]] = keys[b] x = torch.cat([torch.full((B, 1), CLS), keys[:, None], (lens + 1)[:, None], torch.full((B, 1), SEP), pas], 1) start = 4 + pos + 1 end = start + lens - 1 return x, start, end class Reader(nn.Module): def __init__(self, d=64): super().__init__() self.emb = nn.Embedding(V, d); self.pos = nn.Embedding(4 + P, d) layer = nn.TransformerEncoderLayer(d, 4, 4 * d, dropout=0.0, batch_first=True, norm_first=True) self.enc = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False) self.qa = nn.Linear(d, 2) # the start and end vectors s, e (145.4) def forward(self, x): h = self.enc(self.emb(x) + self.pos(torch.arange(x.shape[1]))) s, e = self.qa(h).unbind(-1) return s, e model = Reader() opt = torch.optim.AdamW(model.parameters(), lr=3e-3) for step in range(600): x, st, en = batch(64) s, e = model(x) loss = (F.cross_entropy(s, st) + F.cross_entropy(e, en)) / 2 # formula (145.5) opt.zero_grad(); loss.backward(); opt.step() if step % 200 == 0: print(step, round(loss.item(), 3)) # 3.41, 2.61, 0.391 model.eval(); torch.manual_seed(1) x, st, en = batch(500) with torch.no_grad(): s, e = model(x) exact = 0 for b in range(500): (i, j), _ = best_span(s[b, 4:], e[b, 4:], max_len=5) # search passage tokens only exact += int(i + 4 == st[b] and j + 4 == en[b]) print("span exact match:", exact / 500) # 1.0
The printed numbers in the comments come from running this code with PyTorch 2.8 on a CPU. Other versions may give slightly different losses. Note the plateau near 2.6 at step 200: the model first has to learn to match the key, then the loss falls fast. Also note s[b, 4:]: the decoder searches only the passage, never the question, which is the “inside the passage” rule of (145.6).
With a real pretrained model, the same steps use library code. The snippet below downloads a model and is not run here:
from transformers import pipeline zs = pipeline("zero-shot-classification", model="facebook/bart-large-mnli") zs("The camera is great, but at $1,200 the phone costs too much.", candidate_labels=["battery", "price", "camera", "screen"], hypothesis_template="This example is about {}.", multi_label=True) # formula (145.3) per label qa = pipeline("question-answering", model="distilbert-base-cased-distilled-squad") qa(question="When was the tower completed?", context="Construction began in 1887 and the tower was completed in 1889.") # best legal span, (145.6)
Where this is used
| Application | Which idea | How |
|---|---|---|
| Tagging tickets, news or reviews with new labels, no training data | NLI zero-shot (145.2), (145.3) | One hypothesis per label; multi-label mode when a text can have several topics |
| Fact checking a summary or a RAG answer | NLI | Premise = source passage, hypothesis = each generated sentence; flag sentences that are not entailed (session 156) |
| Document search that must quote the source (legal, medical, manuals) | Extractive QA | The answer is a span with an exact location, so it can be highlighted and checked |
| Open-domain QA and RAG | Retrieve, then read (145.9) | Dense or BM25 retriever, then an extractive reader or an LLM (session 152) |
| Video key-moment detection (temporal grounding) | Start and end prediction (145.4) to (145.6) | Same head over time: given “the moment the goal is scored”, predict a start frame and an end frame over the clip’s features |
| Video QA and Video-RAG | Retrieve, then read | Retrieve clips or transcript segments for a question, then let a Video-LLM read them (session 210) |
| Video classification with new labels | Zero-shot by text hypotheses | The CLIP-style version of the same idea: compare the clip with “a video of {label}” (session 169) |
| Evaluating short-answer QA (SQuAD, Natural Questions, TriviaQA) | EM and F1 (145.10) to (145.12) | Official scripts; for LLM outputs add a short-answer prompt or a judge |
Common mistakes
- Wrong label order for the NLI head.
facebook/bart-large-mnliuses 0 = contradiction, 1 = neutral, 2 = entailment [Certain]; other models use other orders. Readid2labelfrom the config; never hard-code index 2. - Single-label scoring for multi-topic text. Softmax across labels (145.2) forces the scores to share one unit of mass, so two true labels both look weak and an off-topic text still gets a confident-looking winner.
- Taking argmax start and argmax end separately. The end can come before the start, or the span can be 200 tokens long. Search legal pairs only (145.6).
- Allowing spans in the question or in special tokens. Mask every position outside the passage before decoding, except
[CLS]for the null score. - Character-to-token offset bugs. SQuAD gives character offsets. Map them with the tokenizer’s offset mapping and check by decoding the gold span back to text for a few hundred examples.
- Truncating long passages instead of windowing. The answer is often after token 384. Use overlapping windows (145.8) and remember that “stride” means step in BERT code and overlap in Hugging Face tokenizers.
- Comparing EM/F1 numbers computed with different normalizations. A scorer that does not drop articles and punctuation reports lower numbers for the same outputs. Use the official script.
- Blaming the reader when the retriever failed. Measure recall@\(k\) first; the reader cannot find an answer that is not in its input.
Interview questions
What are the three NLI labels? Give an example of each.basic
How do you classify text into labels you have no training data for, using an NLI model?core
What is a hypothesis-only baseline and why does it matter?core
How does BERT do extractive QA? What are the new parameters and the loss?core
[CLS] question [SEP] passage [SEP]. BERT adds a start vector \(\mathbf{s}\) and an end vector \(\mathbf{e}\) (one linear layer with 2 outputs). Each token gets a start logit \(\mathbf{s}^\top\mathbf{h}_i\) and an end logit \(\mathbf{e}^\top\mathbf{h}_j\); softmaxes over positions give \(p_{\text{start}}\) and \(p_{\text{end}}\) (145.4). The loss is the average of the cross-entropies at the gold start and gold end (145.5). Everything else is the pretrained encoder, fine-tuned end to end (3 epochs, lr \(5\times10^{-5}\), batch 32 in the paper).Why not take the argmax of the start logits and the argmax of the end logits?core
How do you answer questions over a passage longer than the model’s input?advanced
How does a model handle unanswerable questions in SQuAD 2.0?advanced
[CLS]. At test time compute the null score \(s_{\text{null}} = s_{\text{CLS}} + e_{\text{CLS}}\) and the best legal span score; predict “no answer” if \(s_{\text{null}}\) minus the best span score is above a threshold \(\tau\) (145.7). Tune \(\tau\) on the dev set for F1; it trades answering wrongly against abstaining wrongly. SQuAD 2.0 added over 50,000 adversarial unanswerable questions; a system with 86% F1 on SQuAD 1.1 fell to 66% on it.Define EM and F1. Why is F1 never below EM?basic
Design an open-domain QA system over 10 million documents. Where do errors come from?advanced
With strong LLMs, is extractive QA still useful?advanced
Your LLM gets F1 0.3 on a QA set, but people say its answers are good. What is going on?core
Exercises
- An NLI zero-shot classifier gives entailment logits \((2, 1, 0)\) for three labels. Compute the single-label scores. For the first label the contradiction logit is 1; compute its multi-label score.
- Compute EM and F1 for the prediction “Barack Obama” against the gold answer “President Barack Obama”.
- A passage has 800 tokens, \(M = 256\), the question has 13 tokens, and the step is \(r = 128\). How many windows are there, and where does each start?
- A passage has 12 tokens. How many spans with \(i \le j\) are there? How many with \(L_{\max} = 3\)?
- In Figure 2, which value of \(\tau\) makes the model answer the second question, and is that good?
- Your retriever has recall@10 of 0.8 and your reader is right 70% of the time when the answer is in its passages. Estimate the end-to-end accuracy. Which stage would you improve first?
Answerscheck
Key takeaways
- NLI labels a premise-hypothesis pair as entailment, neutral or contradiction with a cross-encoder and a 3-way head (145.1). SNLI and MultiNLI are the standard data; hypothesis-only models score 67% and 53%, so check for shortcuts.
- An NLI model is a zero-shot classifier: one hypothesis per label, softmax across labels for single-label (145.2), \(\sigma(z_E - z_C)\) per label for multi-label (145.3).
- Extractive QA adds a start and an end vector (145.4), trains with two cross-entropies (145.5), and decodes the best legal span (145.6).
- Long passages are split into overlapping windows (145.8); unanswerable questions use the
[CLS]null score and a tuned threshold (145.7). - Open-domain QA retrieves with a fast bi-encoder (145.9) and reads with a slow cross-encoder or a generator; the stages multiply, so measure recall first.
- EM and F1 normalize, compare tokens, and take the max over gold answers (145.10 to 145.12). They undervalue long LLM answers, so control the answer format or add a judge.
Key formulas
References for this session
- Bowman et al. (2015). A large annotated corpus for learning natural language inference
- Williams et al. (2017). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
- Rajpurkar et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Bowman et al. (2015). A large annotated corpus for learning natural language inference
- Williams et al. (2017). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
- Rajpurkar et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Rajpurkar, Jia & Liang (2018). Know What You Don't Know: Unanswerable Questions for SQuAD
- Gururangan et al. (2018). Annotation Artifacts in Natural Language Inference Data
- Yin, Hay & Roth (2019). Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
- Chen, Fisch, Weston & Bordes (2017). Reading Wikipedia to Answer Open-Domain Questions
- Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering
- Izacard & Grave (2020). Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Meta AI / Hugging Face (model). facebook/bart-large-mnli model card and config.json (id2label: 0 contradiction, 1 neutral, 2 entailment)
- Hugging Face (code). transformers source: pipelines/zero_shot_classification.py (hypothesis template and the single-label and multi-label scores)
- Google Research (code). BERT source: run_squad.py (max_seq_length 384, doc_stride 128, max_answer_length 30, null score)
- Rajpurkar et al. (code). Official SQuAD v1.1 evaluation script (normalize_answer, exact_match_score, f1_score)