Hallucination, calibration and uncertainty
A language model writes a wrong answer in the same calm, fluent voice as a right one. This session explains why that happens (the training objective, missing knowledge, grading that rewards guessing, and decoding), how to measure whether a model’s confidence can be trusted (calibration, ECE, temperature scaling), how to detect a likely hallucination (self-consistency, semantic entropy, retrieval checks), and how to design products that say “I don’t know” at the right moments.
What you will learn
- What a hallucination is: factuality errors and faithfulness errors, with examples
- Four causes: the next-token objective, facts seen only once, grading that rewards guessing, and decoding that compounds errors
- Calibration: reliability diagrams, expected calibration error (ECE), Brier score, and temperature scaling fitted by NLL
- Why the expected-score rule gives a clean abstention threshold t = k/(1+k) for a wrong-answer penalty k
- Detection without labels: self-consistency, SelfCheckGPT, and semantic entropy (and why plain token entropy fails on paraphrases)
- Detection with sources: retrieval checks, claim-level verification, Chain-of-Verification and FActScore
- Product designs that reduce harm: grounding with citations, abstention, tools for exact facts, verification passes and honest UI
Why this matters
Ask a model for the birthday of a little-known scientist, and it may answer “14 March 1962” with full confidence. The date can be invented. The sentence looks exactly like a true one: same grammar, same calm tone. This is a hallucination: fluent, plausible text that is not supported by the facts or by the sources the model was given.
Hallucinations are the main reason LLM products fail in law, medicine, finance and customer support. A lawyer who files a brief with invented case citations, a support bot that promises a refund policy that does not exist, a coding assistant that calls an API function that was never written: in each case the model did not “lie” on purpose. It did what it was trained to do, which is to produce likely text. Likely text and true text are not the same thing.
The useful engineering question is not “how do we make hallucination zero?” (we cannot, see below) but three smaller questions. Why does the model produce a wrong answer instead of saying “I don’t know”? How confident is it, and can we trust that confidence? And how do we detect a likely error and design the product so that the error does less harm? This session answers all three, with numbers.
In session 155 we learned how to measure a system. Here we measure one special property: whether the system knows when it is wrong. In session 157 we will look inside the model for the signals that carry this knowledge.
What counts as a hallucination
The survey of Huang et al. splits hallucinations into two families. [Certain]
- Factuality hallucination: the output disagrees with facts about the world. Example: “The Eiffel Tower was finished in 1925.” (It was finished in 1889.) Sub-types: wrong facts, and invented facts (a paper, a person, a court case that does not exist).
- Faithfulness hallucination: the output disagrees with the input: the user’s instruction, the document to summarize, or the retrieved passages. Example: a summary says “the company’s profit rose” when the article says it fell. The output may even be true in the world and still be unfaithful, because it was not in the source.
The difference matters for detection. Factuality needs world knowledge or a search engine to check. Faithfulness can be checked against the given text alone, which is much easier to automate. In RAG systems (session 152) we mostly care about faithfulness: every claim must be supported by a retrieved passage.
Vision-language models have their own version: object hallucination, describing objects that are not in the image (the POPE benchmark asks yes/no questions like “Is there a dog in the image?” to measure it). Video models add event hallucination: describing an action that never happens in the clip, or giving the wrong time for it.
Why models hallucinate
There is not one cause. There are four, and they stack.
Cause 1: the training objective never says “unknown”
Pretraining minimizes the negative log-likelihood of the next token (session 3):
- Read the symbols
- \(x_t\)
- the token at position \(t\) in a training document
- \(x_{\lt t}\)
- all tokens before it (the context)
- \(p_\theta\)
- the model’s probability, with weights \(\theta\)
This objective rewards putting probability on what the text says next, not on what is true. After “Marie Curie was born in”, the training text says “1867”, so the model learns to put mass on years. For a person the model saw only once, it still must put its probability somewhere, and it puts it on plausible years. Sampling from that distribution produces a plausible year. The model has no special token for “I have no memory of this”. The format of a biography (born in, a year, a city) is easy to learn; the specific fact is hard.
Cause 2: some facts are seen only once
Kalai and Vempala proved a striking result: a model that is calibrated (its probabilities match reality, defined below) must hallucinate on “arbitrary” facts, such as birthdays, that appear only once in training, even if the training data has no errors. [Certain] Kalai, Nachum, Vempala and Zhang (2025) state the practical version: the hallucination rate of a base model on such facts should be at least about the singleton rate, the fraction of those facts that appear exactly once in the training data. [Certain]
Their example: if 20% of birthday facts appear exactly once in the pretraining data, expect the base model to hallucinate on at least about 20% of birthday questions. [Certain] The intuition comes from an old statistics idea (Good–Turing): the fraction of things you saw once is a good estimate of how much you have not seen at all. A fact seen once gives the model no way to tell it apart from the many plausible facts it never saw.
Fine-tuning can make this worse. Gekhman et al. found that when supervised fine-tuning teaches facts the model did not know from pretraining, the model learns them slowly, and once it does learn them, its tendency to hallucinate grows. [Certain] Training the model to answer questions it cannot answer teaches it to answer confidently anyway.
Cause 3: grading rewards guessing
Most benchmarks give 1 point for a correct answer and 0 for both a wrong answer and “I don’t know”. Under this rule, guessing is always at least as good as abstaining. Kalai et al. (2025) argue that this is the main reason post-trained models keep hallucinating: training and evaluation reward good test-takers, and good test-takers guess. [Certain]
Let us write it down. Suppose the model believes its answer is correct with probability \(p\). A correct answer earns 1 point, a wrong answer costs \(k\) points, and “I don’t know” earns 0:
- Read the symbols
- \(p\)
- the probability that the answer is correct (the model’s confidence, if it is calibrated)
- \(k\)
- the penalty for a wrong answer; \(k = 0\) is the usual benchmark rule
- \(t\)
- the confidence threshold above which answering beats abstaining
With \(k = 0\), the threshold is \(t = 0\): always answer. With \(k = 1\), answer only above 50%. With \(k = 2\), above 2/3. With \(k = 9\), above 90%. Kalai et al. propose putting this rule in the instructions (“answer only if you are more than \(t\) confident; mistakes cost \(t/(1-t)\) points”), so that the best behavior is clear and can be graded. [Certain] Note that \(t/(1-t) = k\): the two forms say the same thing.
The model is 60% sure. Under the usual 0/1 rule, answering earns \(0.6\) on average and abstaining earns 0, so it answers. Now let a wrong answer cost \(k = 2\) (a wrong medical fact is worse than no fact). Answering earns \(0.6 - 0.4 \times 2 = -0.2\). Abstaining earns 0. So it should abstain. The threshold is \(t = 2/3 \approx 0.667\), and 0.6 is below it. Try it in Figure 1.
How to read Figure 1
Set \(k = 0\): the blue line is never below 0, so answering is always best. This is how most benchmarks grade, and why a model tuned to win them learns to guess. Raise \(k\): the line tilts, and the point where it crosses 0 (the orange threshold) moves right. Any confidence left of the threshold should give “I don’t know”. This only works if \(p\) is a calibrated probability, which is the subject of the next sections.
Cause 4: decoding compounds small errors
Generation picks one token at a time (session 79). Sampling with a high temperature or a large top-\(p\) sometimes picks a low-probability token, and once a wrong name or number is written, the model continues as if it were true: the next tokens are conditioned on it. Nothing in the loop goes back to fix it. Long answers multiply the risk. If each independent factual claim is wrong with probability \(e\), an answer with \(n\) claims contains at least one error with probability
With \(e = 2\%\) and \(n = 50\) claims (a long biography), that is \(1 - 0.98^{50} = 63.6\%\). Even with \(e = 1\%\) it is \(39.5\%\). This is why long-form factuality is measured per claim (FActScore splits text into atomic facts and checks each one) rather than per answer. [Certain]
A fifth, softer cause: models copy human mistakes. TruthfulQA collected questions where many people hold a false belief; its authors reported that in their tests the largest models were often the least truthful, because they imitated the popular wrong answer better. [Likely] Figure 2 puts the causes and their main fixes side by side.
Calibration: can we trust the confidence?
A model is calibrated if, among all the answers it gives with confidence 0.8, about 80% are correct. Formally, for the predicted answer \(\hat{Y}\) and its confidence \(\hat{P}\):
Calibration is different from accuracy. A weather forecaster who says “30% rain” on days when it rains 30% of the time is perfectly calibrated, even though they never say “it will rain” with certainty. A model that is 55% accurate can be perfectly calibrated if its confidences average 55% in the right places. And a model can be very accurate and badly calibrated, if it says 99% when it is right 90% of the time.
Measuring it: the reliability diagram and ECE
Sort the predictions into \(M\) bins by confidence (for example 10 bins of width 0.1). For each bin \(B_m\), compare the average confidence \(\text{conf}(B_m)\) with the fraction correct \(\text{acc}(B_m)\). A plot of accuracy against confidence is a reliability diagram; a calibrated model lies on the diagonal. The expected calibration error summarizes the gaps, weighted by how many predictions fall in each bin:
- Read the symbols
- \(N\)
- the number of predictions
- \(B_m\)
- the predictions whose confidence falls in bin \(m\); \(|B_m|\) is how many
- \(\text{acc}(B_m)\)
- the fraction of those predictions that are correct
- \(\text{conf}(B_m)\)
- their average confidence
100 answers. 40 have confidence about 0.95 and 70% of them are right. 30 have confidence 0.75 and 60% are right. 30 have confidence 0.55 and 50% are right. Then \(\text{ECE} = 0.4 \times |0.70 - 0.95| + 0.3 \times |0.60 - 0.75| + 0.3 \times |0.50 - 0.55| = 0.100 + 0.045 + 0.015 = 0.16\). Every bin is overconfident: the model says more than it knows.
Figure 3 runs formula (156.6) in the browser on 20 real-looking predictions. First every prediction drops into the bin of its confidence. Then each bin computes its accuracy, its mean confidence and its share of the ECE, and the sum grows bin by bin. Change the number of bins to see that ECE is not one fixed property of a model: it also depends on how you bin.
- \(B_1, \dots, B_M \leftarrow\) empty bins; ECE \(\leftarrow 0\)
- for each prediction \(i\) with confidence \(\hat{p}_i\):
- \(m \leftarrow \min(M,\ \lfloor \hat{p}_i M \rfloor + 1)\); add \(i\) to \(B_m\)
- for each non-empty bin \(B_m\):
- \(\text{acc} \leftarrow\) fraction correct; \(\text{conf} \leftarrow\) mean \(\hat{p}\)
- ECE \(\leftarrow\) ECE \(+ \frac{|B_m|}{N}\,|\text{acc} - \text{conf}|\) // (156.6)
- return ECE
How to read Figure 3
Press Play or Step. The highlighted line of the algorithm box runs in this step; the box next to it shows the same formula with the real numbers. With 10 bins, prediction 1 has confidence 0.97 and goes to the last bin. When all 20 are placed, the bins are evaluated from left to right. The last bin holds 6 predictions with mean confidence 0.955 but only 4 of them correct (0.667), so it adds \(\frac{6}{20} \times 0.288 = 0.0865\); the total is ECE = 0.193. With 5 bins every bin is overconfident and ECE = 0.186. With 1 bin, ECE is just \(|0.60 - 0.786| = 0.186\). The 10-bin value is larger because the bin 0.6–0.7 is slightly underconfident (0.667 versus 0.643): with coarse bins, over- and underconfident predictions cancel each other.
ECE has known weaknesses: it depends on the number of bins, and it only looks at the top answer. A proper scoring rule rewards honest probabilities without bins. The two common ones are the negative log-likelihood (NLL, the same cross-entropy as training) and the Brier score, the squared error between the probability vector and the one-hot truth:
Here \(p_{ic}\) is the probability the model gives to class \(c\) on example \(i\), \(K\) is the number of classes, and \(\mathbb{1}[y_i = c]\) is 1 for the true class and 0 otherwise. Report ECE because people understand it; tune and compare with NLL or Brier because they cannot be gamed by binning.
Why modern networks are overconfident, and the one-number fix
Guo et al. (2017) found that modern deep networks are much less calibrated than older, smaller ones: they are overconfident. [Certain] Training keeps pushing the logits of the correct class higher, long after the accuracy has stopped improving, so the softmax becomes too sharp. Their fix is surprisingly simple: temperature scaling. Divide all logits by one number \(T \gt 0\), learned on a held-out set by minimizing NLL:
\(T \gt 1\) makes the distribution flatter (less confident); \(T \lt 1\) makes it sharper. Because dividing by a positive number does not change which logit is largest, accuracy does not change. Only the confidence does. Example: logits \((3, 1, 0, 0)\) give a top probability of 0.81 at \(T = 1\) and 0.55 at \(T = 2\). [Certain]
How do we find \(T^\ast\)? It is a one-dimensional problem, so a grid search works. Gradient descent shows better what happens. To keep \(T\) positive, optimize \(u = \ln T\) instead. The derivative of the NLL in (156.8) with respect to \(u\) is short:
- Read the symbols
- \(z_{i, y_i}\)
- the logit of the true answer of question \(i\)
- \(\sum_c \hat{p}_{ic} z_{ic}\)
- the average logit under the current scaled probabilities
- \(\eta\)
- the learning rate
Read the sign. If the true answers have higher logits than the model’s average, the gradient is positive, so \(u\) goes down and \(T\) shrinks: the model may be sharper. If the true answers have lower logits than average (the model is often confidently wrong), the gradient is negative and \(T\) grows. At \(T^\ast\) the two balance. Figure 4 runs this loop on 1,000 questions and redraws the reliability diagram after every step.
What about LLMs? Kadavath et al. found that large pretrained models are fairly well calibrated on multiple-choice and true/false questions when the question has the right format, and that a model can be asked to judge its own answer: P(True), the probability it gives to “this answer is true”. [Certain] The GPT-4 report showed that the pretrained model was well calibrated on MMLU and that post-training (RLHF) made calibration noticeably worse. [Likely] This matches cause 3: preference training rewards answers that sound confident. So a chat model’s token probabilities are often overconfident, and you must recalibrate them before you use them as thresholds.
- \(u \leftarrow \ln T_0\)
- repeat: \(T \leftarrow e^{u}\); \(\hat{p}_i \leftarrow \softmax(\mathbf{z}_i / T)\) for all \(i\)
- \(\mathcal{L} \leftarrow -\frac{1}{N}\sum_i \log \hat{p}_{i, y_i}\) // (156.8)
- \(g \leftarrow \frac{1}{N}\sum_i (z_{i,y_i} - \sum_c \hat{p}_{ic} z_{ic}) / T\) // (156.9)
- if \(|g| \lt 0.001\): stop
- \(u \leftarrow u - \eta\, g\)
How to read Figure 4
At \(T = 1\) the blue bars are far below the diagonal on the right: answers given with 90–100% confidence are right only about 74% of the time. ECE is about 0.22. Press Play: each iteration computes the NLL and the gradient (156.9) on all 1,000 questions, then moves \(u\). From \(T = 1\) with \(\eta = 1\) the first gradient is −0.761, so \(T\) jumps to 2.14; the gradient shrinks as \(T\) nears the minimum, and the loop stops after 8 updates at \(T = 2.47\), close to the true 2.5 used to make the data. ECE drops to about 0.03, NLL falls from 1.332 to 1.041, and accuracy stays at 55.1% for every \(T\). Try \(\eta = 2\): the first step overshoots to \(T = 4.58\) (now underconfident: bars above the diagonal) and the next steps come back. From \(T = 0.5\) with \(\eta = 2\) the first step flies far off the chart, because the gradient at a very sharp softmax is large. You can also drag the slider to set \(T\) by hand.
Uncertainty for free-form answers
Temperature scaling works when the answer is one of a few classes. A chat answer is a sentence, and the same meaning can be written in many ways. How do we get one confidence number for it?
Token probabilities and sequence entropy
The simplest score is the probability of the generated answer, \(\log p(\mathbf{y} \mid x) = \sum_j \log p(y_j \mid y_{\lt j}, x)\), usually divided by the length so long answers are not punished. A related score is the entropy of the answer distribution, estimated from \(S\) samples \(\mathbf{y}^{(1)}, \dots, \mathbf{y}^{(S)}\):
The problem: these scores measure uncertainty about the words, not about the meaning. “Jane Austen.”, “It was written by Jane Austen.” and “The author is Jane Austen.” are three different strings. A model that is completely sure of the fact can still spread its probability over many phrasings, so the entropy looks high. This is what Kuhn, Gal and Farquhar call the problem of semantic equivalence. [Certain]
Self-consistency and SelfCheckGPT
A first fix: sample several answers and check whether they agree. If the model knows a fact, it tends to produce it again and again; if it is guessing, the samples disagree. This is the idea of self-consistency (used for reasoning by Wang et al., session 72) and of SelfCheckGPT, which samples extra responses and checks each sentence of the main answer against them, with no access to the model’s internals and no external database. [Certain] The simplest agreement score is the fraction of samples that share the most common meaning.
Semantic entropy
Kuhn et al. make this precise. Sample \(S\) answers. Group them into meaning clusters: two answers go in the same cluster if each one entails the other (an NLI model, session 145, or an LLM judge decides this; it is called bidirectional entailment). Then compute the entropy over clusters instead of over strings:
- Read the symbols
- \(c\)
- one meaning cluster (for example “Canberra”)
- \(n_c\)
- how many of the \(S\) samples fell into cluster \(c\)
- \(p(c \mid x)\)
- the probability of meaning \(c\); the count version shown here is the “discrete” semantic entropy; one can also sum the sequence probabilities of the members
Low semantic entropy: the model keeps saying the same thing, in any words. High semantic entropy: it keeps saying different things, which is a strong sign of a confabulation (an arbitrary, made-up answer). Farquhar et al. showed in Nature (2024) that semantic entropy detects such errors across several datasets and models better than token-level baselines. [Certain] It cannot catch an error the model makes consistently: if training taught a wrong fact, all samples agree on it, and semantic entropy is low.
Ten samples, all meaning “Jane Austen” but with 9 different wordings. Entropy over normalized strings: \(-\big(0.2 \ln 0.2 + 8 \times 0.1 \ln 0.1\big) = 2.164\) nats. Semantic entropy: one cluster, \(\text{SE} = 0\). Now ten samples for a question the model cannot know, giving 8 different years: \(\text{SE} = -(2 \times 0.2 \ln 0.2 + 6 \times 0.1 \ln 0.1) = 2.025\) nats. String entropy cannot tell the two cases apart (2.164 in both!); semantic entropy can. A split of 8 versus 2 gives \(-(0.8 \ln 0.8 + 0.2 \ln 0.2) = 0.500\) nats. For comparison, 10 equally likely meanings would give \(\ln 10 = 2.303\).
Figure 5 runs this procedure step by step. Samples arrive one at a time. Each new sample is compared with the representative (the first member) of every existing cluster; if both directions of entailment hold, it joins that cluster, otherwise it starts a new one. The entropies are recomputed after every sample, so you can watch the estimate settle.
- clusters \(\leftarrow\) [ ]
- for \(s = 1 \dots S\): \(\mathbf{y}^{(s)} \leftarrow\) sample at temperature about 1
- for each cluster \(c\): if entails\((\mathbf{y}^{(s)}, \mathbf{r}_c)\) and entails\((\mathbf{r}_c, \mathbf{y}^{(s)})\): add to \(c\)
- if no cluster matched: new cluster with \(\mathbf{r} = \mathbf{y}^{(s)}\)
- \(p_c \leftarrow n_c / S\); \(\text{SE} \leftarrow -\sum_c p_c \ln p_c\) // (156.11)
- answer \(\leftarrow\) largest cluster; agreement \(\leftarrow \max_c p_c\) // self-consistency
- if \(\text{SE} > \tau\): flag “not sure”; else return the answer
How to read Figure 5
Press Play or Step. The highlighted line of the algorithm box runs in this step; the box next to it lists the entailment checks and the formula with real numbers. “Paraphrases only”: every sample joins the first cluster, so string entropy climbs to 2.164 nats while semantic entropy stays 0, and the answer is not flagged. “Mostly agree”: sample 5 (“Sydney.”) fails the check against “Canberra.” and opens a second cluster; at the end 8 samples say Canberra and 2 say Sydney, SE is 0.500 nats and the agreement is 80%. “Scattered”: 8 clusters, SE 2.025 nats, flagged. Move the sample slider down to 2: with so few samples the estimate is noisy, and real systems use about 5 to 10. In “Random draw”, press “New samples” several times: with \(S = 10\) the SE changes from draw to draw, because it is an estimate from few samples. Each extra sample is a full generation, so this detector multiplies the cost of an answer.
- input: question \(x\), model, number of samples \(S\), entailment checker, threshold \(\tau\)
- \(\mathbf{y}^{(0)} \leftarrow\) the main answer (greedy or low temperature)
- for \(s = 1 \dots S\): \(\mathbf{y}^{(s)} \leftarrow\) sample at temperature about 1 // diversity is needed
- clusters \(\leftarrow\) [ ]
- for each sample \(\mathbf{y}\): put it in the first cluster whose representative \(\mathbf{r}\) satisfies entails(\(\mathbf{y}, \mathbf{r}\)) and entails(\(\mathbf{r}, \mathbf{y}\)); else start a new cluster
- \(p_c \leftarrow n_c / S\) for each cluster; \(\text{SE} \leftarrow -\sum_c p_c \log p_c\) // formula (156.11)
- if \(\text{SE} \gt \tau\): return “not sure” (abstain, retrieve, or ask a human)
- else: return \(\mathbf{y}^{(0)}\) with agreement \(\max_c p_c\)
Other signals: verbal confidence, P(True) and probes
- Verbal confidence: ask the model “how confident are you, 0 to 100%?”. Cheap and black-box, but the numbers tend to cluster on round values and are often overconfident after RLHF. [Likely] Calibrate them like any score.
- P(True) and P(IK): Kadavath et al. ask the model whether a proposed answer is true and read the probability of “True”; it works better when the model first sees several of its own samples. They also train a head that predicts “I know the answer” (P(IK)) from the question alone. [Certain]
- Probes on hidden states: a small linear classifier trained on the model’s internal activations can often predict whether an answer will be correct, which suggests the model “knows more than it says”. This is a topic for session 157. [Likely]
Checking against sources
Consistency checks catch answers the model is unsure about. They cannot catch confident errors. For those you need an outside reference: retrieved documents, a database, a tool. This is the strongest detection, and it is how production systems do it.
- Retrieval check (faithfulness): split the answer into claims; for each claim, ask an NLI model or an LLM judge “is this claim supported by these passages?”. Unsupported claims are removed or marked. This is the groundedness metric of RAG evaluation (session 153).
- Claim-level fact checking: FActScore splits a long answer into atomic facts and checks each against a knowledge source (Wikipedia in the paper); the score is the fraction supported. [Certain]
- Chain-of-Verification (CoVe): the model (1) drafts an answer, (2) writes verification questions about its own claims, (3) answers them independently, without seeing the draft, so it does not copy its own error, and (4) writes a corrected final answer. [Certain]
- Citations: make the model cite a passage id for each sentence, then check automatically that the cited passage exists and supports the sentence. A citation that points nowhere is itself a hallucination.
- input: question \(x\), retriever, generator, support checker, confidence threshold \(t\)
- passages \(\leftarrow\) retrieve\((x)\); if the best retrieval score is low: return “I could not find this in the documents”
- draft \(\leftarrow\) generate\((x\), passages, “answer only from the passages, cite [id] after each sentence, say not found if they do not contain the answer”)
- for each sentence \(s\) in draft: supported\((s) \leftarrow\) check\((s\), passage[cited id]\()\) // NLI or LLM judge
- drop sentences that are not supported; if nothing is left: return “not found”
- \(p \leftarrow\) calibrated confidence (for example agreement over a few samples, recalibrated on labeled data)
- if \(p \le t\): return “not sure” + the passages for the user to read // formula (156.3) with \(t = k/(1+k)\)
- return the supported sentences with their citations
Abstaining: selective prediction
Once we have a calibrated confidence, we can let the model answer only when the confidence is above a threshold \(t\). This is selective prediction. Two numbers describe it:
Coverage is the fraction of questions the model answers. Selective risk is the error rate among the answered ones. Raising \(t\) lowers both: fewer answers, but fewer of them wrong. The best \(t\) depends on the cost of a wrong answer, through formula (156.3). If the confidences are calibrated, the theory says \(t = k/(1 + k)\) maximizes the score; Figure 6 lets you check this on the recalibrated data from Figure 4.
How to read Figure 6
At \(t = 0\) the model answers everything: 55.1% accuracy. With \(k = 1\) that scores \(0.551 - 0.449 = 0.102\) per question. Raise \(t\) to 0.5: it answers 57.2% of questions at 65.9% accuracy, and the score rises to 0.182. With \(k = 2\), answering everything scores \(-0.347\) (worse than always saying “I don’t know”), and the best region moves to around \(t = 0.6\) to \(0.67\). The readout shows the best threshold found on a grid next to the theory value \(k/(1+k)\). They are close because the confidences are calibrated; on uncalibrated confidences they would not be.
Product designs that reduce hallucination
Detection is half the job. The other half is designing the product so that hallucinations are rare and cheap when they happen.
| Design | What it does | Cost or limit |
|---|---|---|
| Ground answers in retrieved documents, with citations | Turns factuality into faithfulness, which can be checked automatically; the user can click the source | Retrieval misses cause “not found” or wrong context (session 152) |
| Allow and reward “I don’t know” | Instruction plus examples of abstention in SFT data; grade with a penalty \(k\) so guessing does not win | Too many abstentions annoy users; tune \(t\) per use case |
| Use tools for exact facts | Dates, prices, math, account data come from a calculator, database or API, not from the weights (session 154) | Tool errors and wrong tool arguments are new failure modes |
| Constrain the output | Pick from a fixed list (product ids, ICD codes) with structured output; the model cannot invent an id | Only works when the answer space is known |
| Verification pass | CoVe or a separate checker model before the answer is shown | \(2\times\) to \(5\times\) more tokens and latency |
| Lower temperature for factual tasks | Fewer low-probability tokens; greedy or \(T\) near 0 for extraction and QA | Does not fix wrong knowledge; less diversity |
| Honest UI | Show confidence or “please check”, show sources, highlight unsupported sentences | Users ignore warnings that appear on every answer |
| Human review for high stakes | Route low-confidence or high-risk answers (legal, medical, payments) to a person | Human time; needs a good confidence signal to route well |
| Monitor in production | Sample live answers, measure groundedness and the abstention rate, track them per release (session 155) | Needs labeled data or a trusted judge |
Putting it together in code
The PyTorch code below builds an overconfident four-choice “model”, measures ECE, fits a temperature by NLL with LBFGS, computes semantic entropy from cluster ids, and applies the abstention rule. It is deterministic; the printed values are shown in the comments.
import torch import torch.nn.functional as F torch.manual_seed(0) # 1) A toy 4-choice quiz. The "true" logits give the real chance of each answer. # The model reports logits 2.5 times too large: it is overconfident. N, K = 5000, 4 z_true = 1.2 * torch.randn(N, K) labels = torch.multinomial(F.softmax(z_true, -1), 1).squeeze(1) logits = 2.5 * z_true def ece(probs, labels, bins=10): # formula (156.6) conf, pred = probs.max(-1) correct = (pred == labels).float() idx = torch.clamp((conf * bins).long(), max=bins - 1) total = 0.0 for b in range(bins): m = idx == b if m.any(): total += m.float().mean() * (conf[m].mean() - correct[m].mean()).abs() return float(total) # 2) Temperature scaling, formula (156.8): one number T, fitted on held-out data. def fit_temperature(logits, labels): log_t = torch.zeros(1, requires_grad=True) # T = exp(log_t) stays positive opt = torch.optim.LBFGS([log_t], lr=0.1, max_iter=200) def closure(): opt.zero_grad() loss = F.cross_entropy(logits / log_t.exp(), labels) loss.backward() return loss opt.step(closure) return float(log_t.detach().exp()) val, test = slice(0, 2500), slice(2500, N) T = fit_temperature(logits[val], labels[val]) p1 = F.softmax(logits[test], -1) pT = F.softmax(logits[test] / T, -1) print(f"T = {T:.2f}") # T = 2.62 print(f"accuracy {(p1.argmax(-1) == labels[test]).float().mean():.3f}") # 0.551, same for any T print(f"ECE before {ece(p1, labels[test]):.3f} after {ece(pT, labels[test]):.3f}") # 0.228 -> 0.023 # 3) Semantic entropy, formula (156.11), from cluster ids of S sampled answers. def semantic_entropy(cluster_ids): counts = torch.bincount(torch.tensor(cluster_ids)).float() p = counts[counts > 0] / counts.sum() return float(-(p * p.log()).sum()) + 0.0 print(f"{semantic_entropy([0]*10):.3f}") # 0.000 paraphrases print(f"{semantic_entropy([0]*8 + [1]*2):.3f}") # 0.500 8 vs 2 print(f"{semantic_entropy([0,1,0,2,3,4,1,5,6,7]):.3f}") # 2.025 scattered # 4) Abstain unless confident: t = k / (1 + k), formula (156.3). k = 2.0 t = k / (1 + k) conf, pred = pT.max(-1) ok = (pred == labels[test]).float() answer = conf > t score = (ok * answer).sum() - k * ((1 - ok) * answer).sum() print(f"coverage {answer.float().mean():.3f} accuracy when answering {ok[answer].mean():.3f}") # 0.233, 0.794 print(f"score {score / len(ok):.3f} answer-all {(ok.sum() - k * (1 - ok).sum()) / len(ok):.3f}") # 0.089 vs -0.346
Note that the fitted \(T\) here is 2.62, not exactly 2.5: it is estimated from 2,500 noisy labels. In a real LLM system, the “logits” would be the model’s scores for the answer options (or a confidence feature such as agreement), and the labels would come from a small human-checked set.
Where this is used
| Application | What goes wrong | Main tools |
|---|---|---|
| Enterprise QA and support bots | Invented policies, prices, steps | RAG with citations, support check, “not found” answer, abstention threshold |
| Legal and medical assistants | Invented cases, drugs, doses | Retrieval from trusted sources only, claim checks, human review, high \(k\) |
| Summarization | Unfaithful details added to a summary | NLI-based faithfulness check against the source (session 147) |
| Code assistants | Calls to functions or packages that do not exist | Run the code, type-check, look up the real API, tests |
| Classification with LLMs | Confidences that are too high to threshold | Temperature scaling on a labeled set, reliability diagrams (session 144) |
| Image captioning and VQA | Objects that are not in the image | POPE-style yes/no probes, grounding boxes, refusing when unsure |
| Video QA and Video-RAG | Events or timestamps that are not in the video | Retrieve clips first, cite timestamps, check the answer against the retrieved frames; key-moment detection answers must point to a real segment |
| Agents | Invented tool results, wrong arguments | Use only real tool outputs, verify before acting, ask the user when uncertain (session 154) |
Common mistakes
- Treating raw token probabilities of a chat model as calibrated. Post-training tends to make them overconfident. Measure a reliability diagram on labeled data and recalibrate first.
- Fitting the temperature on the test set. \(T\) must be fitted on a held-out validation set, then evaluated on a separate test set, or the ECE you report is too optimistic.
- Using token entropy for free-form answers. Paraphrases make it high even when the model is sure. Cluster by meaning (semantic entropy) or check agreement.
- Believing consistency means truth. A model can repeat the same wrong fact in every sample. Consistency checks catch guessing, not confident errors; use sources for those.
- Grading with 0/1 accuracy only. This rewards guessing and hides the abstention behavior you want. Report accuracy, abstention rate and a penalized score.
- Fine-tuning new facts into the model to stop hallucination. It can increase hallucination; put facts in retrieval.
- Using one global threshold for every use case. The right \(t\) comes from the cost of a wrong answer, which differs between a chit-chat bot and a dosage calculator.
- Reporting ECE with very few examples per bin. With 10 bins and 200 examples, bin accuracies are noisy. Report the number of examples and a proper score (NLL or Brier) too.
Interview questions
Why do LLMs hallucinate? Give at least three causes.basic
What is the difference between a factuality and a faithfulness hallucination?basic
Define calibration and ECE. Can a model be accurate but badly calibrated?core
How does temperature scaling work, and why does it not change accuracy?core
Derive the confidence threshold for abstaining when a wrong answer costs k points.core
Why does plain sequence entropy fail as an uncertainty measure for free-form answers, and what does semantic entropy do instead?core
Which hallucinations can sampling-based detection (self-consistency, SelfCheckGPT, semantic entropy) not catch?advanced
Design a question-answering assistant for a bank that must not invent facts.advanced
What did Kadavath et al. find about whether LLMs know what they know?advanced
Why can RLHF make calibration worse, and what can you do about it?advanced
Why is ECE alone not enough to compare two calibration methods?core
Exercises
- The model is 80% confident and a wrong answer costs \(k = 3\). Should it answer? What is the threshold?
- Half of the predictions have confidence 0.9 and accuracy 0.6; the other half have confidence 0.6 and accuracy 0.55. Compute the ECE.
- Compute the semantic entropy of 10 samples that split 5 and 5 between two meanings, and of 10 samples in one meaning.
- Each claim in a long answer is wrong with probability 1%. What is the chance that an answer with 50 claims has at least one error?
- Logits \((3, 1, 0, 0)\). Compute the top probability at \(T = 1\) and \(T = 2\). Does the predicted class change?
- In Figure 6, set \(k = 2\) and find the threshold with the best score. Compare it with \(k/(1+k)\).
- Explain why a hallucination detector based on self-consistency would not flag “The Great Wall of China is visible from space with the naked eye” if the model believes it.
Answerscheck
Key takeaways
- Hallucinations are fluent text not supported by facts (factuality) or by the given input (faithfulness). They come from the next-token objective (156.1), facts seen once (156.2), grading that rewards guessing (156.3), and decoding that compounds errors (156.4).
- With a wrong-answer penalty \(k\), answer only when the calibrated confidence is above \(t = k/(1+k)\); standard 0/1 grading (\(k = 0\)) always rewards guessing.
- Calibration (156.5) is measured with reliability diagrams, ECE (156.6) and proper scores such as Brier (156.7). Temperature scaling (156.8) fixes overconfidence with one number and does not change accuracy.
- For free-form answers, token entropy (156.10) is fooled by paraphrases; semantic entropy (156.11) clusters samples by meaning and detects confabulations, but not consistent errors.
- Confident errors need outside checks: retrieval support, claim-level fact checks, CoVe, tools. Products combine grounding, abstention (156.12), tools, verification and honest UI.
Key formulas
References for this session
- Huang et al. (2023). A Survey on Hallucination in Large Language Models
- Kadavath et al. (2022). Language Models (Mostly) Know What They Know
- Guo et al. (2017). On Calibration of Modern Neural Networks
- Kalai, Nachum, Vempala & Zhang (2025). Why Language Models Hallucinate
- Kalai & Vempala (2023). Calibrated Language Models Must Hallucinate
- Kuhn, Gal & Farquhar (2023). Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
- Farquhar, Kossen, Kuhn & Gal (2024). Detecting hallucinations in large language models using semantic entropy (Nature)
- Manakul, Liusie & Gales (2023). SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
- Dhuliawala et al. (2023). Chain-of-Verification Reduces Hallucination in Large Language Models
- Lin, Hilton & Evans (2021). TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Gekhman et al. (2024). Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
- OpenAI (2023). GPT-4 Technical Report
- Wang et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Min et al. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models