LLM & VLM courseBlog
147Track 6: LLM applications · session 5 of 20

Summarization

A summary keeps what matters in a text and drops the rest. This session compares extractive summaries (copy the best sentences) with abstractive ones (write new text), encoder–decoder models such as BART and T5 with decoder-only LLMs, and the three ways to summarize documents that do not fit in the context window. It ends with the hardest part: making sure the summary says only what the source says.

Time: 3 hours (reading 2 hours, exercises 1 hour)Level: BeginnerBefore this: Sessions 24 and 79 help; Session 149 goes deeper into ROUGE

What you will learn

  • The difference between extractive and abstractive summaries, and why Lead-3 is a strong baseline for news
  • How to pick sentences with MMR (relevance minus redundancy) and how oracle labels are built for training
  • How BART, T5 and PEGASUS are trained for summarization, and how decoder-only LLMs do it with a prompt or with human feedback
  • How beam search length penalty and minimum length control summary length
  • Map-reduce, refine and hierarchical summaries for long documents: number of calls, tokens and sequential depth
  • Intrinsic and extrinsic hallucination, and how to measure faithfulness with atomic facts (FActScore) and NLI (SummaC)

Why this matters

People read less than they are asked to read. Meeting notes, support tickets, legal contracts, research papers, news, chat threads and video transcripts all pile up. A summary is a shorter text that keeps the important content of a longer one. Summarization is one of the most used LLM features in real products, and one of the easiest to get subtly wrong.

The task looks easy because any LLM will happily write a fluent paragraph. The hard parts are hidden: deciding what is important, fitting a 300-page document into a model that reads 8,000 tokens at a time, and, most of all, making sure the summary does not say things the source never said. A summary that is fluent but wrong is worse than no summary, because the reader trusts it and will not check the source.

This session builds the topic in four steps. First, the two families of summaries: extractive (copy sentences) and abstractive (write new text). Second, the models: encoder–decoders such as BART and T5, and decoder-only LLMs. Third, documents that do not fit in one call: map-reduce, refine and hierarchical summaries. Fourth, faithfulness: how to check that a summary is supported by its source. The previous session (session 146) pulled exact fields out of text; here the output is free text, which makes checking much harder.

What a summary is: extractive vs abstractive

Let the source document be \(x\) and the summary be \(y\). A simple first number is the compression ratio, the length of the summary divided by the length of the source (in words or tokens):

\[ r = \frac{|y|}{|x|} \tag{147.1} \]

A one-line headline of a news story may have \(r\) around 0.05; an executive summary of a report may have \(r\) around 0.1 to 0.3. Lower \(r\) means more must be dropped, so the choices about what matters get harder.

There are two ways to build \(y\):

  • Extractive summarization selects whole sentences (or phrases) from \(x\) and copies them. Every sentence in the summary appears in the source, so the summary cannot invent facts inside a sentence. But it can be choppy, repeat itself, and break references (“He said…” without saying who “he” is).
  • Abstractive summarization generates new text, word by word, like a human writer. It can merge facts from several sentences, rephrase and shorten. But a generator can also produce words that are not supported by the source: this is hallucination.

Two classic news datasets show the difference. CNN/DailyMail pairs news articles with a few bullet-point highlights written by journalists; the highlights reuse many words of the article, so extractive methods do well. XSum pairs BBC articles with a single-sentence summary that often uses new words; extractive methods do badly. The simplest extractive baseline, Lead-3, just copies the first three sentences. From the BART paper’s table, Lead-3 gets ROUGE-1 of 40.42 on CNN/DailyMail but only 16.30 on XSum. [Certain] (ROUGE-1 measures word overlap with a human reference summary; formula 147.9 below. Higher is better; scores are often written ×100.) News writers put the main fact first, so Lead-3 is a strong baseline for news and a weak one for almost everything else.

Extractive summarization: choose the sentences

Split \(x\) into sentences \(s_1, \dots, s_n\). An extractive summary is a set \(S\) of sentence indices. We want the set that covers the most important content without going over a length budget \(L\):

\[ S^{*} = \argmax_{S \subseteq \{1,\dots,n\}} \ \operatorname{score}(S) \quad \text{subject to} \quad \sum_{i \in S} |s_i| \le L \tag{147.2} \]

This is a budgeted selection problem; with a general score it is hard to solve exactly, so almost everyone uses a greedy method: add the best sentence, then the best next one, and so on. Two things make a sentence good: it is relevant (it talks about what the document is about), and it is not redundant (it does not repeat a sentence already chosen). Maximal marginal relevance (MMR) combines the two:

\[ i^{*} = \argmax_{i \notin S} \Big[\, \lambda \,\operatorname{rel}(s_i) \;-\; (1 - \lambda) \max_{j \in S} \operatorname{sim}(s_i, s_j) \Big] \tag{147.3} \]
Read the symbols
\(\operatorname{rel}(s_i)\)
how relevant sentence \(i\) is to the whole document; here the cosine similarity between its word-count vector and the word-count vector of the whole document (the “centroid”)
\(\operatorname{sim}(s_i, s_j)\)
cosine similarity between two sentences; high means they say similar things
\(\lambda\)
a number between 0 and 1. With \(\lambda = 1\) only relevance counts; smaller \(\lambda\) punishes repetition more
\(S\)
the sentences already chosen; the max term is 0 when \(S\) is empty

In modern systems, the word-count vectors are replaced by sentence embeddings (session 150) and the relevance may be the similarity to a user’s question, but the formula is the same. Figure 1 runs MMR, Lead-\(k\) and an oracle (explained below) on a short news story.

    Figure 1. Three extractive summarizers on an eight-sentence story (119 words). Bars show each sentence’s relevance to the document centroid. Chosen sentences are highlighted, with their pick order. The readout compares the summary with a 46-word human reference using ROUGE-1 and ROUGE-2 F1.
    How to read Figure 1

    With MMR, sentence 4 is always picked first: it has the word “bridge” twice, and “bridge” is the most frequent word in the document, so the centroid favors it. That is a real weakness of frequency-based relevance: repetition looks like importance. Now switch to Lead-k: the first three sentences get a higher ROUGE-1 (0.644) than MMR (0.412), because the writer put the key facts first. The greedy oracle shows the best you could do by copying sentences, but it uses the reference, so it can only be used to make training labels, never at test time.

    Worked example: the second MMR pick

    Sentence 4 has the highest relevance, \(\operatorname{rel} = 0.544\), so it is picked first. For the second pick with \(\lambda = 0.7\), each sentence gets \(0.7\,\operatorname{rel}(s_i) - 0.3\,\operatorname{sim}(s_i, s_4)\):

    • Sentence 2: \(0.7 \times 0.507 - 0.3 \times 0.154 = 0.309\)
    • Sentence 5: \(0.7 \times 0.474 - 0.3 \times 0.161 = 0.283\)
    • Sentence 1: \(0.7 \times 0.462 - 0.3 \times 0.178 = 0.270\)

    Sentence 2 wins. Sentence 1, which states the main event (the bridge closes on March 3), loses partly because it shares words with sentence 4 and partly because it is short. The result, ROUGE-1 of 0.412 for \(k = 3\), is far below Lead-3. Unsupervised scoring does not know what a reader needs.

    Oracle labels and supervised extractors

    Better extractors are trained. A neural encoder (for example BERT) reads the document and predicts, for each sentence, “keep” or “drop”. But datasets contain abstractive reference summaries, not keep/drop labels. The standard trick is the greedy oracle: repeatedly add the sentence that most increases ROUGE against the reference, and stop when no sentence helps. The chosen sentences become the “keep” labels. In Figure 1, the oracle with \(k = 3\) picks sentences 1, 6 and 3, and reaches ROUGE-1 of 0.667 and ROUGE-2 of 0.390.

    Algorithm 1: greedy MMR extraction
    1. input: sentences \(s_1..s_n\), relevance \(\operatorname{rel}\), similarity \(\operatorname{sim}\), budget \(k\) sentences (or \(L\) words), weight \(\lambda\)
    2. \(S \leftarrow \emptyset\)
    3. while \(|S| \lt k\) and some sentence still fits the budget:
    4.    for each \(i \notin S\): \(\text{score}_i \leftarrow \lambda \operatorname{rel}(s_i) - (1-\lambda)\max_{j\in S}\operatorname{sim}(s_i,s_j)\) // formula (147.3)
    5.    \(S \leftarrow S \cup \{\argmax_i \text{score}_i\}\)
    6. return the sentences in \(S\), sorted by their position in the document // reading order, not pick order

    Abstractive summarization with encoder–decoders

    An abstractive model generates \(y = (y_1, \dots, y_m)\) one token at a time, each token conditioned on the source and on the tokens already written. It is trained on (document, reference summary) pairs with the usual next-token cross-entropy:

    \[ \mathcal{L}(\theta) = -\sum_{t=1}^{m} \log p_\theta\!\left(y_t \mid y_{\lt t},\, x\right) \tag{147.4} \]
    Read the symbols
    \(x\)
    the source document (the input tokens)
    \(y_t\)
    the \(t\)-th token of the reference summary; \(y_{\lt t}\) are the tokens before it (teacher forcing: the true previous tokens are given during training)
    \(p_\theta\)
    the model’s next-token distribution, with weights \(\theta\)

    Before transformers, this was an RNN encoder–decoder with attention, and its big weakness was getting rare words and names wrong. The pointer-generator network (See et al., 2017) added a switch that lets the decoder either generate a word from the vocabulary or copy a word from the source through the attention weights. Copying is a large part of good summarization: names, numbers and dates should be copied, not re-generated. Transformers with cross-attention learn to copy well without a special switch. [Likely]

    The encoder–decoder transformer fits the task naturally: a bidirectional encoder reads the whole document (every token sees every other token), and a causal decoder writes the summary while looking at the encoder output through cross-attention at every layer (session 24, session 20). Three pretrained models defined this era:

    • BART (Lewis et al., 2019) is pretrained as a denoising autoencoder: corrupt a document, then rebuild the original. The final large model masks 30% of tokens using spans (text infilling) and shuffles the order of all sentences. [Certain] Rebuilding a whole document from a damaged copy is close to summarization: the decoder learns to write long, fluent text that stays close to the input. BART-large has 12 encoder layers, 12 decoder layers and hidden size 1024, and reads at most 1,024 tokens. [Certain]
    • T5 (Raffel et al., 2019) casts every task as text-to-text. To summarize, you put the prefix summarize: in front of the article. [Certain]
    • PEGASUS (Zhang et al., 2019) designs the pretraining task for summarization: remove important sentences from a document and make the model generate them from the rest (“gap-sentence generation”). It reported state-of-the-art ROUGE on all 12 summarization datasets it was tested on, at the time. [Certain]
    System (from the BART paper, Table 3)CNN/DM R-1 / R-2 / R-LXSum R-1 / R-2 / R-L
    Lead-3 (copy first 3 sentences)40.42 / 17.62 / 36.6716.30 / 1.60 / 11.95
    Pointer-generator (PTGEN)36.44 / 15.66 / 33.4229.70 / 9.21 / 23.24
    BertSumExtAbs42.13 / 19.60 / 39.1838.81 / 16.50 / 31.27
    BART44.16 / 21.28 / 40.9045.14 / 22.27 / 37.25

    Read the table carefully. On CNN/DM, the strong pretrained model beats Lead-3 by under 4 ROUGE-1 points. On XSum it beats Lead-3 by almost 29 points, because XSum needs real rewriting. [Certain] Which dataset you test on decides whether “abstractive” matters.

    Decoding a summary: beam search and length control

    Training teaches \(p_\theta\); at test time we still need to choose the output. Summarization models are usually decoded with beam search: keep the \(B\) best partial summaries at each step, extend each one by every possible next token, and keep the \(B\) best again (session 79). The “best” is measured by the total log-probability \(\log p(y \mid x) = \sum_t \log p(y_t \mid y_{\lt t}, x)\). Every token adds a negative number, so longer outputs always have lower total log-probability, and plain beam search prefers summaries that are too short. The fix is a length penalty that divides by a power of the length:

    \[ \operatorname{score}(y) = \frac{\log p_\theta(y \mid x)}{|y|^{\alpha}} \tag{147.5} \]
    Read the symbols
    \(|y|\)
    the number of tokens in the candidate summary
    \(\alpha\)
    the length penalty exponent. \(\alpha = 0\): plain total log-probability (favors short). \(\alpha = 1\): average log-probability per token. \(\alpha \gt 1\): favors long outputs even more

    This is the form Hugging Face generate() uses for its length_penalty argument. [Likely] The published config of facebook/bart-large-cnn sets num_beams=4, length_penalty=2.0, min_length=56, max_length=142 and no_repeat_ngram_size=3. [Certain] Each one is a length or repetition control: min_length forbids the end-of-sequence token before 56 tokens, and no_repeat_ngram_size=3 forbids repeating any 3-token sequence, a cheap cure for the “the bridge, the bridge, the bridge” loops that summarizers fall into. Figure 2 shows how \(\alpha\) and the minimum length decide which of four finished beams wins.

    \(\operatorname{score}(y) = \log p_\theta(y \mid x) \,/\, |y|^{\alpha}\) (bar length = how negative)
    Figure 2. Four finished beam candidates (toy numbers): H1 has 12 tokens and total log-probability −9.0, H2 has 30 and −16.5, H3 has 56 and −27.0, H4 has 90 and −45.0. The bars show formula (147.5); the winner is the one closest to zero. Candidates shorter than the minimum length are not allowed.
    How to read Figure 2

    At \(\alpha = 0\) the 12-token H1 wins: the shortest output has the least negative total. Move \(\alpha\) up: H2 wins from 0.7, H3 from 0.8, and the longest one, H4, from 1.1. At \(\alpha = 1\) the winner is the one with the best average per token (H3, −0.482 per token). With BART-CNN’s \(\alpha = 2\) the longest candidate wins easily, and max_length is then what stops the output. Set the minimum length to 56 and only H3 and H4 remain.

    Decoder-only LLMs as summarizers

    A decoder-only model has no separate encoder. The document and the instruction go into the same token sequence, and the summary is simply the continuation. Causal attention means each source token only sees tokens before it, but every summary token sees the whole source, so in practice this works very well. GPT-2 already showed that adding TL;DR: after an article makes a language model produce a rough summary with no training for the task. [Certain] Today you write an instruction: “Summarize the meeting below in five bullet points for an engineer who missed it. Use only facts from the transcript.” (session 143).

    Two results changed the field:

    • Learning to summarize from human feedback (Stiennon et al., 2020). They trained 1.3B and 6.7B models on 123,169 filtered Reddit posts (the TL;DR dataset), collected human comparisons between pairs of summaries, trained a reward model on them, and fine-tuned the summarizer with reinforcement learning (PPO). [Certain] The reward is the reward model’s score minus a penalty for moving too far from the supervised model:
      \[ R(x, y) = r_\phi(x, y) - \beta \log \frac{\pi_{\mathrm{RL}}(y \mid x)}{\pi_{\mathrm{SFT}}(y \mid x)} \tag{147.6} \]
      with \(\beta = 0.05\) in the main runs. [Certain] Even after controlling for length, people preferred the 6.7B model’s summaries over the human reference summaries about 65% of the time, and the paper showed that optimizing the learned reward gives summaries people like more than optimizing ROUGE. [Certain] This paper is the direct ancestor of the RLHF recipe used for chat models (session 67). The KL term matters: without it, the policy learns to exploit the reward model (reward hacking).
    • News summarization in the era of GPT-3 (Goyal, Li and Durrett, 2022). Human raters strongly preferred summaries from prompted GPT-3, with only a task description, over summaries from models fine-tuned on news datasets, while reference-based metrics such as ROUGE did not show this. [Certain] Part of the reason: the dataset references themselves are imperfect, so a model trained to imitate them inherits their problems.
    Encoder–decoder (BART, T5, PEGASUS)Decoder-only LLM (prompted or fine-tuned)
    Size in practiceUsually hundreds of millions of parameters (T5 goes up to 11B); runs on one small GPU or CPUBillions of parameters; usually a served API or a large GPU
    Input lengthOften 512 to 1,024 tokens (BART: 1,024)Tens of thousands to over a million tokens, depending on the model
    Training neededFine-tune on thousands of (document, summary) pairsOften none: an instruction is enough; fine-tune for style or cost
    Control of styleFixed by the training dataChange it in the prompt (bullets, audience, length, language)
    Typical use todayHigh-volume, narrow domain, low cost or on deviceGeneral purpose, long inputs, many formats

    A common production pattern is to use a large LLM to write summaries for a few thousand documents, check them, then fine-tune a small model on them (distillation) to serve millions of documents cheaply. [Likely] The small model is the same idea as Session 146’s “small fine-tuned model beats big prompted one”: narrow task, many requests.

    Long documents: map-reduce, refine and hierarchical summaries

    A book of 300,000 tokens does not fit in an 8,000-token model. Even if a model accepts 300,000 tokens, it may use the middle of the input poorly: Liu et al. found that models use information at the start and end of a long context better than information in the middle (“lost in the middle”). [Certain] And one huge call is slow and expensive. There are four strategies.

    1. Stuff: put the whole document in one prompt. Best quality when it fits, but only possible if \(N \le C\), where \(N\) is the document length and \(C\) the tokens one call can take.
    2. Map-reduce: split the document into chunks of \(C\) tokens. Map: summarize each chunk independently, into about \(s\) tokens. Reduce: concatenate the chunk summaries and summarize them. All map calls can run in parallel.
    3. Hierarchical (recursive map-reduce): if the concatenated summaries are still longer than \(C\), group them, summarize each group, and repeat. This builds a tree of summaries: leaves are chunk summaries, the root is the final summary. The middle levels are useful by themselves, for example as chapter summaries or as retrieval units for RAG (session 153).
    4. Refine: read the chunks in order. Summarize the first; then, for each next chunk, give the model the running summary plus the new chunk and ask it to update the summary. Each call sees the story so far, but the calls must run one after another.

    With chunk size \(C\) and \(s\) tokens per summary, one reduce call can merge \(g = \lfloor C / s \rfloor\) summaries. Map-reduce then needs

    \[ k_0 = \left\lceil \frac{N}{C} \right\rceil, \qquad k_{\ell+1} = \left\lceil \frac{k_\ell}{g} \right\rceil \ \text{ until } k = 1, \qquad \text{calls} = \sum_\ell k_\ell, \qquad \text{depth} = 1 + \lceil \log_g k_0 \rceil \tag{147.7} \]

    The depth is the number of rounds that must wait for each other; with enough parallel workers, latency grows with depth, not with the number of calls. Refine uses chunks of \(C - s\) tokens, so that a chunk plus the running summary fits in one call:

    \[ \text{calls}_{\text{refine}} = \text{depth}_{\text{refine}} = \left\lceil \frac{N}{C - s} \right\rceil \tag{147.8} \]
    Read the symbols
    \(N\)
    document length in tokens
    \(C\)
    input tokens one call can take (context window minus room for the instruction and the output)
    \(s\)
    length of each intermediate summary
    \(k_\ell\)
    number of calls at level \(\ell\) of the tree; level 0 is the map step
    \(g\)
    how many summaries one reduce call can read
    Figure 3. How many LLM calls each strategy makes, how many input and output tokens it processes, and how many rounds must run one after another. Each box is one call. Rows of the map-reduce tree are levels; boxes in one row can run in parallel. Very long rows are cut and the count is written at the end.
    How to read Figure 3

    Start with the default: 300,000 tokens, 8,000 per call, 500-token summaries. Map-reduce needs 38 + 3 + 1 = 42 calls but only 3 rounds. Refine needs 40 calls in a chain of 40 rounds. Both read about the same number of input tokens, so the cost is similar; the latency is not. Now raise \(C\) to 32,000: map-reduce needs only 2 levels. Raise \(s\) so that one reduce call can merge only a few summaries and watch the tree grow taller. Stuff is only possible when \(N \le C\).

    Worked example: a 300,000-token book with an 8,000-token budget

    Map-reduce. \(k_0 = \lceil 300{,}000 / 8{,}000 \rceil = 38\) map calls. Each reduce call can read \(g = \lfloor 8{,}000 / 500 \rfloor = 16\) summaries, so level 1 has \(\lceil 38/16 \rceil = 3\) calls and level 2 has 1. Total: 42 calls, depth 3. Input tokens: \(300{,}000\) for the map step, plus \(38 \times 500 = 19{,}000\) and \(3 \times 500 = 1{,}500\) for the reduce steps: 320,500. Output tokens: \(42 \times 500 = 21{,}000\).

    Refine. Chunks of \(8{,}000 - 500 = 7{,}500\) tokens give \(\lceil 300{,}000/7{,}500 \rceil = 40\) calls, all sequential. Input: \(300{,}000 + 39 \times 500 = 319{,}500\) tokens. Output: \(40 \times 500 = 20{,}000\).

    Same cost, very different latency: if one call takes \(t\) seconds, map-reduce finishes in about \(3t\) with 38 parallel workers, refine in about \(40t\).

    Quality differs too. Map-reduce chunks do not see each other, so a fact explained in chunk 3 and used in chunk 9 can be lost, and each chunk summary weighs its chunk equally even if it is unimportant. Refine sees the story so far, but early content tends to fade as it is rewritten 40 times, and errors made early are carried forward. [Likely] Good practice: split at natural boundaries (sections, chapters, speaker turns), overlap chunks slightly, ask the map step to keep names, numbers and dates exactly, and keep a pointer from each summary sentence to its chunk so it can be checked.

    Algorithm 2: hierarchical (recursive map-reduce) summarization
    1. input: document \(x\) of \(N\) tokens, per-call budget \(C\), summary length \(s\), instruction \(P\)
    2. if \(N \le C\): return LLM(\(P\), \(x\)) // stuff
    3. chunks \(\leftarrow\) split \(x\) at section or paragraph boundaries into pieces of at most \(C\) tokens
    4. parts \(\leftarrow\) [LLM(\(P_{\text{map}}\), c, max \(s\) tokens) for c in chunks] // parallel
    5. while len(parts) \(\gt 1\):
    6.    groups \(\leftarrow\) pack consecutive parts into groups of total length \(\le C\) // \(g = \lfloor C/s \rfloor\) each
    7.    parts \(\leftarrow\) [LLM(\(P_{\text{reduce}}\), join(gr)) for gr in groups] // one level of formula (147.7)
    8. return parts[0], and keep every level of the tree for checking and retrieval
    Algorithm 3: refine summarization
    1. input: document \(x\), budget \(C\), summary length \(s\)
    2. chunks \(\leftarrow\) split \(x\) into pieces of at most \(C - s\) tokens
    3. summary \(\leftarrow\) LLM(\(P_{\text{first}}\), chunks[0])
    4. for c in chunks[1:]: summary \(\leftarrow\) LLM(\(P_{\text{refine}}\), summary, c) // “update the summary with the new text”; sequential
    5. return summary

    Faithfulness and hallucination

    A summary is faithful if everything it says is supported by the source. Maynez et al. (2020) studied this carefully and found that abstractive models “are highly prone to hallucinate content that is unfaithful to the input document”; their annotators found substantial hallucinated content in all model-generated summaries they studied. [Certain] They defined two kinds:

    • Intrinsic hallucination: the summary contradicts the source, usually by mixing up its parts. “The repairs will cost 2.4 million dollars” when the source says 4.2. Or the wrong person did the action.
    • Extrinsic hallucination: the summary adds information that is not in the source. It might even be true in the world (“the second closure since a flood in 2019”), but the reader cannot check it from the source, and often it is false.

    Why does it happen? The model has strong priors from pretraining and fills in what is “usually” true. Training references sometimes contain facts that are not in the article (common in XSum-style one-line summaries), so the model learns that adding facts is allowed. [Likely] And beam search rewards fluent, high-probability text, not checked text.

    Why ROUGE cannot see it

    ROUGE-N compares the \(n\)-grams (sequences of \(n\) words) of the candidate with those of a reference summary:

    \[ \text{R}_N = \frac{\sum_{g} \min\big(c_{\text{cand}}(g),\, c_{\text{ref}}(g)\big)}{\sum_{g} c_{\text{ref}}(g)}, \quad \text{P}_N = \frac{\sum_{g} \min\big(c_{\text{cand}}(g),\, c_{\text{ref}}(g)\big)}{\sum_{g} c_{\text{cand}}(g)}, \quad \text{F}_N = \frac{2\,\text{P}_N \text{R}_N}{\text{P}_N + \text{R}_N} \tag{147.9} \]

    Here \(c(g)\) counts how often \(n\)-gram \(g\) appears. ROUGE rewards word overlap. It does not know which words are facts. If the reference says “4.2 million” and the candidate says “2.4 million”, both have the unigrams “2”, “4” and “million”, so ROUGE-1 is exactly the same. Only two bigrams change. ROUGE also punishes a correct summary that uses other words. Session 149 covers ROUGE and its relatives in full.

    Checking facts: atomic facts, NLI and QA

    FActScore (Min et al., 2023) breaks a generated text into atomic facts, short statements with one piece of information each, and checks each against a knowledge source \(\mathcal{C}\). In the paper, \(\mathcal{C}\) is Wikipedia; for summarization, \(\mathcal{C}\) is the source document. The score of one output is the fraction of its atomic facts that are supported:

    \[ f(y) = \frac{1}{|\mathcal{A}_y|} \sum_{a \in \mathcal{A}_y} \mathbb{I}\big[\, a \text{ is supported by } \mathcal{C} \,\big] \tag{147.10} \]
    Read the symbols
    \(\mathcal{A}_y\)
    the list of atomic facts extracted from output \(y\) (by a person or by an LLM)
    \(\mathbb{I}[\cdot]\)
    1 if the statement is true, 0 if not
    \(\mathcal{C}\)
    the knowledge source used to check; the source document for summaries

    Both contradicted and unverifiable facts count as unsupported. FActScore is a precision: it does not reward covering the important content, and a summary that says almost nothing can score 100%. The paper notes this and reports the number of facts alongside the score; with human checking, ChatGPT reached 58.3% on its biography task, InstructGPT 42.5%, and the retrieval-based PerplexityAI 71.5%. [Certain] Pair FActScore with a coverage measure (does the summary contain the key points?).

    Two older, cheaper checks are still common. NLI-based checks use a natural language inference model (session 145) that gives the probability that a premise entails a hypothesis. SummaC-ZS (Laban et al., 2021) builds a matrix of entailment scores \(E_{ij}\) between document sentence \(i\) and summary sentence \(j\), takes the best supporting document sentence for each summary sentence, and averages:

    \[ \text{SummaC}_{\text{ZS}}(x, y) = \frac{1}{M} \sum_{j=1}^{M} \max_{i} E_{ij} \tag{147.11} \]

    where \(M\) is the number of summary sentences. In the paper’s example the column maxima are 0.98, 0.99 and 0.04, giving 0.67: one summary sentence is not supported anywhere. [Certain] QA-based checks (QAGS, Wang et al., 2020) generate questions from the summary, answer them from both the summary and the source, and compare the answers. And the most common check today is an LLM judge that reads source and summary and lists unsupported claims (session 155); G-Eval is one published recipe. LLM judges are themselves imperfect, so calibrate them against a few hundred human labels before trusting them. [Likely]

      supportedcontradicted (intrinsic)not in source (extrinsic)
      Figure 4. Three summaries of the bridge story from Figure 1. ROUGE compares each with the human reference; FActScore checks each atomic fact against the source. Click a fact to change its verdict (supported → contradicted → not in source), as a human checker or an LLM judge might.
      How to read Figure 4

      Summary A is fully faithful but rephrases, so its ROUGE-1 is the lowest (0.610). Summary B copies the source closely but swaps 4.2 into 2.4: ROUGE-1 0.813, one contradicted fact. Summary C has the highest ROUGE of all (0.826 and 0.778) and adds a flood that the source never mentions. The ROUGE ranking is C, B, A; the faithfulness ranking is A, then C and B. Click a supported fact in A to see how one wrong verdict by a judge moves the score by 1/7.

      Worked example: ROUGE-1 and FActScore for Summary B

      Summary B has 45 words; the reference has 46. They share 37 unigrams (counting repeats up to the smaller count). So \(\text{P}_1 = 37/45 = 0.822\), \(\text{R}_1 = 37/46 = 0.804\), and \(\text{F}_1 = 0.813\). For bigrams: 23 shared out of 44 and 45, \(\text{F}_2 = 0.517\). If we fix the number back to 4.2, ROUGE-1 stays 0.813 and ROUGE-2 rises only to 0.562.

      FActScore: B has 6 atomic facts; 5 are supported and “the repairs will cost 2.4 million dollars” is contradicted, so \(f = 5/6 = 0.833\). For C, 6 of 7 facts are supported: \(f = 0.857\). A gets \(7/7 = 1\). Note that FActScore treats B’s wrong number and C’s added flood the same; in a product, you might weigh a wrong number as worse.

      Algorithm 4: checking the faithfulness of a summary
      1. input: source \(x\) (split into passages), summary \(y\)
      2. facts \(\leftarrow\) LLM(“List every atomic fact in this summary, one per line”, \(y\))
      3. for each fact \(a\):
      4.    evidence \(\leftarrow\) the top passages of \(x\) for \(a\) // retrieval, or all of \(x\) if short
      5.    verdict\(_a \leftarrow\) judge(\(a\), evidence) \(\in\) {supported, contradicted, not in source} // NLI model or LLM judge
      6. score \(\leftarrow\) (number supported) / (number of facts) // formula (147.10)
      7. return score, the list of unsupported facts with evidence, and the number of facts // show the reader what failed

      Reducing hallucination

      • Ground the prompt: “use only information from the text”, “if something is unclear, say so”, and ask for quotes or source pointers next to each claim.
      • Clean the training data: drop reference summaries whose facts are not in the source before fine-tuning. [Likely]
      • Check and repair: run Algorithm 4, then ask the model to remove or fix unsupported sentences.
      • Keep numbers and names extractive: copy them from the source, never re-type them; a post-check that every number in the summary appears in the source is cheap and catches many intrinsic errors.
      • Tune toward faithfulness: preference training with faithful summaries as “chosen” (session 68).

      Session 156 covers hallucination in general; summaries are the easiest case to check because the source is right there.

      Summarization in PyTorch

      The code below runs with only PyTorch and no downloads. It builds an MMR extractor (formula 147.3) on word-count vectors, computes ROUGE-1 and ROUGE-2 (formula 147.9), runs recursive map-reduce (formula 147.7) with the extractor standing in for an LLM call, and computes a FActScore (formula 147.10). It is deterministic: no random numbers.

      import re, torch
      from collections import Counter
      
      STOP = set("the a an of to in on for and will is was be by that which it its "
                 "at from with this as are about each them their has have last".split())
      def words(s):   return re.findall(r"[a-z0-9]+", s.lower())
      def content(s): return [w for w in words(s) if w not in STOP]
      
      def count_matrix(sents):
          vocab = sorted({w for s in sents for w in content(s)})
          idx = {w: i for i, w in enumerate(vocab)}
          C = torch.zeros(len(sents), len(vocab))
          for r, s in enumerate(sents):
              for w in content(s):
                  C[r, idx[w]] += 1
          return C
      
      def mmr_select(sents, k, lam=0.7):
          """Greedy MMR, formula (147.3). Returns sentence indices in pick order."""
          C = count_matrix(sents)
          X = C / C.norm(dim=1, keepdim=True).clamp_min(1e-12)   # unit rows
          cen = C.sum(0); cen = cen / cen.norm()
          rel = X @ cen                                          # cosine to the centroid
          sim = X @ X.T                                          # sentence-sentence cosine
          picked = []
          for _ in range(k):
              red = sim[:, picked].max(1).values if picked else torch.zeros(len(sents))
              score = lam * rel - (1 - lam) * red
              score[picked] = -float("inf")
              picked.append(int(score.argmax()))
          return picked
      
      def ngrams(t, n): return Counter(tuple(t[i:i + n]) for i in range(len(t) - n + 1))
      def rouge_f(cand, ref, n):
          """ROUGE-N F1, formula (147.9): clipped n-gram overlap."""
          c, r = ngrams(words(cand), n), ngrams(words(ref), n)
          o = sum((c & r).values())
          if o == 0: return 0.0
          p, q = o / sum(c.values()), o / sum(r.values())
          return 2 * p * q / (p + q)
      
      DOC = ["Larkfield will close the Old Mill Bridge for repairs starting on March 3.",
             "Engineers found deep cracks in two of the bridge's steel supports during an inspection in January.",
             "The repairs will cost 4.2 million dollars and should take about five months.",
             "While the bridge is closed, buses will use the North Road bridge, adding ten minutes to each trip.",
             "Local shop owners near the bridge worry that fewer customers will reach them during the closure.",
             "The mayor promised small city grants to shops that lose business.",
             "The bridge was built in 1962 and carries about 18,000 cars a day.",
             "The city council approved the repair budget last week by a vote of seven to two."]
      REF = ("Larkfield will close the Old Mill Bridge from March 3 for five months of repairs costing "
             "4.2 million dollars after engineers found cracks in its steel supports. Buses will use the "
             "North Road bridge, and the city will give grants to shops that lose business.")
      
      # 1) extractive: Lead-3 vs MMR (lambda = 0.7)
      lead = " ".join(DOC[:3])
      pick = mmr_select(DOC, 3, lam=0.7)
      mmr = " ".join(DOC[i] for i in sorted(pick))           # keep document order
      print("MMR picks", [i + 1 for i in pick])                 # [4, 2, 5]
      for name, s in [("Lead-3", lead), ("MMR", mmr)]:
          print(name, round(rouge_f(s, REF, 1), 3), round(rouge_f(s, REF, 2), 3))  # Lead-3 0.644 0.318, MMR 0.412 0.189
      
      # 2) map-reduce with a stand-in summarizer, formula (147.7)
      def summarize(text, budget_words):          # stand-in for one LLM call
          sents = re.split(r"(?<=[.!?])\s+", text.strip())
          keep, used = [], 0
          for i in mmr_select(sents, len(sents)):  # best sentences first
              n = len(words(sents[i]))
              if used + n <= budget_words:
                  keep.append(i); used += n
          return " ".join(sents[i] for i in sorted(keep))
      
      def map_reduce(sents, chunk_words, summary_words):
          calls, level = [], sents
          while True:
              chunks, cur = [], []
              for s in level:                      # pack whole sentences into chunks
                  if cur and sum(len(words(x)) for x in cur) + len(words(s)) > chunk_words:
                      chunks.append(" ".join(cur)); cur = []
                  cur.append(s)
              chunks.append(" ".join(cur))
              parts = [summarize(c, summary_words) for c in chunks]
              calls.append(len(chunks))           # LLM calls at this level
              if len(chunks) == 1:
                  return parts[0], calls
              level = [p for p in parts if p]       # summaries become the next input
      
      long_doc = DOC * 4                            # 32 sentences, 476 words
      final, n_calls = map_reduce(long_doc, chunk_words=70, summary_words=30)
      print("calls per level", n_calls, "| final words", len(words(final)))  # [8, 4, 2, 1] | 29
      
      # 3) FActScore for one summary, formula (147.10)
      verdicts = {"bridge closes from March 3": "supported",
                  "repairs cost 2.4 million dollars": "contradicted",
                  "repairs take about five months": "supported",
                  "buses use the North Road bridge": "supported"}
      f = sum(v == "supported" for v in verdicts.values()) / len(verdicts)
      print("FActScore", f)                         # 0.75

      Output: MMR picks sentences [4, 2, 5]; Lead-3 gets ROUGE-1/2 of 0.644/0.318 against 0.412/0.189 for MMR, matching Figure 1. The map-reduce run on 476 words with 70-word chunks and 30-word summaries makes [8, 4, 2, 1] calls per level: because a reduce call fits only 2 summaries here, the tree has 4 levels. The FActScore is 0.75. With a real model, replace summarize() by one LLM call, for example with Hugging Face:

      # needs the transformers library and a model download; not run here
      from transformers import pipeline
      summ = pipeline("summarization", model="facebook/bart-large-cnn")
      out = summ(article, num_beams=4, length_penalty=2.0, min_length=56,
                 max_length=142, no_repeat_ngram_size=3, truncation=True)  # BART reads at most 1,024 tokens
      print(out[0]["summary_text"])

      Where this is used

      ApplicationTypical approachWhat is hard
      News digests and headlinesPrompted LLM, or fine-tuned BART/PEGASUS for high volumeLead bias; exact names and numbers
      Meeting and call summariesSpeech recognition, then an LLM with speaker turns as chunks; action items as structured outputWho said what; decisions vs suggestions; transcription errors
      Long reports, contracts, papersHierarchical map-reduce at section boundaries, or long-context stuffingLost in the middle; clauses that change meaning (“unless…”)
      Support tickets and chat threadsSmall fine-tuned model distilled from an LLM [Likely]Cost per ticket; private data
      Search and RAG answersQuery-focused summary of retrieved passages, with citations (session 152)Mixing facts from different sources
      Video summarization and key-moment detectionExtractive: score shots or frames (relevance minus redundancy, as in MMR) and keep the best; abstractive: caption clips, then map-reduce the captionsDefining “important”; temporal order
      Video-RAG and long-video QAA tree of clip captions → scene summaries → episode summary, used as an index (session 210)Errors in captions propagate up the tree
      Lecture or video chaptersTranscript map-reduce with timestamps kept per chunkAccurate time boundaries

      Common mistakes

      • Trusting ROUGE alone. A summary with a wrong number can score higher than a correct paraphrase (Figure 4). Always add a faithfulness check and some human reading.
      • Silent truncation. BART reads 1,024 tokens; a tokenizer with truncation=True drops the rest without warning, so the summary covers only the start of the document. Check input lengths.
      • No length control. Plain beam search (\(\alpha = 0\)) gives very short outputs; LLM prompts without a length make outputs vary from one line to one page. Set the length in tokens, sentences or bullets.
      • Chunking in the middle of a sentence or table. Split at headings, paragraphs or speaker turns, and overlap slightly.
      • Map-reduce prompts that lose details. If the map step says “summarize briefly”, names and numbers disappear before the reduce step ever sees them. Ask the map step to keep entities, numbers and dates.
      • Using refine when latency matters. 40 sequential calls are 40 times the latency of one; use map-reduce for long inputs unless order really matters.
      • Evaluating on a news dataset and deploying on meetings. Lead-3 wins on news, so news results say little about other domains.
      • Counting “not in source” as fine because it is true. In summarization, a true but unsupported fact is still a hallucination: the reader cannot check it and the next one may be false.

      Interview questions

      What is the difference between extractive and abstractive summarization? When would you pick each?basic
      Extractive selects sentences from the source and copies them; abstractive generates new text. Extractive is cheap, fast and cannot change facts inside a sentence, so it fits legal, medical or compliance settings where exact wording matters, or as a first step for very long inputs. Abstractive is shorter and more readable, can merge facts and follow a format, but can hallucinate. Many systems mix them: extract key passages, then write an abstractive summary of only those, with checks.
      Why is Lead-3 such a strong baseline on CNN/DailyMail but weak on XSum?core
      News articles follow the “inverted pyramid”: the main facts come first, and CNN/DM highlights reuse many article words. So the first three sentences already overlap well with the reference (ROUGE-1 40.42). XSum references are one new sentence written in other words, so copied sentences overlap little (ROUGE-1 16.30). Lesson: always report a Lead baseline, and do not trust results from one domain in another.
      Explain MMR. What happens at \(\lambda = 1\) and at small \(\lambda\)?basic
      MMR picks sentences greedily by \(\lambda\,\text{rel} - (1-\lambda)\max\text{sim}\) to the chosen ones. At \(\lambda = 1\) it is pure relevance, so it can pick several sentences that say the same thing. Small \(\lambda\) favors sentences unlike the chosen ones, which increases coverage but can bring in off-topic sentences. Typical values are around 0.5 to 0.8, tuned on data. The same idea is used to diversify retrieval results in RAG.
      How do you create training labels for a supervised extractive summarizer?core
      Datasets have abstractive references, not per-sentence labels. Build a greedy oracle: repeatedly add the source sentence that most increases ROUGE (for example ROUGE-1 + ROUGE-2) against the reference, stop when nothing improves. Label the chosen sentences 1 and others 0, then train a sentence classifier (for example on BERT sentence representations). The oracle is an upper bound for copying sentences and needs the reference, so it cannot be used at test time.
      Encoder–decoder (BART, T5) or decoder-only LLM for a summarization product?core
      Encoder–decoders are small (hundreds of millions of parameters), fast and cheap, and very good once fine-tuned on in-domain pairs, but have short inputs (BART 1,024 tokens) and a style fixed by the training data. Decoder-only LLMs need no task data, handle long inputs and many formats through prompting, and humans often prefer their summaries (Goyal et al., 2022), but cost more per request. For high volume in one domain, distill LLM summaries into a small model; for varied, long, low-volume documents, use the LLM.
      Why does BART’s pretraining help summarization? What does PEGASUS do differently?core
      BART corrupts text (30% of tokens masked in spans, sentences shuffled) and trains the decoder to rebuild the whole document, so it learns to write long fluent text grounded in a bidirectionally encoded input, which is what summarization needs. PEGASUS makes the pretraining task look even more like summarization: it removes important sentences from a document and trains the model to generate them from the rest (gap-sentence generation).
      You must summarize a 300,000-token document with a model that takes 8,000 tokens per call. Design it.advanced
      Split at section boundaries into chunks of about 8,000 tokens (38 chunks), map each to a 500-token summary in parallel, asking to keep names, numbers and dates. One reduce call can read 16 summaries, so reduce in two levels (3 calls, then 1): 42 calls, depth 3, about 320,500 input tokens. Refine would need 40 sequential calls for about the same tokens, so 13 times the latency. Keep the tree (section summaries are useful), keep source pointers, and run a faithfulness check on the final summary against the chunks it came from.
      What are intrinsic and extrinsic hallucinations? Give an example of each.basic
      Intrinsic: the summary contradicts the source, for example “cost 2.4 million” when the source says 4.2 million, or the wrong person did the action. Extrinsic: the summary adds content that cannot be checked from the source, for example “the second closure since a 2019 flood”. Extrinsic content may be true in the world but is still unfaithful, because the reader cannot verify it from the document.
      Why is ROUGE not enough, and how would you evaluate faithfulness?core
      ROUGE counts \(n\)-gram overlap with a reference: it gives the same ROUGE-1 to “4.2 million” and “2.4 million”, rewards copying, and punishes good paraphrases. Humans preferred GPT-3 summaries that ROUGE did not favor. For faithfulness: split the summary into atomic facts and check each against the source (FActScore-style), use an NLI score such as SummaC (best entailment per summary sentence, averaged), or QA-based checks (QAGS), or an LLM judge calibrated on human labels. Report coverage too, because these are precision measures.
      What does the length penalty in beam search do?advanced
      Total log-probability only goes down as tokens are added, so plain beam search favors short outputs. Dividing by \(|y|^\alpha\) changes this: \(\alpha = 1\) compares average log-probability per token, \(\alpha \gt 1\) favors longer outputs. BART-CNN uses \(\alpha = 2\), together with min_length=56, max_length=142 and no_repeat_ngram_size=3. In practice, min/max length and n-gram blocking matter as much as \(\alpha\).
      How did Stiennon et al. train a summarizer from human feedback, and why is the KL term there?advanced
      Start from a supervised fine-tuned model, sample pairs of summaries, ask people which is better, train a reward model on these comparisons, then optimize the policy with PPO on reward \(r_\phi(x,y) - \beta\log(\pi_{RL}/\pi_{SFT})\) with \(\beta = 0.05\). The KL term keeps the policy close to the SFT model; without it the policy finds outputs that fool the reward model (reward hacking) and that the reward model never saw. Their 6.7B model was preferred over human references about 65% of the time after length control.
      A long-context model can read the whole document. Should you still chunk?advanced
      Often yes, or at least test both. Models use the middle of long inputs less well (“lost in the middle”); one huge call has high latency and cost and is all-or-nothing on failure; chunked summaries give traceable intermediate results and can be cached and reused. Stuffing wins when cross-references across the whole document matter and the document fits comfortably. Evaluate both on your data with a coverage and a faithfulness metric.

      Exercises

      1. A 119-word story has a 46-word reference summary. What is the compression ratio (formula 147.1)?
      2. Map-reduce with \(N = 100{,}000\), \(C = 8{,}000\), \(s = 1{,}000\): how many calls per level, total calls, depth and input tokens? And for refine? Check with Figure 3.
      3. In Figure 2, which candidate wins at \(\alpha = 1\)? Compute its per-token score.
      4. A summary has 12 atomic facts; 9 are supported, 2 contradicted, 1 not in source. What is \(f(y)\)? Why might a one-sentence summary score higher?
      5. An NLI model gives entailment scores \(E = \begin{pmatrix} 0.9 & 0.1 \\ 0.2 & 0.3 \\ 0.1 & 0.2 \end{pmatrix}\) for 3 document sentences (rows) and 2 summary sentences (columns). Compute SummaC-ZS (formula 147.11). Which summary sentence would you check?
      6. Explain why ROUGE-1 is unchanged when “4.2” becomes “2.4”, and what ROUGE-2 sees.
      Answerscheck
      1. \(r = 46/119 = 0.39\). 2. Map: \(\lceil 100{,}000/8{,}000 \rceil = 13\); \(g = 8\); then \(\lceil 13/8 \rceil = 2\), then 1. Total 16 calls, depth 3, input \(100{,}000 + 13 \times 1{,}000 + 2 \times 1{,}000 = 115{,}000\) tokens. Refine: \(\lceil 100{,}000/7{,}000 \rceil = 15\) calls, all sequential, input \(100{,}000 + 14 \times 1{,}000 = 114{,}000\). 3. H3: \(-27.0/56 = -0.482\), better than H4 (−0.500), H2 (−0.550) and H1 (−0.750). 4. \(f = 9/12 = 0.75\); FActScore is a precision, so a summary with one safe fact can get 1.0 while missing everything important. Report the number of facts and a coverage measure as well. 5. Column maxima are 0.9 and 0.3, so the score is \((0.9 + 0.3)/2 = 0.6\). Summary sentence 2 has no strongly entailing source sentence: check it. 6. The tokenizer splits “4.2” into “4” and “2”; “2.4” has the same unigrams, so the clipped unigram counts are identical. ROUGE-2 sees the bigram (“4”, “2”) vs (“2”, “4”), plus the bigrams around them, so it drops a little (0.562 to 0.517 in Figure 4), far less than the error deserves.

      Key takeaways

      • Extractive summaries copy sentences (MMR, formula 147.3; oracle labels for training); abstractive summaries generate text (formula 147.4) and can hallucinate. Lead-3 is a strong news baseline and a weak one elsewhere.
      • BART, T5 and PEGASUS are encoder–decoders pretrained for rebuilding text; they are small and strong when fine-tuned, but read about 1,024 tokens. Decoder-only LLMs summarize from a prompt, handle long inputs, and RLHF for summaries (formula 147.6) started the modern alignment recipe.
      • Length is controlled with the beam length penalty (147.5), minimum and maximum length, and n-gram blocking; in LLMs, with the prompt.
      • For long documents: stuff if it fits; map-reduce (147.7) for low latency (depth about \(1 + \log_g k_0\)); refine (147.8) only when order matters; keep the hierarchical tree for checking and retrieval.
      • Faithfulness is the main quality problem. ROUGE (147.9) cannot see a swapped number; check atomic facts (FActScore, 147.10), NLI (SummaC, 147.11), QA or a calibrated LLM judge, and report coverage too.

      Key formulas

      \[ i^{*} = \argmax_{i \notin S}\big[\lambda\,\mathrm{rel}(s_i) - (1-\lambda)\max_{j\in S}\mathrm{sim}(s_i,s_j)\big], \qquad \mathrm{score}(y) = \frac{\log p(y\mid x)}{|y|^{\alpha}} \]
      \[ k_0 = \left\lceil \frac{N}{C} \right\rceil,\ \ \text{depth} = 1 + \lceil \log_{\lfloor C/s \rfloor} k_0 \rceil, \qquad f(y) = \frac{1}{|\mathcal{A}_y|}\sum_{a\in\mathcal{A}_y}\mathbb{I}[a \text{ supported}] \]

      References for this session

      1. Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
      2. Stiennon et al. (2020). Learning to summarize from human feedback
      3. Min et al. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
      4. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
      5. Lin (2004). ROUGE: A Package for Automatic Evaluation of Summaries
      6. Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts
      7. Liu et al. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
      8. Zhang et al. (2019). PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
      9. Maynez et al. (2020). On Faithfulness and Factuality in Abstractive Summarization
      10. Goyal, Li and Durrett (2022). News Summarization and Evaluation in the Era of GPT-3
      11. Laban et al. (2021). SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
      12. Wang, Cho and Lewis (2020). Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
      13. Carbonell and Goldstein (1998). The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries
      14. See, Liu and Manning (2017). Get To The Point: Summarization with Pointer-Generator Networks
      15. Narayan, Cohen and Lapata (2018). Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization