one model, five classic tasks
In this lab you solve five classic NLP tasks (sentiment, named entities, natural language inference, extractive question answering and summarization) on their public datasets with two kinds of models: a fine-tuned BERT-style encoder and a small decoder LLM, first prompted and then LoRA-tuned. You score each task with its proper metric, test whether differences are real, and estimate and measure the latency and cost of every prediction.
What you will learn
- How one pretrained encoder becomes five task models with three kinds of heads: one label per text, one label per token, and a start and end span
- How to turn every task into text for a decoder LLM, and why label scoring is better than parsing free answers for classification
- How to fine-tune BERT for NER (first sub-token labels) and SQuAD (overlapping windows and offset mapping), and why Lead-3 is the baseline for news summaries
- How to LoRA-tune a small decoder on the same 5,000 examples as the encoder so the comparison is fair
- Which metric each task needs (accuracy, entity F1, exact match and F1, ROUGE-Lsum) and how each simpler metric would mislead
- How to tell a real difference from noise with standard errors and a paired bootstrap
- How to estimate latency and cost per example with a roofline model, and how to measure p50 and p90 latency
Why this matters
Five tasks have shaped NLP for twenty years: sentiment (is this text positive or negative?), named entity recognition (which words are people, places and companies?), natural language inference (does sentence A imply sentence B?), extractive question answering (which words of a passage answer the question?) and summarization (write a short version). In this lab you solve all five on their standard public datasets, with two kinds of models, and you write down the quality, the cost and the speed of each.
The two kinds of models are the two answers that the industry gives today. The first is a fine-tuned encoder in the BERT style (session 33): a 108M-parameter model that reads the whole text in both directions, with a small task-specific output layer (a head) on top. The second is a small decoder LLM (Qwen2.5-0.5B-Instruct, 494M parameters) that treats every task as text in, text out: first only with a prompt, then with a LoRA adapter trained on the same examples as the encoder (session 57).
The question is practical and comes up in every applied team: “Do we still need BERT, or can a small LLM do it?” The honest answer depends on the task, the metric, the latency budget and the price, and you can only give it with measurements. That is what this lab produces: one table, five rows, three methods, with the right metric and a cost column.
You will:
- See how one pretrained model becomes five task models with three kinds of heads (Figure 1).
- Fine-tune
bert-base-casedon SST-2, MNLI, CoNLL-2003 and SQuAD, and run the Lead-3 baseline on CNN/DailyMail. - Run
Qwen2.5-0.5B-Instructon all five tasks with 0-shot and 3-shot prompts, then LoRA-tune it on each task. - Score each task with its proper metric: accuracy, entity F1, exact match and F1, ROUGE (Figures 2 and 3).
- Predict and measure latency and cost per example (Figure 4), and fill the table.
The earlier sessions explained each task on its own: session 55, session 144, session 145, session 146 and session 147. This lab puts them side by side, under the same rules, so the comparison is fair.
Setup
Hardware. One NVIDIA GPU with 16 to 24 GB (T4, L4, A10G, RTX 4090) is enough. The encoder runs take minutes each; the decoder runs take longer because the model is 4.6 times larger and generates text [Likely]. Everything also runs on a CPU, only slowly: we ran every script on a laptop CPU with tiny subsets (64 training examples, 2 steps) to check that the code works end to end. We did not run the full GPU experiments for you, and this page reports no results from them. You will measure them.
pip install "transformers>=4.56" datasets peft trl accelerate rouge_score
The code was checked with transformers 4.57, datasets 4.5, peft 0.17 and trl 0.24 on PyTorch 2.8. The scripts turn on BF16 whenever a CUDA GPU is present. A T4 has no BF16 support, so on a T4 (free Colab) use FP16 instead: fp16=True in the training arguments and torch.float16 when loading [Certain].
The two models. We counted the parameters by building each model from its config.json.
| Model | Shape | Parameters | Role |
|---|---|---|---|
google-bert/bert-base-cased | 12 layers, \(d = 768\), 12 heads, vocabulary 28,996, at most 512 tokens | 108,310,272 (22,268,928 of them in the token embeddings) | Encoder, all weights fine-tuned, one copy per task |
Qwen/Qwen2.5-0.5B-Instruct | 24 layers, \(d = 896\), 14 query heads, 2 KV heads, vocabulary 151,936, tied embeddings | 494,032,768 | Decoder, prompted (frozen) or with a LoRA adapter per task |
We use the cased BERT on purpose. Capital letters are a strong signal for names, and the BERT authors also used a case-preserving model for CoNLL-2003 [Certain]. We use the instruct version of Qwen for both the prompted and the LoRA runs, so the only difference between those two columns is the adapter.
The five datasets. All are on the Hugging Face Hub. The split sizes come from the Hub’s dataset viewer.
| Task | Dataset (Hub id) | Splits | We train on | We test on | Metric |
|---|---|---|---|---|---|
| Sentiment | SST-2: nyu-mll/glue, config sst2 | 67,349 train, 872 validation (the test labels are hidden) | 5,000 | all 872 validation | Accuracy |
| NLI | MNLI: nyu-mll/glue, config mnli | 392,702 train, 9,815 validation matched, 9,832 mismatched | 5,000 | 1,000 validation matched | Accuracy |
| NER | CoNLL-2003: lhoestq/conll2003 | 14,041 train, 3,250 validation, 3,453 test sentences | 5,000 | 1,000 test | Entity-level F1 |
| Extractive QA | SQuAD v1.1: rajpurkar/squad | 87,599 train, 10,570 validation | 5,000 | 1,000 validation | Exact match and F1 |
| Summarization | CNN/DailyMail 3.0.0: abisee/cnn_dailymail | 287,113 train, 13,368 validation, 11,490 test | 5,000 | 500 test | ROUGE-1, ROUGE-2, ROUGE-Lsum |
Every subset is drawn with shuffle(seed=0), and every method sees the same 5,000 training examples per task. That is the most important rule of this lab. If the encoder trains on 67,000 reviews and the LLM on 5,000, you compare data sizes, not models. The test sets are small on purpose so the decoder runs stay short; formula (159.9) tells you how much noise that adds.
Two dataset details that save time. The GLUE copy of MNLI uses the labels 0 = entailment, 1 = neutral, 2 = contradiction. The CoNLL copy stores the tags as numbers; the order is O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-MISC, I-MISC. We checked it on the first training sentence, “EU rejects German call to boycott British lamb .”, whose tags are 3 0 7 0 0 0 7 0 0: EU is B-ORG, German and British are B-MISC.
The table you will fill. Copy it into your notes now.
| Task (metric) | BERT-base fine-tuned | Qwen 0-shot | Qwen 3-shot | Qwen + LoRA | p50 latency: BERT / Qwen | Cost per 1M examples |
|---|---|---|---|---|---|---|
| SST-2 (accuracy) | ||||||
| MNLI-m (accuracy) | ||||||
| CoNLL-2003 (entity F1) | ||||||
| SQuAD v1.1 (EM / F1) | ||||||
| CNN/DM (R-1 / R-2 / R-Lsum) | Lead-3: | 0 / |
Step 1: one pretrained model, five task shapes
The five tasks look different, but there are only three output shapes. Sentiment and NLI give one label per input. NER gives one label per token. Extractive QA gives a span: a start position and an end position inside the passage. Summarization is the odd one: it gives new text, so it needs a model that can generate.
For an encoder, each shape gets its own small head on top of the final hidden vectors \(\mathbf{h}_1, \dots, \mathbf{h}_n\) (one vector of size \(d = 768\) per token). For one label per input, BERT reads the vector of the special first token [CLS]:
For one label per token, the same kind of linear layer is applied to every token. A word such as “Blackburn” may be split into several sub-tokens; we put the label on the first sub-token only and ignore the others in the loss, as the BERT paper did [Certain]. With \(S\) the set of first sub-tokens:
For a span, two vectors \(\mathbf{w}_s\) and \(\mathbf{w}_e\) give every context token a start score and an end score. The answer is the pair with the highest total score, with the start before the end and at most \(L_{\max}\) tokens (we use 30):
- Read the symbols
- \(\mathbf{h}_t\)
- the final hidden vector of token \(t\) (768 numbers for BERT-base)
- \(C\)
- the number of labels: 2 for SST-2, 3 for MNLI, 9 BIO tags for CoNLL-2003
- \(W, \mathbf{b}\)
- the new head, created with random values and trained with the rest of the model
- \(S\)
- the positions of the first sub-token of each word; other sub-tokens get the label
-100, which the loss skips - \(s_i, e_j\)
- the start score of token \(i\) and the end score of token \(j\); training uses cross-entropy over the start position and over the end position
Every head is just a linear layer of size \((d + 1) \times C\):
With \(d = 768\): SST-2 \(769 \times 2 = 1{,}538\), MNLI \(769 \times 3 = 2{,}307\), CoNLL-2003 \(769 \times 9 = 6{,}921\), SQuAD \(769 \times 2 = 1{,}538\) (one start and one end score). The full models have 108,311,810 parameters for SST-2 and 107,726,601 for NER: Hugging Face drops BERT’s pooler layer (590,592 weights) for token classification and QA, because those heads read every token, not [CLS]. So 99.99% of what you fine-tune is the shared encoder; the head is under 0.01%.
The decoder has no task heads. It already has one head, the language-model head over 151,936 tokens, and every task becomes “write the answer as text”. Step through Figure 1 to compare the three ways for each task.
How to read Figure 1
Pick a task, then a method. The model box changes color with the method. Look at three things. First, the encoder needs a different head for each shape, while the decoder uses one interface for all five tasks. Second, the encoder input is short (20.2 tokens for an average CoNLL sentence), while the 3-shot decoder prompt for the same sentence is 337.8 tokens, because it carries the instruction and three worked examples. Third, for summarization the encoder column has no model at all: we use the Lead-3 baseline, which every summarizer must beat.
Step 2: every task as text for the decoder
For the decoder, each example becomes one user message and one target answer. We keep the prompts short and fix the output format exactly, because a parser must read the answer back:
| Task | User message (shortened) | Target answer | How we read the answer |
|---|---|---|---|
| SST-2 | Classify the sentiment … Answer with one word: positive or negative. Sentence: … | positive | label scoring (159.5) |
| MNLI | Premise: … Hypothesis: … Answer with one word: entailment, neutral or contradiction. | neutral | label scoring (159.5) |
| CoNLL | Find the named entities … one entity per line as TYPE: text … Sentence: … | ORG: EUMISC: German | parse lines with a regular expression |
| SQuAD | Answer with a short span copied exactly from the context. Context: … Question: … | Denver Broncos | first line of the output |
| CNN/DM | Summarize the news article in three short sentences. Article: (first 4,000 characters) | the reference highlights | the whole output |
For the two classification tasks we do not let the model write freely. A free answer can be “Positive!”, “The sentiment is positive” or a long explanation, and then the parser decides your accuracy. Instead we score each allowed label: we append the label to the prompt, add up the log-probabilities of its tokens, and pick the label with the highest total (session 144 calls this comparing label likelihoods):
- Read the symbols
- \(\mathcal{Y}\)
- the set of label words, such as {negative, positive}
- \(y_t\)
- the \(t\)-th token of label \(y\); in Qwen’s vocabulary “positive”, “negative” and “neutral” are one token each, while “entailment” and “contradiction” are three tokens each
- \(p_\theta\)
- the decoder’s next-token probability
Summing log-probabilities can favor short labels, because every extra token adds a negative number. Once the model has written “contr”, the tokens “ad” and “iction” are almost certain, so the effect is usually small; still, check the confusion matrix for a bias toward “neutral”, and try the mean per token if you see one [Likely]. For the other three tasks the model must generate, and we use greedy decoding (always the most likely next token), so every run is repeatable.
LoRA tuning uses the same prompts. Each training row is a (prompt, completion) pair, and the loss is next-token cross-entropy on the completion tokens \(C\) only, as in session 54:
With \(r = 16\) on all seven linear layers of every block, LoRA trains 8,798,208 weights, 1.78% of the model (session 62 derives this count). After training we merge the adapter into the weights, so the LoRA model has exactly the speed of the base model.
- input: task \(\tau\), train subset \(D_{tr}\) (5,000 rows, seed 0), test subset \(D_{te}\), metric \(m_\tau\)
- encoder: add the head for the shape of \(\tau\); fine-tune all weights on \(D_{tr}\) (3 epochs, LR \(2 \times 10^{-5}\))
- predict on \(D_{te}\); map sub-token or window outputs back to words or characters
- decoder, prompted: for \(k \in \{0, 3\}\): put the first \(k\) rows of \(D_{tr}\) in the prompt as examples
- if \(\tau\) is a classification task: label scoring (159.5) else: greedy generation, then parse
- decoder + LoRA: train an adapter on \(D_{tr}\) with loss (159.6), 1 epoch; merge it; predict 0-shot
- for every method: score with \(m_\tau\) on the same \(D_{te}\); save every prediction for paired tests
- measure p50 latency at batch 1 on 50 test inputs // bench.py, Step 6
Step 3: the shared code
One file holds everything that must be identical across methods: the data subsets, the prompts, the parsers and the metrics. The other scripts import it. Keeping this in one place is how you make sure that a difference in the table comes from the model and not from the plumbing.
# tasks.py: the five tasks, their data, their prompts and their metrics import os, re, string, collections from datasets import load_dataset from rouge_score import rouge_scorer N_TRAIN = int(os.environ.get("N_TRAIN", 5000)) # same training examples for every method N_EVAL = int(os.environ.get("N_EVAL", 1000)) # summarization uses at most 500 NER_TAGS = ["O", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC", "B-MISC", "I-MISC"] NLI_LABELS = ["entailment", "neutral", "contradiction"] SST_LABELS = ["negative", "positive"] def load(task): """Return (train, test) with a fixed seeded subset.""" if task == "sst2": ds = load_dataset("nyu-mll/glue", "sst2"); tr, te = ds["train"], ds["validation"] elif task == "mnli": ds = load_dataset("nyu-mll/glue", "mnli"); tr, te = ds["train"], ds["validation_matched"] elif task == "conll": ds = load_dataset("lhoestq/conll2003"); tr, te = ds["train"], ds["test"] elif task == "squad": ds = load_dataset("rajpurkar/squad"); tr, te = ds["train"], ds["validation"] elif task == "cnndm": # load one of the three train shards only: we need 5,000 rows, not 287k ds = load_dataset("abisee/cnn_dailymail", data_files={ "train": "3.0.0/train-00000-of-00003.parquet", "test": "3.0.0/test-00000-of-00001.parquet"}) tr, te = ds["train"], ds["test"] n_eval = min(N_EVAL, 500 if task == "cnndm" else N_EVAL, len(te)) tr = tr.shuffle(seed=0).select(range(min(N_TRAIN, len(tr)))) te = te.shuffle(seed=0).select(range(n_eval)) return tr, te # ---------- prompts for the decoder: (user message, target text) ---------- def to_chat(task, ex): if task == "sst2": u = ("Classify the sentiment of this movie review sentence. " "Answer with one word: positive or negative.\n\nSentence: " + ex["sentence"]) return u, SST_LABELS[ex["label"]] if task == "mnli": u = ("Premise: " + ex["premise"] + "\nHypothesis: " + ex["hypothesis"] + "\n\nIs the hypothesis entailed by the premise, neutral, or a contradiction? " "Answer with one word: entailment, neutral or contradiction.") return u, NLI_LABELS[ex["label"]] if task == "conll": u = ("Find the named entities in the sentence. Types: PER (person), ORG (organization), " "LOC (location), MISC (other names). Write one entity per line as TYPE: text. " "Write NONE if there is no entity.\n\nSentence: " + " ".join(ex["tokens"])) ents = bio_to_spans(ex["tokens"], [NER_TAGS[t] for t in ex["ner_tags"]]) return u, "\n".join(f"{t}: {s}" for t, s in ents) or "NONE" if task == "squad": u = ("Answer the question with a short span copied exactly from the context.\n\n" "Context: " + ex["context"] + "\n\nQuestion: " + ex["question"]) return u, ex["answers"]["text"][0] if task == "cnndm": u = "Summarize the news article in three short sentences.\n\nArticle: " + ex["article"][:4000] return u, ex["highlights"] LABELS = {"sst2": SST_LABELS, "mnli": NLI_LABELS} # scored, not generated MAX_NEW = {"conll": 64, "squad": 24, "cnndm": 128} def gold(task, ex): if task == "sst2": return SST_LABELS[ex["label"]] if task == "mnli": return NLI_LABELS[ex["label"]] if task == "conll": return bio_to_spans(ex["tokens"], [NER_TAGS[t] for t in ex["ner_tags"]]) if task == "squad": return ex["answers"]["text"] if task == "cnndm": return ex["highlights"] def parse(task, text): """Turn the decoder's text into a prediction of the same type as gold().""" text = text.strip() if task == "conll": out = [] for line in text.split("\n"): m = re.match(r"\s*(PER|ORG|LOC|MISC)\s*:\s*(.+?)\s*$", line) if m: out.append((m.group(1), m.group(2))) return out if task == "squad": return text.split("\n")[0] return text # ---------- metrics ---------- def bio_to_spans(tokens, tags): """BIO tags -> list of (type, text). A stray I- tag starts a new entity (as in seqeval).""" spans, cur = [], None for tok, tag in zip(tokens, tags): if tag == "O" or tag.startswith("B-") or (cur and tag[2:] != cur[0]): if cur: spans.append((cur[0], " ".join(cur[1]))); cur = None if tag != "O": if cur is None: cur = (tag[2:], []) cur[1].append(tok) if cur: spans.append((cur[0], " ".join(cur[1]))) return spans def entity_prf(preds, golds): """Micro P/R/F1 over (type, text) entities, counted as multisets per sentence.""" tp = n_pred = n_gold = 0 for p, g in zip(preds, golds): cp, cg = collections.Counter(p), collections.Counter(g) tp += sum((cp & cg).values()); n_pred += sum(cp.values()); n_gold += sum(cg.values()) P = tp / n_pred if n_pred else 0.0; R = tp / n_gold if n_gold else 0.0 return {"precision": P, "recall": R, "f1": 2 * P * R / (P + R) if P + R else 0.0} def norm_answer(s): # the official SQuAD v1.1 normalization s = "".join(ch for ch in s.lower() if ch not in set(string.punctuation)) s = re.sub(r"\b(a|an|the)\b", " ", s) return " ".join(s.split()) def squad_f1(pred, gt): p, g = norm_answer(pred).split(), norm_answer(gt).split() common = sum((collections.Counter(p) & collections.Counter(g)).values()) if common == 0: return 0.0 P, R = common / len(p), common / len(g) return 2 * P * R / (P + R) def squad_em_f1(preds, golds): em = sum(max(float(norm_answer(p) == norm_answer(g)) for g in gs) for p, gs in zip(preds, golds)) f1 = sum(max(squad_f1(p, g) for g in gs) for p, gs in zip(preds, golds)) return {"em": em / len(preds), "f1": f1 / len(preds)} _rouge = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeLsum"], use_stemmer=True) def rouge(preds, golds): # rougeLsum wants one sentence per line nl = lambda s: "\n".join(x.strip() for x in re.split(r"(?<=[.!?])\s+", s.strip()) if x.strip()) sc = [_rouge.score(nl(g), nl(p)) for p, g in zip(preds, golds)] return {k: sum(s[k].fmeasure for s in sc) / len(sc) for k in ["rouge1", "rouge2", "rougeLsum"]} def score(task, preds, golds): if task in ("sst2", "mnli"): return {"accuracy": sum(p == g for p, g in zip(preds, golds)) / len(golds)} if task == "conll": return entity_prf(preds, golds) if task == "squad": return squad_em_f1(preds, golds) if task == "cnndm": return rouge(preds, golds)
Three details in this file matter. First, bio_to_spans turns BIO tags into entities the same way for the encoder and for the gold labels, and treats a stray I- tag as the start of a new entity, as the standard seqeval package does in its default mode. Second, norm_answer is the normalization of the official SQuAD v1.1 evaluation script: lower case, remove punctuation, remove the articles “a”, “an”, “the”, and collapse spaces. Third, ROUGE-Lsum needs one sentence per line, so rouge() splits the text into sentences first; without that, rougeLsum is the same as plain rougeL [Certain].
Step 4: the encoder runs
# encoder.py: fine-tune BERT on one task. Usage: python encoder.py conll import sys, os, json, numpy as np, torch from transformers import (AutoTokenizer, AutoModelForSequenceClassification, AutoModelForTokenClassification, AutoModelForQuestionAnswering, Trainer, TrainingArguments, DataCollatorWithPadding, DataCollatorForTokenClassification, set_seed) from tasks import load, score, gold, bio_to_spans, NER_TAGS, SST_LABELS, NLI_LABELS task = sys.argv[1]; set_seed(0) ENC = os.environ.get("ENC", "google-bert/bert-base-cased") # cased: capital letters matter for NER tok = AutoTokenizer.from_pretrained(ENC) tr, te = load(task) args = TrainingArguments(output_dir=f"out/enc-{task}", learning_rate=3e-5 if task == "squad" else 2e-5, num_train_epochs=float(os.environ.get("EPOCHS", 3)), per_device_train_batch_size=32, per_device_eval_batch_size=64, warmup_ratio=0.1, weight_decay=0.01, max_steps=int(os.environ.get("MAX_STEPS", -1)), bf16=torch.cuda.is_available(), save_strategy="no", report_to="none", seed=0, logging_steps=50) if task in ("sst2", "mnli"): # one label per text (or text pair) names = SST_LABELS if task == "sst2" else NLI_LABELS enc = (lambda b: tok(b["sentence"], truncation=True, max_length=128)) if task == "sst2" else \ (lambda b: tok(b["premise"], b["hypothesis"], truncation=True, max_length=128)) model = AutoModelForSequenceClassification.from_pretrained(ENC, num_labels=len(names)) trainer = Trainer(model=model, args=args, train_dataset=tr.map(enc, batched=True), data_collator=DataCollatorWithPadding(tok)) trainer.train() logits = trainer.predict(te.map(enc, batched=True)).predictions preds = [names[i] for i in logits.argmax(-1)] elif task == "conll": # one label per word def enc(b): out = tok(b["tokens"], is_split_into_words=True, truncation=True, max_length=256) out["labels"] = [] for i, tags in enumerate(b["ner_tags"]): prev, lab = None, [] for w in out.word_ids(i): # label only the FIRST sub-token of each word lab.append(-100 if w is None or w == prev else tags[w]); prev = w out["labels"].append(lab) return out model = AutoModelForTokenClassification.from_pretrained(ENC, num_labels=len(NER_TAGS)) trainer = Trainer(model=model, args=args, train_dataset=tr.map(enc, batched=True, remove_columns=tr.column_names), data_collator=DataCollatorForTokenClassification(tok)) trainer.train() te_enc = te.map(enc, batched=True, remove_columns=te.column_names) out = trainer.predict(te_enc) preds = [] for ex, lg, lab in zip(te, out.predictions, out.label_ids): tags = [NER_TAGS[p] for p, l in zip(lg.argmax(-1), lab) if l != -100] # back to one tag per word tags += ["O"] * (len(ex["tokens"]) - len(tags)) # words cut by truncation preds.append(bio_to_spans(ex["tokens"], tags)) elif task == "squad": # point to the answer: a start and an end token def enc_train(b): out = tok(b["question"], b["context"], truncation="only_second", max_length=384, stride=128, return_overflowing_tokens=True, return_offsets_mapping=True, padding="max_length") out["start_positions"], out["end_positions"] = [], [] for i, offs in enumerate(out["offset_mapping"]): a = b["answers"][out["overflow_to_sample_mapping"][i]] s_char = a["answer_start"][0]; e_char = s_char + len(a["text"][0]) seq = out.sequence_ids(i) ctx = [k for k, s in enumerate(seq) if s == 1] if offs[ctx[0]][0] > s_char or offs[ctx[-1]][1] < e_char: # answer not in this window out["start_positions"].append(0); out["end_positions"].append(0); continue st = next(k for k in ctx if offs[k][1] > s_char) en = max(k for k in ctx if offs[k][0] < e_char) out["start_positions"].append(st); out["end_positions"].append(en) out.pop("offset_mapping"); out.pop("overflow_to_sample_mapping") return out def enc_eval(b, idx): out = tok(b["question"], b["context"], truncation="only_second", max_length=384, stride=128, return_overflowing_tokens=True, return_offsets_mapping=True, padding="max_length") out["ex_id"] = [idx[j] for j in out["overflow_to_sample_mapping"]] # global example index out["offset_mapping"] = [[o if s == 1 else None for o, s in zip(offs, out.sequence_ids(i))] for i, offs in enumerate(out["offset_mapping"])] out.pop("overflow_to_sample_mapping") return out model = AutoModelForQuestionAnswering.from_pretrained(ENC) trainer = Trainer(model=model, args=args, train_dataset=tr.map(enc_train, batched=True, remove_columns=tr.column_names)) trainer.train() feats = te.map(enc_eval, batched=True, with_indices=True, remove_columns=te.column_names) S, E = trainer.predict(feats.remove_columns(["ex_id", "offset_mapping"])).predictions ids, offsets, ctxs = feats["ex_id"], feats["offset_mapping"], te["context"] best = {} # example index -> (score, answer text) for i, offs, s, e in zip(ids, offsets, S, E): for a in np.argsort(s)[-20:]: for b in np.argsort(e)[-20:]: if offs[a] is None or offs[b] is None or b < a or b - a + 1 > 30: continue sc = s[a] + e[b] if sc > best.get(i, (-1e9, ""))[0]: best[i] = (sc, ctxs[i][offs[a][0]:offs[b][1]]) preds = [best.get(i, (0, ""))[1] for i in range(len(te))] elif task == "cnndm": # an encoder cannot write: report the Lead-3 baseline import re first3 = lambda a: " ".join(re.split(r"(?<=[.!?])\s+", a.strip())[:3]) preds = [first3(a) for a in te["article"]] golds = [gold(task, ex) for ex in te] res = {"task": task, "model": ENC, **score(task, preds, golds)} print(json.dumps(res)); json.dump(res, open(f"enc-{task}.json", "w"))
Run it four times (python encoder.py sst2, mnli, conll, squad) and once with cnndm for the Lead-3 baseline. The hyperparameters come from the ranges the BERT authors recommend for fine-tuning: batch size 16 or 32, learning rate \(5 \times 10^{-5}\), \(3 \times 10^{-5}\) or \(2 \times 10^{-5}\), and 2 to 4 epochs [Certain]. Notes on the two hard cases:
- NER: sub-tokens back to words. The model predicts one tag per sub-token. We keep the prediction of the first sub-token of each word, so the number of predicted tags equals the number of words, and then we build entities with the same function as for the gold tags.
- QA: windows over long passages. A SQuAD passage plus question is 183.6 BERT tokens on average in our test subset, but 6 of the 1,000 are longer than 512 tokens. The tokenizer cuts long inputs into windows of 384 tokens that overlap by 128 (
stride). In training, a window that does not contain the answer is labeled with position 0 (the[CLS]token). In evaluation, we search the top 20 start and top 20 end scores in every window, keep spans of at most 30 tokens, and map the best span back to characters with the offset mapping (the character positions of every token).
Why Lead-3? News articles put the main facts first. Copying the first three sentences is a strong summarization baseline on CNN/DailyMail, and many abstractive models of the 2017 era did not clearly beat it on ROUGE [Likely]. It costs nothing and needs no GPU. If your LoRA summarizer does not beat Lead-3, report that honestly.
Step 5: the decoder runs
# decoder.py: one small decoder LLM on one task, prompted (k-shot) or LoRA-tuned. # Usage: python decoder.py squad prompt 0 | python decoder.py squad prompt 3 | python decoder.py squad lora import sys, os, json, time, torch from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed from tasks import load, to_chat, parse, score, gold, LABELS, MAX_NEW task, mode = sys.argv[1], sys.argv[2]; k = int(sys.argv[3]) if len(sys.argv) > 3 else 0 set_seed(0) DEC = os.environ.get("DEC", "Qwen/Qwen2.5-0.5B-Instruct") dev = "cuda" if torch.cuda.is_available() else "cpu" tok = AutoTokenizer.from_pretrained(DEC); tok.padding_side = "left" # left-pad for batched generation tr, te = load(task) shots = [to_chat(task, tr[i]) for i in range(k)] # the same k examples for every test input def messages(user): m = [] for u, a in shots: m += [{"role": "user", "content": u}, {"role": "assistant", "content": a}] return m + [{"role": "user", "content": user}] if mode == "lora": # SFT on (prompt, completion) pairs, loss on the completion only from peft import LoraConfig from trl import SFTConfig, SFTTrainer rows = [to_chat(task, ex) for ex in tr] from datasets import Dataset data = Dataset.from_list([{"prompt": [{"role": "user", "content": u}], "completion": [{"role": "assistant", "content": a}]} for u, a in rows]) cfg = SFTConfig(output_dir=f"out/lora-{task}", learning_rate=2e-4, num_train_epochs=1, per_device_train_batch_size=8, gradient_accumulation_steps=2, max_length=1536, lr_scheduler_type="cosine", warmup_ratio=0.03, bf16=(dev == "cuda"), max_steps=int(os.environ.get("MAX_STEPS", -1)), logging_steps=25, save_strategy="no", report_to="none", seed=0) lcfg = LoraConfig(task_type="CAUSAL_LM", r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear") base = AutoModelForCausalLM.from_pretrained(DEC, dtype=torch.bfloat16 if dev == "cuda" else torch.float32) trainer = SFTTrainer(model=base, args=cfg, train_dataset=data, processing_class=tok, peft_config=lcfg) trainer.train() model = trainer.model.merge_and_unload() # serve the merged model: no extra latency else: model = AutoModelForCausalLM.from_pretrained(DEC, dtype=torch.bfloat16 if dev == "cuda" else torch.float32) model.to(dev).eval() prompts = [tok.apply_chat_template(messages(to_chat(task, ex)[0]), tokenize=False, add_generation_prompt=True) for ex in te] t0 = time.time(); preds = [] with torch.no_grad(): if task in LABELS: # classification: score each label, pick the most likely lab_ids = [tok(l, add_special_tokens=False)["input_ids"] for l in LABELS[task]] for p in prompts: ids = tok(p, add_special_tokens=False)["input_ids"] batch = [ids + l for l in lab_ids] L = max(map(len, batch)) x = torch.tensor([[tok.pad_token_id] * (L - len(b)) + b for b in batch], device=dev) att = (torch.arange(L, device=dev)[None] >= torch.tensor([L - len(b) for b in batch], device=dev)[:, None]).long() logp = model(input_ids=x, attention_mask=att).logits.float().log_softmax(-1) s = [sum(logp[j, L - len(l) - 1 + t, l[t]].item() for t in range(len(l))) for j, l in enumerate(lab_ids)] preds.append(LABELS[task][max(range(len(s)), key=s.__getitem__)]) else: # generation: greedy, then parse the text for i in range(0, len(prompts), 16): b = tok(prompts[i:i + 16], return_tensors="pt", padding=True, add_special_tokens=False).to(dev) g = model.generate(**b, max_new_tokens=MAX_NEW[task], do_sample=False, pad_token_id=tok.pad_token_id) for row in tok.batch_decode(g[:, b["input_ids"].shape[1]:], skip_special_tokens=True): preds.append(parse(task, row)) dt = time.time() - t0 golds = [gold(task, ex) for ex in te] name = f"dec-{task}-{mode}{k if mode == 'prompt' else ''}" res = {"run": name, **score(task, preds, golds), "eval_seconds": dt, "n": len(te)} print(json.dumps(res)); json.dump({**res, "preds": preds}, open(name + ".json", "w"), default=str)
Run, for every task: python decoder.py TASK prompt 0, python decoder.py TASK prompt 3 and python decoder.py TASK lora. For CNN/DM, the 3-shot prompt is 2,967 tokens on average (three full articles with summaries), so it is slow and expensive; it is still worth one run, because it shows what in-context examples cost (Figure 4).
Points to check in your own outputs:
- Read 20 raw outputs per task before you trust a number. For NER and QA, most early errors are format errors: an entity type in lower case, a full sentence instead of a span, a quote around the answer. Each is a parser problem or a prompt problem, not a knowledge problem, and you should report which one it is.
- Left padding. The script pads batches on the left. With right padding, a decoder would continue from pad tokens and produce garbage for the shorter prompts.
- The same three shots for every input. The script uses the first \(k\) training rows. This keeps runs repeatable. Choosing shots per input (for example the most similar training examples) is a fair exercise, but then report it as a different method.
- Train on the target, not the prompt. TRL’s
SFTTrainercomputes the loss on the completion only for prompt-completion data, and the chat template adds the end-of-turn token<|im_end|>after the answer, which teaches the model to stop [Likely].
Step 6: the right metric for each task
The table mixes five metrics. Each one exists because a simpler number would mislead. This step shows how.
NER: entity F1, not token accuracy
In the CoNLL-2003 test set, 82.5% of the 46,435 tokens have the tag O (not an entity). A model that predicts O everywhere has 82.5% token accuracy and finds no entity at all. So NER is scored on entities: an entity counts as correct only if its type and its exact boundaries are both right.
- Read the symbols
- pred, gold
- the sets of predicted and true entities, each a triple (type, start, end) for the encoder, summed over all sentences (micro average)
- \(P, R\)
- precision: the share of predicted entities that are right; recall: the share of true entities that were found
The decoder writes entity text, not positions. So in tasks.py we compare (type, text) pairs per sentence for both models, counted as multisets. This is a slightly easier test than position matching: if the same name appears twice in a sentence and the model finds it once, position matching knows which one it found, and text matching does not. On CoNLL this rarely matters, but say which version you report [Likely].
seqeval package exactly.How to read Figure 2
Start with “Boundary error”: the model tags only “York” as a location, not “New York”. Two tags are wrong, so token accuracy is still 0.846, but the entity is wrong and F1 drops to 0.750. “Split entity” tags “Tim” and “Cook” as two people: one wrong tag, token accuracy 0.923, but precision falls to 0.600 because the model now predicts five entities, and F1 is 0.667. “All O” keeps 6 of 13 tags right (0.462) with F1 0. Click tokens to build your own error and predict the F1 before you look.
QA: exact match and F1 after normalization
SQuAD has up to three human answers per question in the validation set, and they often differ by a word (“1889” versus “March 1889”). The official script normalizes each string, then computes two scores and keeps the best over the gold answers \(G\):
- Read the symbols
- \(\hat{a}, g\)
- the predicted answer and one gold answer, as lists of normalized words
- \(c_g\)
- the number of shared words (each word counted at most as often as it appears in both)
- \(\mathbb{1}[\cdot]\)
- 1 if the two normalized strings are identical, else 0
Context: The Eiffel Tower was completed in March 1889 for the World’s Fair in Paris. Question: When was the Eiffel Tower completed?
| Gold answer | Normalized | Shared | P | R | F1 | EM |
|---|
How to read Figure 3
“March 1889.” gets EM 1: the period and the case disappear in normalization. The full sentence “It was completed in March 1889” gets EM 0 but F1 0.667 against the gold “in March 1889”. The wrong month, “May 1889”, also gets F1 0.667, against the gold “1889”: token F1 gives partial credit for a wrong fact. This is why you report EM and F1 together, and why a decoder that writes long answers can get decent F1 and poor EM.
Normalized prediction: [may, 1889], 2 words. Gold “March 1889”: 1 shared word, \(P = 1/2\), \(R = 1/2\), \(F_1 = 0.5\). Gold “1889”: 1 shared word, \(P = 1/2\), \(R = 1/1\), \(F_1 = 2 \cdot 0.5 \cdot 1 / 1.5 = 0.667\). Gold “in March 1889”: \(P = 1/2\), \(R = 1/3\), \(F_1 = 0.4\). The maximum is 0.667, and EM is 0.
Summarization: ROUGE, and its limits
ROUGE counts overlapping words (ROUGE-1), word pairs (ROUGE-2) and the longest common subsequence with the reference (session 149 derives all three). For multi-sentence summaries such as CNN/DM highlights, the usual choice is ROUGE-Lsum, which computes the longest common subsequence sentence by sentence. Two warnings. ROUGE rewards copying: Lead-3 copies the article and scores well. And ROUGE cannot see a wrong fact that uses the right words. Read 20 summaries yourself, and if you need a quality judgment, add an LLM judge with position swapping as in the session 62 lab.
Is a difference real?
Every number in your table is an estimate from a small test set. For an accuracy \(\hat{p}\) on \(n\) examples:
With \(\hat{p} = 0.90\) on the 872 SST-2 sentences, SE = 0.0102, so a 95% interval is about \(\pm 2.0\) points. With \(\hat{p} = 0.80\) on 1,000 MNLI pairs, it is \(\pm 2.5\) points. Two methods 1.5 points apart are not clearly different on these test sets. For F1, EM and ROUGE there is no simple formula, so use a paired bootstrap: because all methods predict on the same examples, resample the examples and recompute the difference.
- input: per-example predictions of methods \(a\) and \(b\) on the same \(n\) test examples, a metric \(m\), \(B = 1{,}000\) rounds, a fixed seed
- \(\Delta \leftarrow m(a) - m(b)\) on the full test set
- for \(r = 1, \dots, B\):
- draw \(n\) example indices with replacement
- \(\Delta_r \leftarrow m(a) - m(b)\) on those indices // the same indices for both methods
- report \(\Delta\) and the 2.5% and 97.5% quantiles of \(\Delta_1, \dots, \Delta_B\)
- if the interval contains 0: the difference is not clear on this test set
For entity F1, resample whole sentences, not entities, and recompute the micro F1 from the resampled counts. The pairing is what makes this test strong: most examples are easy for both methods and cancel out.
Step 7: cost and latency
Quality is half the answer. The other half is what each prediction costs. Start with the compute. A forward pass costs about 2 FLOPs per weight per token (one multiply and one add), as in session 25. For an encoder that reads \(n_{in}\) tokens once, and a decoder that reads a prompt of \(n_p\) tokens and then writes \(n_{out}\) tokens:
But at batch size 1 the FLOPs do not decide the time. The decoder writes its answer one token at a time, and each new token needs a full pass over the weights (session 78). Reading 494M BF16 weights is 0.988 GB per token. So a simple lower bound on the time (a “roofline”) takes the larger of the compute time and the memory time for each pass:
- Read the symbols
- \(N_E, N_D\)
- weights used in matrix multiplications: 85,645,056 for BERT-base (all but the embedding tables), 494,032,768 for Qwen2.5-0.5B (its tied embedding is also the output layer, so it is read at every step)
- \(F, BW\)
- the GPU’s peak dense BF16 FLOP rate and its memory bandwidth
- \(b\)
- the batch size; \(T\) is the time for the whole batch, so the time per example is \(T / b\)
- \(n_{ctx}, \kappa\)
- tokens already in the KV cache, and KV-cache bytes per token. For Qwen2.5-0.5B: K and V, 24 layers, 2 KV heads of size 64, 2 bytes each, so \(\kappa = 2 \cdot 24 \cdot 2 \cdot 64 \cdot 2 = 12{,}288\)
The first term is the prefill (all prompt tokens in one pass); the second is the decode loop. The encoder has only the first term. The money follows directly. With a GPU price of \(c\) dollars per hour:
bench.py.How to read Figure 4
Start at QA on an L4, batch 1. The encoder needs one pass, bounded by reading its 0.171 GB of weights: 0.571 ms. The decoder needs one prefill pass and 4.7 decode passes, each reading 0.988 GB: 18.82 ms in total, 33.0 times slower. Now switch to Summary: there is no encoder bar, and the decoder needs 68.8 decode steps, 234.8 ms. Then raise the batch to 64: the decode passes now serve 64 examples each, and the time per QA example drops to 2.15 ms. Batching helps the decoder far more than the encoder, because the decoder at batch 1 wastes most of the GPU’s compute. Finally, switch QA to the 3-shot prompt at batch 64: the prompt grows from 228.5 to 854.5 tokens and the cost per example goes up 3.4 times. LoRA lets you drop the shots.
Encoder, 183.6 tokens: compute \(2 \times 85.6\text{M} \times 183.6 / 121 \times 10^{12} = 0.26\) ms; memory \(0.171 \text{ GB} / 300 \text{ GB/s} = 0.571\) ms. The larger one wins: 0.571 ms.
Decoder, 228.5 prompt tokens and 5.7 answer tokens (including the end token): the prefill compute is \(2 \times 494\text{M} \times 228.5 / 121 \times 10^{12} = 1.87\) ms, its memory time is \(0.988 / 300 = 3.29\) ms, so 3.29 ms. Then 4.7 decode steps of 3.29 ms each, plus a little for reading the KV cache: 15.52 ms. Total 18.82 ms. At $1 per GPU-hour, one million questions cost $0.16 with the encoder and $5.23 with the decoder, before any real-world overhead.
The estimate ignores attention FLOPs, kernel launch overheads, tokenization and Python time, so real numbers are higher, often by a factor that is larger for small models [Likely]. Measure p50 and p90 latency (the median and the 90th percentile) yourself:
# bench.py: latency per request (batch 1) for the encoder and the decoder on one task. # Usage: python bench.py squad (prints p50 and p90 milliseconds over 50 requests) import sys, os, time, statistics, torch from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM from tasks import load, to_chat, LABELS, MAX_NEW task = sys.argv[1]; dev = "cuda" if torch.cuda.is_available() else "cpu" dt = torch.bfloat16 if dev == "cuda" else torch.float32 _, te = load(task); te = te.select(range(min(50, len(te)))) sync = torch.cuda.synchronize if dev == "cuda" else (lambda: None) def timed(fn, items, warmup=3): for x in items[:warmup]: fn(x) # warm up kernels and caches first ms = [] for x in items: sync(); t = time.perf_counter(); fn(x); sync(); ms.append(1000 * (time.perf_counter() - t)) q = statistics.quantiles(ms, n=10) return f"p50 {statistics.median(ms):7.1f} ms p90 {q[8]:7.1f} ms" # encoder: one forward pass (the head adds almost nothing) bt = AutoTokenizer.from_pretrained("google-bert/bert-base-cased") enc = AutoModel.from_pretrained("google-bert/bert-base-cased", dtype=dt).to(dev).eval() def text_of(ex): return {"sst2": lambda: (ex["sentence"],), "mnli": lambda: (ex["premise"], ex["hypothesis"]), "conll": lambda: (" ".join(ex["tokens"]),), "squad": lambda: (ex["question"], ex["context"]), "cnndm": lambda: (ex["article"],)}[task]() @torch.no_grad() def run_enc(ex): enc(**bt(*text_of(ex), truncation=True, max_length=384, return_tensors="pt").to(dev)) print(f"{task:6s} encoder ", timed(run_enc, list(te))) # decoder: prefill the prompt, then generate (or score the labels in one batched forward) qt = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct") dec = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype=dt).to(dev).eval() @torch.no_grad() def run_dec(ex): p = qt.apply_chat_template([{"role": "user", "content": to_chat(task, ex)[0]}], tokenize=False, add_generation_prompt=True) if task in LABELS: x = qt([p + l for l in LABELS[task]], return_tensors="pt", padding=True, add_special_tokens=False).to(dev) dec(**x) else: x = qt(p, return_tensors="pt", add_special_tokens=False).to(dev) dec.generate(**x, max_new_tokens=MAX_NEW[task], do_sample=False, pad_token_id=qt.pad_token_id) print(f"{task:6s} decoder 0-shot ", timed(run_dec, list(te)))
Run it for each task, write the p50 numbers in the latency column, and use formula (159.12) with your measured time for the cost column. Then compare with Figure 4: the ratio between the decoder and the encoder is usually more stable than the absolute numbers.
Step 8: a toy you can run anywhere
Before the GPU runs, here is the whole idea in about 70 lines of plain PyTorch, small enough for any CPU. A toy language has filler words, “names”, the words “good” and “bad”, and an answer marker “@”. One shared Transformer encoder is trained with three heads at once: a sequence label (more “good” or more “bad”?), BIO tags for names, and a start and end score for the name after “@”. The losses are just added.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0); torch.set_num_threads(1) # A toy language: ids 0-9 are filler words, 10-19 names, 20 = "good", 21 = "bad", 22 = "@" (answer marker) V, T, D = 23, 16, 64 def make(n, g): x = torch.randint(0, 10, (n, T), generator=g) tags = torch.zeros(n, T, dtype=torch.long) # 0 = O, 1 = B-NAME, 2 = I-NAME for i in range(n): for _ in range(2): # two names of length 1 to 3 L = int(torch.randint(1, 4, (1,), generator=g)); s = int(torch.randint(1, T - L, (1,), generator=g)) if tags[i, s - 1:s + L + 1].any(): continue # keep names apart x[i, s:s + L] = torch.randint(10, 20, (L,), generator=g); tags[i, s] = 1; tags[i, s + 1:s + L] = 2 span = torch.zeros(n, 2, dtype=torch.long) # QA: the name right after "@" for i in range(n): st = int((tags[i] == 1).nonzero()[0]); en = st while en + 1 < T and tags[i, en + 1] == 2: en += 1 x[i, st - 1] = 22; span[i] = torch.tensor([st, en]) pol = torch.randint(0, 2, (n,), generator=g) # sentiment: 2 of one word, 1 of the other for i in range(n): free = ((tags[i] == 0) & (x[i] != 22)).nonzero().flatten() pos = free[torch.randperm(len(free), generator=g)[:3]] x[i, pos[:2]] = 20 if pol[i] == 1 else 21; x[i, pos[2]] = 21 if pol[i] == 1 else 20 return x, pol, tags, span class Model(nn.Module): # ONE encoder, three small heads def __init__(self): super().__init__() self.emb = nn.Embedding(V, D); self.pos = nn.Parameter(torch.randn(T, D) * 0.02) layer = nn.TransformerEncoderLayer(D, 4, 128, dropout=0.0, batch_first=True) self.enc = nn.TransformerEncoder(layer, 2) self.cls = nn.Linear(D, 2) # sequence classification: 2 labels self.tag = nn.Linear(D, 3) # token classification: O, B, I self.span = nn.Linear(D, 2) # extractive QA: a start score and an end score per token def forward(self, x): h = self.enc(self.emb(x) + self.pos) return self.cls(h.mean(1)), self.tag(h), self.span(h).unbind(-1) def spans(tags): # BIO -> set of (start, end) out, s = set(), None for t, y in enumerate(tags + [0]): if s is not None and y != 2: out.add((s, t - 1)); s = None if y == 1 or (y == 2 and s is None): s = t return out g = torch.Generator().manual_seed(0) xtr, ptr, ttr, str_ = make(2000, g); xte, pte, tte, ste = make(500, g) m = Model(); opt = torch.optim.AdamW(m.parameters(), lr=3e-3) for step in range(300): i = torch.randint(0, 2000, (64,), generator=g) c, tg, (s, e) = m(xtr[i]) loss = F.cross_entropy(c, ptr[i]) + F.cross_entropy(tg.reshape(-1, 3), ttr[i].reshape(-1)) \ + F.cross_entropy(s, str_[i, 0]) + F.cross_entropy(e, str_[i, 1]) opt.zero_grad(); loss.backward(); opt.step() m.eval() with torch.no_grad(): c, tg, (s, e) = m(xte) acc = (c.argmax(-1) == pte).float().mean().item() tok_acc = (tg.argmax(-1) == tte).float().mean().item() tp = npred = ngold = 0 for p, t in zip(tg.argmax(-1).tolist(), tte.tolist()): P, G = spans(p), spans(t); tp += len(P & G); npred += len(P); ngold += len(G) prec, rec = tp / npred, tp / ngold score = s[:, :, None] + e[:, None, :] # best span with start <= end <= start + 2 ok = torch.triu(torch.ones(T, T, dtype=torch.bool)) & ~torch.triu(torch.ones(T, T, dtype=torch.bool), 3) best = score.masked_fill(~ok, -1e9).flatten(1).argmax(-1) em = ((best // T == ste[:, 0]) & (best % T == ste[:, 1])).float().mean().item() print(f"params {sum(p.numel() for p in m.parameters()):,} heads {sum(p.numel() for q in (m.cls, m.tag, m.span) for p in q.parameters())}") print(f"final loss {loss.item():.3f}") print(f"sentiment accuracy {acc:.3f}") print(f"NER token accuracy {tok_acc:.3f} entity P {prec:.3f} R {rec:.3f} F1 {2*prec*rec/(prec+rec):.3f}") print(f"QA exact match {em:.3f}")
Output (PyTorch 2.8, CPU, about 5 seconds; the same on every run):
params 69,895 heads 455 final loss 0.451 sentiment accuracy 1.000 NER token accuracy 0.942 entity P 0.485 R 0.655 F1 0.557 QA exact match 0.894
Read it as a lab result. The three heads together have 455 weights, on top of a 69,440-weight shared encoder. Sentiment is easy and reaches 1.000. The NER head has a token accuracy of 0.942 but an entity F1 of 0.557: most tokens are filler (O), so token accuracy looks good while many name boundaries are still wrong. The same gap will appear, smaller, in your CoNLL run. The span head finds the exact answer 89.4% of the time.
Step 9: read your table
First compare your encoder numbers with the BERT paper, to know that your pipeline works. The paper reports for BERT-base [Certain]:
| Task | BERT-base in the paper | Their setting | Why yours will differ |
|---|---|---|---|
| SST-2 | 93.5 accuracy (GLUE test); 92.7 on dev in the ablation table | 67k training sentences | 5,000 training sentences |
| MNLI-m | 84.6 accuracy (GLUE test); 84.4 on dev | 392k training pairs | 5,000 pairs: expect a large gap |
| CoNLL-2003 | 92.4 test F1 (96.4 dev) | full training set, document context, cased model | 5,000 sentences, sentence context, text matching |
| SQuAD v1.1 | 80.8 EM, 88.5 F1 (dev) | 87.6k training questions | 5,000 questions |
Then write one sentence per row: which method wins, by how much, whether the paired interval excludes 0, and what it costs. Patterns to look for, not to assume:
- Classification with fixed labels (SST-2, MNLI). Fine-tuned encoders are strong and cheap here. A 0.5B prompted decoder is often weak on MNLI, because NLI needs the exact meaning of “neutral” versus “contradiction”; LoRA usually closes much of the gap [Likely].
- NER is where format matters most. The encoder tags every token in one pass. The prompted decoder must write every entity with the exact boundaries and type, and small models make many format and boundary errors [Likely].
- Extractive QA. A prompted decoder often writes a correct answer that is longer than the gold span, so its EM is low while F1 is fairer (Figure 3). LoRA teaches the answer length.
- Summarization is the only task where the decoder has no encoder rival in this lab. The question is whether it beats Lead-3.
- Cost. For the four tasks that an encoder can do, the encoder is cheaper per example at batch 1 in every case of Figure 4. A decoder earns its cost when one model must serve many tasks, when labels change often, or when the output is free text.
Where this is used
| Setting | Usual choice | Why |
|---|---|---|
| High-volume classification (spam, intent, toxicity, sentiment) | Fine-tuned encoder or distilled encoder | One pass, small, easy to batch; the label set is fixed |
| NER and PII redaction on large streams | Encoder token classifier | Exact spans and types; character offsets come for free from the tokenizer |
| Extraction with a changing schema (new fields every month) | Decoder LLM with JSON output, LoRA once the schema settles | A new field is a prompt change, not a new head (session 146) |
| Reading comprehension inside RAG | Decoder LLM generating with citations | Answers need to combine passages; extractive spans are too rigid (session 152) |
| NLI as a checker: does the answer follow from the source? | Encoder NLI model | Cheap faithfulness checks for summaries and RAG answers [Likely] |
| News, meeting and ticket summaries | Decoder LLM, often LoRA-tuned | Free text output; check against Lead-style baselines and with a judge |
| Video classification and action recognition | Frozen video encoder plus a linear head, or a VLM with label scoring | The same two families: a head on an encoder versus a generative model over video tokens [Likely] |
| Key-moment detection in video | Span head over frame or segment features | The same start and end scoring as extractive QA (159.3), over time instead of tokens [Likely] |
| Video-RAG and video summaries | VLM generating from retrieved clips and transcripts | Free-text answers; scored with ROUGE plus human or LLM judgment [Likely] |
Common mistakes
- Different training data per method. The encoder gets the full training set, the LLM gets 5,000 rows, or the shots come from the test set. Fix the subsets once, in one file.
- Token accuracy for NER. An all-
Otagger gets 82.5% token accuracy on CoNLL-2003 with zero entities found. Report entity F1, and say whether you match by position or by text. - Labels on every sub-token. If all sub-tokens of “Blackburn” get
I-PERwhile only the first is compared, or the reverse, your F1 silently changes. Label the first sub-token, use-100elsewhere, and map back to words the same way at test time. - Truncating SQuAD contexts instead of using windows. Answers past token 384 become impossible. Use
strideandreturn_overflowing_tokens, and map back with offsets. - Letting the parser decide the score. Free-form classification answers (“The sentiment is clearly positive!”) need a parser that will be wrong sometimes. Use label scoring (159.5), or a strict format and a count of unparseable outputs.
- Using
rougeLsumwithout newlines. Without one sentence per line it equalsrougeL, and your numbers will not match published ones. - Comparing latency with different batch sizes or prompts. The decoder’s cost depends heavily on the batch size and on the number of shots in the prompt (Figure 4). Report both next to every latency.
- Declaring a winner from a 1-point gap. On 872 or 1,000 examples the 95% interval is about ±2 points (159.9). Use the paired bootstrap (Algorithm 2).
Interview questions
Why is NER scored with entity-level F1 and not token accuracy?basic
How does BERT handle a word that is split into several sub-tokens for NER?core
word_ids() tells which word each sub-token belongs to. In training, the first sub-token of each word gets the word’s label, and the other sub-tokens get -100 so the loss ignores them (formula 159.2). At test time you take the prediction of the first sub-token of each word, so you get exactly one tag per word, then build entities from the BIO tags. Labeling all sub-tokens is also possible, but training and evaluation must use the same rule.Explain exact match and F1 in SQuAD. Why report both?basic
How does a BERT QA model deal with passages longer than 512 tokens?core
You use a decoder LLM for sentiment. Why score the label words instead of generating an answer?core
Why can’t you summarize CNN/DailyMail with BERT, and what is Lead-3?basic
Estimate the batch-1 latency of BERT-base versus a 0.5B decoder on an extractive QA example on a GPU with 300 GB/s.advanced
Why does batching help the decoder more than the encoder?advanced
Your LoRA-tuned decoder scores 1 point higher than BERT on 1,000 MNLI examples. Is it better?core
When would you choose a small decoder LLM over a fine-tuned encoder for a classic task?advanced
Why must the decoder use left padding in batched generation, while the encoder does not care?core
Exercises
- Compute the head size (159.4) for a BERT-large token classifier (\(d = 1{,}024\)) on CoNLL-2003, and for an MNLI head on a model with \(d = 896\).
- Start from the “Perfect” preset in Figure 2. Which change of a single tag gives the lowest F1? Which changes leave F1 at 1? Predict, then check by clicking.
- Compute EM and F1 for the prediction “the March of 1889” against the three gold answers of Figure 3.
- Using formula (159.11), estimate the batch-1 time for one CNN/DM summary on an A100 80GB (2,039 GB/s) with a 740.4-token prompt and 69.8 answer tokens. Ignore compute and the KV cache.
- Your SST-2 accuracies are 0.912 (BERT) and 0.897 (Qwen + LoRA) on 872 sentences. Compute both standard errors. What else do you need to decide if the difference is real?
- Extend
decoder.pyso that one LoRA adapter is trained on all five tasks at once (25,000 rows, shuffled). Compare it with the five single-task adapters. What do you gain at serving time?
Answerscheck
Key takeaways
- Five tasks, three output shapes: one label per input (159.1), one label per token (159.2), a span (159.3), plus free text for summaries. Encoder heads are tiny (159.4): 1,538 to 6,921 weights on a 108M model.
- A decoder LLM handles every task as text. Use label scoring (159.5) for classification and strict formats plus a parser for the rest; LoRA (159.6) teaches the format and the task with 1.78% of the weights.
- Use the right metric: accuracy for SST-2 and MNLI, entity F1 (159.7) for NER, EM and F1 (159.8) for SQuAD, ROUGE-Lsum for CNN/DM, and always the same test examples for all methods.
- On 1,000 examples a 95% interval is about ±2 points (159.9). Use paired tests before you call a winner.
- At batch 1, the encoder pays one pass; the decoder pays one pass per generated token (159.11). For QA on an L4 the lower bound is 0.57 ms versus 18.8 ms; batching narrows the gap.
- Pick the encoder for high-volume, fixed-label tasks; pick the decoder when tasks change, outputs are text, or one model must do everything.
Key formulas
References for this session
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Rajpurkar et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Hugging Face (code). PEFT library
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Rajpurkar et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Hugging Face (code). PEFT library
- Wang et al. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Tjong Kim Sang & De Meulder (2003). Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
- Williams et al. (2017). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
- Hermann et al. (2015). Teaching Machines to Read and Comprehend
- See, Liu & Manning (2017). Get To The Point: Summarization with Pointer-Generator Networks
- Qwen Team (2024). Qwen2.5 Technical Report
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Lin (2004). ROUGE: A Package for Automatic Evaluation of Summaries
- Hugging Face (code). TRL library