Fine-tuning for classic NLP tasks
Many real NLP jobs are not chat: decide if a review is positive, find the names in a contract, or write a short summary of a long text. This session shows how to fine-tune a pretrained model for the three classic task shapes (one label per text, one label per token, text in and text out), how to choose an encoder or a decoder for each, and how to measure the result honestly when the classes are imbalanced (F1, entity F1, ROUGE).
What you will learn
- The three task shapes: sequence classification (sentiment, topic, intent), token classification (named entities, tagging) and sequence-to-sequence (summarization, translation)
- How a classification head works: pool the hidden states ([CLS], mean, or last token), one linear layer, cross-entropy; and why it adds only about a thousand parameters
- Token classification with BIO tags, how to align word labels with subword tokens, and why entity-level F1 is the right metric
- Sequence-to-sequence fine-tuning with an encoder–decoder or a decoder-only model, and how ROUGE scores a summary
- Class imbalance: why accuracy lies, class weights, focal loss and threshold tuning, and the exact link between inverse-frequency weights and the decision threshold
- How to choose between a fine-tuned encoder, an encoder–decoder and a prompted or fine-tuned LLM for each task
Why this matters
In session 54 we taught a base model to answer like an assistant. But many real NLP jobs are not conversations. A bank wants to know if a complaint is about fraud. A search engine wants the names of people and places in a query. A newsroom wants a three-line summary of a long article. These are the classic NLP tasks, and fine-tuning a pretrained model is still the standard way to solve them well and cheaply.
All classic tasks come in three shapes:
- Sequence classification: one label for the whole text. Sentiment (is this review positive or negative?), topic (sports, politics, tech), intent (does the user want to cancel, pay or talk to a person?).
- Token classification: one label for every token. Named-entity recognition (NER: which words are a person, a place, a company?), part-of-speech tagging, finding personal data to hide.
- Sequence-to-sequence: text in, new text out. Summarization and translation.
For each shape, this session answers four questions: what to put on top of the pretrained model, which loss to train, which model family to use (encoder, decoder, or encoder–decoder), and which number to report. The last question is where most projects go wrong. With 1% positive examples, a model that always says “no” is 99% accurate and completely useless.
Real numbers set the scene. BERT-base (110M parameters) fine-tuned on the SST-2 sentiment set (67k training sentences) reached 93.5% test accuracy, and BERT-large (340M) reached 94.9%. On CoNLL-2003 NER, fine-tuned BERT-base and BERT-large reached 92.4 and 92.8 test F1. [Certain] Fine-tuning took only 2 to 4 epochs. These small models are still used in production because they are fast: a 110M encoder costs about 60 times less compute per token than a 7B LLM.
Three task shapes on one pretrained model
A pretrained transformer turns \(T\) input tokens into \(T\) hidden vectors \(H = (\mathbf{h}_1, \dots, \mathbf{h}_T)\), each of width \(d\) (768 for BERT-base, 4096 for Llama 2 7B). These vectors already encode a lot of meaning (session 12). Fine-tuning adds a small head on top and then trains the head and (usually) the whole body on labeled examples. The three shapes differ only in which hidden vectors the head reads and what it outputs.
How to read Figure 1
Tokens are at the bottom, the transformer body in the middle, the hidden states above it, and the head on top. Blue dots are the hidden states the head reads. In sequence classification only one vector is read (the [CLS] token); in token classification every token is read; in a decoder classifier the last real token is read, because only it has seen the whole text; in sequence-to-sequence the decoder’s normal language-model head predicts the next output word at every position. Notice how tiny the head is: for sentiment on BERT-base it is 1,538 numbers next to 110 million.
Sequence classification: one label per text
The head
We need one vector for the whole text, then a score per class. Three ways to get the vector (this is called pooling):
- [CLS] pooling (BERT): use the hidden state of the special first token [CLS]. Through self-attention it can read every other token. The BERT paper does exactly this. [Certain]
- Mean pooling: average the hidden states of all real (non-padding) tokens. Often as good or better for encoders (session 33).
- Last-token pooling (decoders): in a causal model, token \(t\) sees only tokens \(1..t\). Only the last real token has seen the whole text, so the head reads that one. Hugging Face’s
LlamaForSequenceClassificationdoes this and finds the last token from the padding mask, and Raschka’s GPT-2 spam classifier (RAS1 chapter 6) also reads the last token. [Likely]
The head is one linear layer, then softmax:
- Read the symbols
- \(\mathbf{h}_t \in \R^{d}\)
- the last-layer hidden state of token \(t\)
- \(\operatorname{pool}\)
- pick [CLS], or average the real tokens, or pick the last real token
- \(W \in \R^{K \times d}, \mathbf{b} \in \R^{K}\)
- the new head weights; \(K\) is the number of classes. These start random; everything else starts from the pretrained model
- \(\mathbf{z}, \mathbf{p} \in \R^{K}\)
- the class scores (logits) and the class probabilities
The loss is cross-entropy with the true class \(y\) (session 3). Its gradient with respect to the logits is simple, and it is what flows back into the whole network:
Here \(\mathbf{e}_y\) is the one-hot vector of the true class. The gradient pushes the true class up by \(1 - p_y\) and every other class down by its own probability.
BERT-base, \(K = 2\) classes (positive, negative). The head has \(2 \times 768 = 1{,}536\) weights plus 2 biases: 1,538 new parameters, about 0.0014% of the 110M body.
For “the film was great” the head gives logits \(\mathbf{z} = (2.0,\ 0.5)\). Softmax: \(\mathbf{p} = (0.818,\ 0.182)\). The true class is positive, so \(\mathcal{L} = -\log 0.818 = 0.201\). The gradient on the logits is \(\mathbf{p} - \mathbf{e}_y = (-0.182,\ +0.182)\): raise the positive score, lower the negative one. Backpropagation carries this signal through \(W\) into the [CLS] vector and then into all 12 layers.
Multi-label: several labels per text
Topic tags often overlap: a news story can be about both sports and business. Then the classes are not exclusive and softmax is wrong, because it forces the probabilities to sum to 1. Use one independent sigmoid per label and binary cross-entropy:
At inference, each label is on if \(\sigma(z_k) \gt \tau_k\). The thresholds \(\tau_k\) are tuned per label on the validation set, not fixed at 0.5. In PyTorch this is BCEWithLogitsLoss.
The fine-tuning recipe
The BERT paper fine-tuned all weights, with dropout 0.1, and found these values work well across tasks: batch size 16 or 32, Adam learning rate \(5\times10^{-5}\), \(3\times10^{-5}\) or \(2\times10^{-5}\), and 2, 3 or 4 epochs. They note that large datasets (100k+ examples) were much less sensitive to these choices than small ones, and that BERT-large was sometimes unstable on small sets, so they ran several random restarts and kept the best on the dev set. [Certain] The learning rate is about 10 times lower than in pretraining: the body is already good, and big steps destroy what it knows (catastrophic forgetting). A short linear warmup and linear decay are standard (session 4).
Token classification: one label per token
BIO tags
In NER we want spans: “Ada Lovelace” is a person, “London” is a place. We turn spans into one label per word with the BIO scheme: B-X marks the beginning of an entity of type X, I-X a word inside it, and O a word outside any entity. With \(E\) entity types there are \(2E + 1\) tags. CoNLL-2003 has 4 types (PER, LOC, ORG, MISC), so 9 tags.
| Word | Ada | Lovelace | met | Charles | Babbage | in | London |
|---|---|---|---|---|---|---|---|
| Tag | B-PER | I-PER | O | B-PER | I-PER | O | B-LOC |
Why B at all? Without it, “gave Ada Charles’s book” with two people next to each other would look like one long name. B marks where a new entity starts.
Subwords: whose label is it?
Labels are given per word, but the model sees subword tokens (session 8). Suppose the tokenizer splits “Lovelace” into Love + ##lace (WordPiece marks a continuation piece with ##). Which token gets the label? The common answer, used in the BERT paper, is: put the word’s label on its first subword and ignore the rest. [Certain] “Ignore” means the label value −100, which PyTorch’s cross-entropy skips by default (ignore_index=-100). Special tokens ([CLS], [SEP]) and padding also get −100. Fast Hugging Face tokenizers give a word_ids() list that maps each token back to its word, which makes this alignment easy.
- Read the symbols
- \(p_{t,\,y_t}\)
- the probability the head gives to the true tag of token \(t\): the same head as (55.1), applied to every \(\mathbf{h}_t\) separately, with no pooling
- \(m_t\)
- the mask: 1 for tokens with a real label, 0 for continuation pieces, special tokens and padding
This is the same loss-mask idea as in SFT, where only assistant tokens count (session 54). At inference, predict a tag for every token, keep the tag of each word’s first subword, and convert the BIO tags back into spans.
Measure entities, not tokens
Most tokens are O, so token accuracy looks high even when the model finds few entities. The standard NER metric (the CoNLL conlleval script, and the seqeval library) counts entities: a predicted entity is correct only if its type and both its start and end match a true entity exactly. Then precision, recall and F1 are computed over entities (formula 55.8 below). Try it in Figure 2.
How to read Figure 2
Start with “A weak model”: 7 of 9 tags are right (77.8% token accuracy), but only 1 of the 3 entities is found exactly, so entity F1 is 0.333. One wrong tag (“Babbage” as O) cuts the second name short, and one wrong type (London as ORG) loses the place. Now try “Split one name”: a single wrong B-PER turns one true entity into two wrong ones, so it costs a false negative and two false positives. Finally, set “Charles” to I-PER instead of B-PER: CoNLL scoring treats an I- after O as the start of an entity, so the score stays perfect even though one tag is “wrong”.
Why encoders are the default for tagging
The tag of a word often depends on the words after it. In “Washington said”, Washington is a person; in “Washington is cold”, a place. An encoder’s hidden state \(\mathbf{h}_t\) has read the whole sentence, left and right. A decoder’s hidden state at position \(t\) has read only tokens \(1..t\), so it must tag “Washington” before seeing the next word. The code later in this session shows the effect on a toy task where a name is a person only if “said” follows: a bidirectional tagger gets entity F1 0.999, the same network with a causal mask gets 0.268. [Certain: our run] Older systems added a CRF layer on top to enforce valid tag sequences (no I-LOC right after B-PER); BERT did well without one. [Certain]
Sequence-to-sequence: summarization and translation
Here the output is new text \(y = (y_1, \dots, y_m)\), so we train the model to generate it, one token at a time, with teacher forcing: at each step the decoder sees the true previous output tokens.
- Read the symbols
- \(x\)
- the source: the article to summarize, or the sentence to translate
- \(y_{\lt t}\)
- the true target tokens before position \(t\)
- \(p_\theta\)
- the model’s next-token distribution over the whole vocabulary; no new head is needed, the language-model head is reused
Two families can do this:
- Encoder–decoder (BART, T5, and translation models like mBART): the encoder reads \(x\) bidirectionally once; the decoder writes \(y\) and reads the encoder through cross-attention (session 24). The loss covers only \(y\) by construction. BART was reported to work especially well when fine-tuned for generation tasks such as summarization. [Likely]
- Decoder-only (GPT, Llama, Qwen): concatenate the source, a separator or prompt, and the target into one sequence, and mask the loss so only target tokens count. This is exactly SFT with a fixed prompt. It is the natural choice when you already serve an LLM, or when the task needs world knowledge.
At inference we decode with beam search or greedy decoding, usually with a length limit; summarizers also often block repeated n-grams. [Likely]
Scoring a summary: ROUGE
We cannot check a summary with exact match: many different summaries are good. ROUGE (Lin, 2004) compares the words of the candidate summary \(c\) with a human reference summary \(r\). ROUGE-N counts shared n-grams (sequences of \(n\) words):
\(R_N\) is a recall: what share of the reference’s n-grams appear in the candidate. The \(\min\) stops a candidate from scoring by repeating a word many times. ROUGE-L uses the longest common subsequence (LCS): the longest list of words that appears in both texts in the same order, not necessarily next to each other:
Lin’s paper defines ROUGE-N as a recall and ROUGE-L with a weighted F-measure; common libraries (such as Google’s rouge-score) report precision, recall and F1 for each, and most papers quote the F1. [Likely] Check which one a paper reports before you compare numbers. Figure 3 computes all three.
Reference:
rouge-score package does by default. Green words also appear in the reference; underlined words form the longest common subsequence.How to read Figure 3
“Good” drops one word (“new”) and scores high everywhere. “Same words, wrong order” keeps every word, so ROUGE-1 is unchanged at 0.933, but ROUGE-2 and ROUGE-L fall (0.462 and 0.400): word order only shows up in the longer measures. Its meaning is nonsense (“the budget approved the council”) yet ROUGE-1 cannot tell. “Paraphrase” says the same thing in other words and scores 0 on all three. Type your own candidate and watch the scores move.
For translation the classic metric is BLEU (n-gram precision with a brevity penalty), with chrF (character n-grams) and learned metrics such as COMET used more and more because they agree better with human judgments. [Likely] For both tasks, a small human or LLM-judge evaluation is worth more than a 0.5-point ROUGE gain.
Class imbalance and the right metric
Why accuracy lies
Real labels are rarely balanced. Fraud, toxic posts and rare intents may be 1–10% of the data. With 900 negatives and 100 positives, a model that always says “negative” has 90% accuracy. It finds no positives at all. We need numbers that look at each class. For one class, count true positives (TP), false positives (FP) and false negatives (FN):
- Read the symbols
- \(P\) (precision)
- of the items the model called positive, the share that really are
- \(R\) (recall)
- of the truly positive items, the share the model found
- \(F_1\)
- the harmonic mean of the two; it is low if either one is low
With \(K\) classes, there are two ways to combine per-class scores:
Macro F1 gives every class the same weight, so rare classes count as much as common ones. Micro F1 pools all counts, so common classes dominate; for single-label multi-class problems micro F1 equals accuracy. Report macro F1 (and per-class numbers) when the rare classes matter. NER entity F1 is a micro F1 over entities.
900 negatives, 100 positives, the model always predicts negative. \(\text{Accuracy} = 900/1000 = \mathbf{90\%}\). For the positive class: \(TP = 0\), so \(F_1 = 0\). For the negative class: \(P = 0.9\), \(R = 1\), \(F_1 = 2(0.9)(1)/1.9 = 0.947\). \(\text{Macro } F_1 = (0.947 + 0)/2 = \mathbf{0.474}\). Accuracy says 90%; macro F1 says the model is worse than a coin for the class we care about.
Fixing the training: class weights and focal loss
The model sees 9 negatives for every positive, so the loss is dominated by negatives. Three common fixes:
- Class weights. Multiply each example’s loss by a weight for its class. The usual choice is inverse frequency:
\[ w_c = \frac{N}{K\, N_c}, \qquad \mathcal{L} = -\frac{\sum_i w_{y_i} \log p_{i,\,y_i}}{\sum_i w_{y_i}} \tag{55.10} \]\(N\) is the number of training examples, \(N_c\) the number in class \(c\). With 900/100: \(w_{\text{neg}} = 1000/1800 = 0.556\), \(w_{\text{pos}} = 1000/200 = 5\). Each positive now counts 9 times as much as a negative. In PyTorch:
F.cross_entropy(logits, y, weight=w); the mean divides by the sum of the weights, as in (55.10). - Resampling. Show positives more often (oversampling) or negatives less often (undersampling). Similar effect to weights, but changes the batches the model sees.
- Focal loss (Lin et al., 2017, from object detection): down-weight examples the model already gets right.
\[ \mathcal{L}_{\text{focal}} = -(1 - p_y)^{\gamma} \log p_y \tag{55.11} \]With \(\gamma = 2\) (the value used in the paper [Certain]), an easy example with \(p_y = 0.9\) has its loss multiplied by \(0.1^2 = 0.01\): from 0.105 to 0.00105. A hard one with \(p_y = 0.1\) keeps 81% of its loss. The many easy negatives stop drowning out the few hard cases.
The hidden link: weights move the threshold
Class weights do not make the model “smarter”; mostly they move its decision boundary. Suppose the model is flexible enough to output the best possible probabilities. For an input whose true chance of being positive is \(p\), plain cross-entropy is minimized by predicting \(q = p\). With weights \(w_0, w_1\), the expected loss \(-w_1 p \log q - w_0 (1-p) \log(1-q)\) is minimized at
With inverse-frequency weights (55.10), \(w_0/(w_0 + w_1) = \pi\), the share of positives. So a weighted model that uses the usual 0.5 cut behaves like an unweighted model that says “positive” whenever \(p \gt \pi\). With 10% positives, that is \(p \gt 0.1\). This explains why weights raise recall and lower precision, and why they can hurt F1: in the code below, plain CE reached F1 0.737 (P 1.000, R 0.584), weighted CE only 0.539 (P 0.447, R 0.680). [Certain: our run] The robust practice is: train (with or without weights), then tune the decision threshold on the validation set for the metric you report. Figure 4 lets you do this.
How to read Figure 4
At \(\pi = 50\%\), threshold 0 is best and accuracy and F1 agree (0.894). Now set \(\pi = 5\%\) and keep the threshold at 0 (this is what inverse-frequency weights do, formula 55.12): recall stays 0.894, but there are now 1,004 false positives for 447 true ones, so precision is 0.308 and F1 0.458. “Plain CE” moves the threshold to \(s = 1.178\), the point where the true probability is 0.5: F1 rises to 0.632. “Best F1” finds \(s = 0.89\) with F1 0.658. The classifier never changed; only the threshold did. Also watch accuracy: always predicting negative already gives 95%.
Encoder or decoder for each task
All three families can do all three tasks. The question is which one gives the best quality per dollar and per millisecond for your data. A practical guide:
| Task | Good default | Why | When to choose something else |
|---|---|---|---|
| Sentiment, topic, intent (labeled data available) | Fine-tuned encoder (BERT, RoBERTa, DeBERTa, ModernBERT) + [CLS] or mean head | Bidirectional reading, small and fast, easy to batch; one forward pass, no generation | Very few labels or labels change often: prompt an LLM (zero- or few-shot), or fine-tune an LLM with a classification head or LoRA |
| Many labels, multi-label topics | Encoder + sigmoid head (55.3) | Independent per-label thresholds | Labels are long descriptions: embed text and labels and compare |
| NER, tagging, personal-data detection | Fine-tuned encoder + token head | Each token sees both sides; exact spans; very fast | Open-ended extraction (“list every obligation in this contract”): an LLM generating JSON |
| Summarization | Encoder–decoder (BART, T5) or an instruction-tuned LLM with SFT | Encoder reads the long input once, bidirectionally; LLMs bring more world knowledge and fluency | Very long inputs: a long-context LLM, or split and merge |
| Translation | Encoder–decoder trained on parallel text, or a large multilingual LLM | Encoder–decoders are strong and efficient for a fixed language pair | Many languages, style control, or document-level context: an LLM |
Some numbers to reason with. Per token, the forward pass costs about \(2N\) FLOPs (session 25): \(2.2\times10^{8}\) for a 110M encoder, \(1.4\times10^{10}\) for a 7B LLM, about 60 times more. For a sentence classifier the encoder also needs one pass and no generation. BERT-style encoders were trained with at most 512 positions [Certain]; ModernBERT supports 8,192 tokens [Certain], which removes much of the old “split long documents” work. On narrow tasks with thousands of labels, fine-tuned small encoders often match or beat prompted LLMs; with only a handful of labels, a prompted LLM usually wins. [Likely] session 144 goes deeper into zero-shot classification with label words and calibration.
Putting it together
- input: pretrained body, labeled train / validation / test sets, task shape (sequence or token), number of labels \(K\)
- tokenize; for token tasks, map word labels to the first subword and set every other label to −100 // formula (55.4)
- add a head \(W \in \R^{K \times d}\), \(\mathbf{b}\), small random init; choose pooling: [CLS], mean, or last real token // formula (55.1)
- optionally compute class weights \(w_c = N/(K N_c)\) from the train set only // formula (55.10)
- AdamW, peak learning rate \(2\times10^{-5}\) to \(5\times10^{-5}\), short warmup, linear decay, 2 to 4 epochs, dropout 0.1
- for each batch: logits \(\leftarrow\) head(pool(body(tokens))); loss \(\leftarrow\) (weighted) cross-entropy; backward; clip; step
- after each epoch, evaluate on validation with the task metric (macro F1, entity F1); keep the best checkpoint
- tune the decision threshold(s) on validation for that metric // formula (55.12), Figure 4
- report the metric once on the test set, with the threshold fixed
- input: gold and predicted BIO tag lists for each sentence (one tag per word)
- function spans(tags): walk left to right; close the open entity when the tag is O, a B-, or an I- of another type; open a new entity at every B-X, and at an I-X that does not continue an open X
- \(G \leftarrow\) spans(gold) as (type, start, end) triples; \(\hat G \leftarrow\) spans(pred)
- \(TP \mathrel{+}= |G \cap \hat G|\), \(n_{\text{pred}} \mathrel{+}= |\hat G|\), \(n_{\text{gold}} \mathrel{+}= |G|\) // exact match of type and both ends
- return \(P = TP/n_{\text{pred}}\), \(R = TP/n_{\text{gold}}\), \(F_1 = 2TP/(n_{\text{pred}} + n_{\text{gold}})\) // formula (55.8), micro over entities
The PyTorch version below is deterministic and runs on a CPU in under a minute. It builds a tiny 2-layer encoder, fine-tunes it for (1) imbalanced sentiment with plain and weighted cross-entropy, and (2) NER where a name is a person only if the word “said” follows, once with full attention and once with a causal mask. The data are synthetic, so the numbers only illustrate the ideas.
import warnings; warnings.filterwarnings("ignore") import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) V, D, T = 40, 64, 16 # vocab size, width, max length PAD, CLS, SAID = 0, 1, 16 # special ids; 16 plays the word "said" POS, NEG = [2, 3, 4, 5], [6, 7, 8, 9] # "good" words and "bad" words IGNORE = -100 # label value the loss skips class Encoder(nn.Module): def __init__(self, n_labels, causal=False): super().__init__() self.causal = causal self.emb, self.pos = nn.Embedding(V, D), nn.Embedding(T, D) layer = nn.TransformerEncoderLayer(D, 4, 4 * D, dropout=0.1, batch_first=True, norm_first=True) self.body = nn.TransformerEncoder(layer, 2) self.head = nn.Linear(D, n_labels) # the only new layer: W and b def forward(self, ids): # returns logits for every position mask = nn.Transformer.generate_square_subsequent_mask(ids.size(1)) if self.causal else None h = self.body(self.emb(ids) + self.pos.weight[: ids.size(1)], mask=mask, src_key_padding_mask=ids.eq(PAD)) return self.head(h) # (B, T, n_labels) def train(model, X, Y, steps=400, seq=True, weight=None): opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01) g = torch.Generator().manual_seed(2) for _ in range(steps): i = torch.randint(0, len(X), (64,), generator=g) out = model(X[i]) if seq: # sequence classification: read the [CLS] position, formula (55.2) with (55.6) loss = F.cross_entropy(out[:, 0], Y[i], weight=weight) else: # token classification: every position, skip IGNORE, formula (55.4) loss = F.cross_entropy(out.reshape(-1, out.size(-1)), Y[i].reshape(-1), ignore_index=IGNORE) opt.zero_grad(); loss.backward(); opt.step() return model.eval() def prf(pred, y): # formula (55.8), positive class tp = ((pred == 1) & (y == 1)).sum().item() n_pred, n_true = (pred == 1).sum().item(), (y == 1).sum().item() return tp / max(1, n_pred), tp / max(1, n_true), 2 * tp / max(1, n_pred + n_true) # ---- 1) sentiment with about 10% positive documents ---- def sentiment_data(n, g): X = torch.randint(17, V, (n, T), generator=g); X[:, 0] = CLS y = (torch.rand(n, generator=g) < 0.1).long() for k in range(n): good = int(torch.randint(0, 3, (1,), generator=g)) + int(y[k]) bad = int(torch.randint(0, 3, (1,), generator=g)) + 1 - int(y[k]) spots = torch.randperm(T - 1, generator=g)[: good + bad] + 1 X[k, spots] = torch.tensor([POS[j % 4] for j in range(good)] + [NEG[j % 4] for j in range(bad)], dtype=torch.long) return X, y g = torch.Generator().manual_seed(1) Xtr, ytr = sentiment_data(3000, g); Xte, yte = sentiment_data(2000, g) cnt = torch.bincount(ytr, minlength=2).float() w = cnt.sum() / (2 * cnt) # formula (55.6) print("class weights", [round(v, 2) for v in w.tolist()]) print("always negative : acc %.3f P %.3f R %.3f F1 %.3f" % ((yte == 0).float().mean(), *prf(torch.zeros_like(yte), yte))) for name, wt in [("plain CE", None), ("weighted CE", w)]: torch.manual_seed(0) m = train(Encoder(2), Xtr, ytr, weight=wt) with torch.no_grad(): p = m(Xte)[:, 0].argmax(-1) print("%-15s: acc %.3f P %.3f R %.3f F1 %.3f" % (name, (p == yte).float().mean(), *prf(p, yte))) # ---- 2) NER: a run of name words is an entity only if "said" comes right after it ---- TAGS = ["O", "B-PER", "I-PER"] def ner_data(n, g): X = torch.randint(17, V, (n, T), generator=g); X[:, 0] = CLS Y = torch.zeros(n, T, dtype=torch.long) for k in range(n): t = 1 + int(torch.randint(0, 3, (1,), generator=g)) while t < T - 4: L = 1 + int(torch.randint(0, 3, (1,), generator=g)) X[k, t:t + L] = torch.randint(10, 16, (L,), generator=g) # name words if torch.rand(1, generator=g) < 0.5: # "... said": a person X[k, t + L] = SAID; Y[k, t] = 1; Y[k, t + 1:t + L] = 2 t += L + 2 + int(torch.randint(0, 3, (1,), generator=g)) Y[:, 0] = IGNORE # no label for [CLS] return X, Y def spans(tags): # BIO tags -> set of (start, end) out, start = set(), None for i, t in enumerate(tags + ["O"]): if start is not None and t != "I-PER": out.add((start, i - 1)); start = None if t == "B-PER" or (t == "I-PER" and start is None): start = i return out Xtr, Ytr = ner_data(3000, g); Xte, Yte = ner_data(1000, g) for name, causal in [("bidirectional", False), ("causal (decoder)", True)]: torch.manual_seed(0) m = train(Encoder(3, causal), Xtr, Ytr, seq=False) with torch.no_grad(): P = m(Xte).argmax(-1) tp = n_pred = n_gold = 0 for k in range(len(Xte)): gold = spans([TAGS[v] for v in Yte[k, 1:].tolist()]) pred = spans([TAGS[v] for v in P[k, 1:].tolist()]) tp += len(gold & pred); n_pred += len(pred); n_gold += len(gold) keep = Yte != IGNORE acc = (P[keep] == Yte[keep]).float().mean().item() print("%-17s: token acc %.3f entity P %.3f R %.3f F1 %.3f" % ( name, acc, tp / max(1, n_pred), tp / n_gold, 2 * tp / (n_pred + n_gold)))
Output (PyTorch 2.8, CPU):
class weights [0.55, 5.23] always negative : acc 0.901 P 0.000 R 0.000 F1 0.000 plain CE : acc 0.959 P 1.000 R 0.584 F1 0.737 weighted CE : acc 0.886 P 0.447 R 0.680 F1 0.539 bidirectional : token acc 1.000 entity P 0.998 R 0.999 F1 0.999 causal (decoder) : token acc 0.842 entity P 0.237 R 0.309 F1 0.268
Read it line by line. “Always negative” is 90.1% accurate with F1 0. Plain CE is careful: every positive it predicts is right (P 1.000) but it finds only 58% of them. Weighted CE finds more (R 0.680) at the cost of many false alarms, as formula (55.12) predicts, and here its F1 is lower; tuning the threshold would let you pick any point between. The causal tagger cannot see “said” when it tags a name, so it can only guess; token accuracy (0.842) still looks fine while entity F1 collapses to 0.268.
Where this is used
| Application | Task shape | Typical model and metric |
|---|---|---|
| Review sentiment, support-ticket routing, intent detection in voice assistants | Sequence classification | Fine-tuned encoder; macro F1, per-intent recall; strict latency budget |
| Spam, toxicity and policy moderation | Multi-label sequence classification, heavy imbalance | Encoder + sigmoid heads, per-label thresholds; precision at a fixed recall |
| NER for search, personal-data redaction, medical and legal entity extraction | Token classification | Encoder + BIO head; entity-level F1 (seqeval) |
| Invoice and form field extraction | Token classification over text plus layout | Layout-aware encoders or VLMs (VLM chapter 8); field-level F1 |
| News, meeting and call summaries | Sequence-to-sequence | BART / T5 or an SFT-tuned LLM; ROUGE plus human or LLM-judge checks |
| Machine translation | Sequence-to-sequence | Encoder–decoder or multilingual LLM; BLEU, chrF, COMET |
| Video classification and action recognition | Sequence classification over frame or patch tokens | Video encoder + [CLS] or mean-pooled head; top-1 accuracy, macro F1 for rare actions |
| Key-moment detection and temporal action segmentation (goals, fouls, scene cuts) | Token classification over time: one BIO-style tag per frame or clip | Bidirectional temporal encoder for offline video (it sees what comes after); event-level F1 with a time-overlap rule, just like entity F1 |
| Video captioning and video summarization | Sequence-to-sequence (video in, text out) | Video encoder + text decoder or Video-LLM; ROUGE, CIDEr, human checks |
| Query routing in Video-RAG and search (“find a clip” vs “answer a question”) | Sequence (intent) classification | Small encoder in front of the expensive model; must be fast |
Common mistakes
- Reporting accuracy on imbalanced data. Always report the majority-class baseline next to it, plus per-class precision, recall and macro F1.
- Misaligned NER labels. After subword tokenization the token list is longer than the word list. Map labels with
word_ids(); put −100 on continuation pieces, special tokens and padding. Off-by-one shifts here give silently bad models. - Token-level F1 for NER. It rewards getting the many
Otags right. Use entity-level F1 with exact span matching (seqeval or conlleval). - Decoder classifier reading the wrong position. With right padding, position −1 is a pad token. Read the last real token, and make sure the tokenizer has a pad token set (many LLM tokenizers do not by default).
- Tuning thresholds or picking checkpoints on the test set. Use the validation set; touch the test set once.
- Too high a learning rate. \(10^{-3}\) on a pretrained body often collapses it to one class. Start at \(2\times10^{-5}\) to \(5\times10^{-5}\) for full fine-tuning.
- Silent truncation. A 512-token encoder cuts long documents; if the label depends on the end (for example the verdict in a long review), split into chunks and combine, or use a long-context model.
- Softmax for multi-label data, or loss on the source in decoder-only seq2seq. Use sigmoid + BCE for overlapping labels, and mask the prompt so only target tokens are trained.
Interview questions
How do you fine-tune BERT for sentiment classification?basic
[CLS] pooling, mean pooling, last-token pooling: when do you use which?core
How do you assign NER labels when a word is split into several subword tokens?core
Why is token accuracy a poor metric for NER, and what do you report instead?core
Why are encoders preferred to decoders for token classification?core
Your positive class is 1% of the data. What do you do?core
What do inverse-frequency class weights do to the decision threshold?advanced
Macro F1 vs micro F1?basic
What does ROUGE measure, and what does it miss?core
Encoder–decoder or decoder-only for summarization?core
Fine-tuning BERT-large on a small dataset is unstable: some runs fail completely. Why and what helps?advanced
In production, would you prompt an LLM or fine-tune an encoder for intent classification?advanced
Exercises
- How many new parameters does a classification head add for 20 intents on a model with \(d = 4096\)? Check with Figure 1.
- A training set has 950 negatives and 50 positives. Compute the inverse-frequency weights (55.10). By formula (55.12), what probability threshold does the weighted model effectively use?
- Gold tags: B-PER I-PER O B-PER I-PER O B-LOC O O. Predicted: B-PER B-PER O B-PER I-PER O B-LOC O O (one tag wrong). Compute token accuracy and entity P, R, F1. Check with Figure 2.
- Reference “the cat sat on the mat”, candidate “the cat was on the mat”. Compute ROUGE-1, ROUGE-2 and ROUGE-L F1.
- With 90% negatives, what are accuracy, micro F1 and macro F1 of the model that always predicts negative?
- With focal loss and \(\gamma = 2\), what is the loss of an example with \(p_y = 0.9\)? With \(p_y = 0.5\)? Compare with plain cross-entropy.
Answerscheck
Key takeaways
- Classic tasks have three shapes: one label per text, one label per token, or text in and text out. The pretrained body is the same; only the head and the loss change (55.1, 55.4, 55.5).
- A classification head is one linear layer (\(Kd + K\) parameters). Pool with [CLS] or the mean in an encoder, with the last real token in a decoder. Fine-tune everything with a small learning rate for a few epochs.
- NER: BIO tags, label the first subword, −100 for the rest, and score with entity-level F1, not token accuracy. Encoders win because each token sees its right context.
- Seq2seq: teacher-forced cross-entropy on target tokens only, with an encoder–decoder or a decoder-only LLM. ROUGE (55.6, 55.7) measures word overlap, not meaning or facts.
- With imbalance, accuracy lies; use precision, recall, F1 and macro F1 (55.8, 55.9). Class weights (55.10) and focal loss (55.11) help training, but weights mainly move the threshold to \(p \gt \pi\) (55.12). Tune the threshold on validation.
Key formulas
References for this session
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Raschka, Build a Large Language Model (From Scratch), chapter 6: Fine-tuning for classification
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 4: Logistic Regression and Text Classification
- Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Lin (2004). ROUGE: A Package for Automatic Evaluation of Summaries
- Papineni et al. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation
- Lin et al. (2017). Focal Loss for Dense Object Detection
- Tjong Kim Sang & De Meulder (2003). Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
- Warner et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
- Rei et al. (2020). COMET: A Neural Framework for MT Evaluation
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models