LLM & VLM courseBlog
33Track 3: Pretraining and scale · session 3 of 22

Encoders for understanding: BERT, RoBERTa, ALBERT and the CLS token

Decoder-only models generate text. Encoder-only models read a whole text at once and turn it into vectors for classification, tagging and search. This session takes apart the three classic encoders: BERT (the input format with [CLS] and [SEP], masked language modeling and next-sentence prediction), RoBERTa (the same model trained better), and ALBERT (the same idea with far fewer parameters). We end with the question every practitioner meets: how to get one vector for a whole text, with the [CLS] token or with mean pooling, and how a classification head works.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 9, 11, 18 and 23 (special tokens, embeddings, masks, the three transformer families)

What you will learn

  • The BERT input format: [CLS] A [SEP] B [SEP], and how token, segment and position embeddings are added
  • BERT's two pretraining tasks, masked language modeling and next-sentence prediction, with the real training recipe
  • What RoBERTa changed (dynamic masking, no NSP, full-sentence inputs, big batches, 10 times more data, byte-level BPE) and the evidence for each change
  • ALBERT's factorized embedding and cross-layer parameter sharing, with exact parameter counts, and why fewer parameters does not mean less compute
  • Why NSP is too easy and how sentence-order prediction fixes it
  • The [CLS] vector vs mean pooling, the padding trap, and a classification head as one linear layer plus softmax
  • Why raw BERT vectors are poor for similarity search, and how Sentence-BERT fixes it

Why this matters

A chat model writes text one token at a time. But a large share of real NLP work does not need any writing. It needs reading: is this review positive, which product category is this, which words are names of people, which of these 10 million documents answers my question? For these jobs, the classic tool is an encoder: a transformer that reads the whole text at once, in both directions, and gives back vectors.

In session 23 you met the encoder-only family and its training task, masked language modeling (MLM). This session goes one level deeper into the three models that defined the family. BERT (2018) set the recipe: a special input format, two pretraining tasks, and “fine-tune the whole model plus one small layer” for each task. RoBERTa (2019) kept the model exactly the same and showed that BERT was simply under-trained. ALBERT (2019) kept the idea and cut the parameter count by up to 18 times. Each one teaches a lesson that still holds for today’s LLMs: the training recipe matters as much as the architecture, and the number of parameters is not the same as the amount of compute.

These models are not history. Encoders are still used in production for classification, named-entity recognition, reranking, and as the text side of retrieval systems, because one forward pass of a 110M-parameter model is cheap. [Certain] ModernBERT (2024) is a recent encoder with the same basic plan and an 8,192-token context. [Certain] In the previous session, session 32, you followed one training step of a decoder. Here the step is the same (forward, loss, backward, update); only the input format, the mask and the loss change.

The BERT input: [CLS] A [SEP] B [SEP]

BERT was designed so that one pretrained model could be fine-tuned for many tasks. Some tasks have one text (sentiment of a review). Others have two (does sentence B follow from sentence A? does this passage answer this question?). So BERT’s input format can hold one or two pieces of text, which the paper calls “sentence A” and “sentence B”. A “sentence” here means any span of text, often several real sentences. [Certain]

  • [CLS] (“classification”) is always the first token. It is not a word. Its final vector is meant to be the summary of the whole input, used for classification.
  • [SEP] (“separator”) closes sentence A, and closes sentence B if there is one.
  • A segment embedding (also called token type) tells every token whether it belongs to A or to B. In Hugging Face this is the token_type_ids input: 0 for A, 1 for B.

The vector that enters the first layer is the sum of three learned vectors, followed by a LayerNorm:

\[ \mathbf{x}_i = \operatorname{LayerNorm}\big( E_{\text{tok}}[t_i] + E_{\text{seg}}[s_i] + E_{\text{pos}}[i] \big) \tag{33.1} \]
Read the symbols
\(t_i\)
the token ID at position \(i\) (for example 101 for [CLS] in bert-base-uncased)
\(s_i\)
the segment of position \(i\): 0 for sentence A, 1 for sentence B
\(E_{\text{tok}}, E_{\text{seg}}, E_{\text{pos}}\)
three learned tables with \(30{,}522\), \(2\) and \(512\) rows of size \(H = 768\) in BERT-base; positions are learned absolute positions (session 13)
\(\mathbf{x}_i\)
the input vector of position \(i\), one row of the matrix that enters layer 1

Figure 1 shows the real token IDs that the Hugging Face tokenizers produce for the pair “My dog is cute” / “He likes playing” (the example from the BERT paper), for BERT, RoBERTa and ALBERT.

sentence Asentence Bspecial token
Figure 1. The encoder input, column by column. Each column is one position. Its input vector is the sum of a token vector (from the ID), a segment vector (A or B) and a position vector, as in formula (33.1). All IDs come from the real bert-base-uncased, roberta-base and albert-base-v2 tokenizers.
How to read Figure 1

Switch between the four formats. BERT lower-cases the text (“My” becomes “my”, ID 2026) and marks sentence B with segment 1. RoBERTa uses <s> and </s> instead of [CLS] and [SEP], puts two </s> between the sentences, and has no segment embedding (its config has type_vocab_size = 1), so the readout shows no segment row. Its position numbers start at 2, not 0, because the Hugging Face RoBERTa code counts positions from the padding index plus one. ALBERT’s SentencePiece tokenizer splits “likes” into “▁like” + “s”. The readout prints exactly the lists you would get from the tokenizer.

Two practical notes. First, the special tokens are added by the tokenizer, not by you; if you add them by hand as text, you may get them twice (session 9). Second, the 512 position rows are a hard limit: BERT cannot read token 513, so long documents must be cut or split into windows. [Certain]

BERT pretraining: two tasks at once

BERT is pretrained on two tasks at the same time, from unlabeled text.

Task 1: masked language modeling (MLM)

As in session 23: choose 15% of the positions; replace 80% of them with [MASK], 10% with a random token, and leave 10% unchanged; then predict the original token at each chosen position from the output vector at that position. The masking happens on WordPiece tokens, “with no special consideration given to partial word pieces”. [Certain] So in “play ##ing”, the model may have to guess only “##ing”, which is easy from the context; later models mask whole words or spans instead (see ALBERT below).

The procedure is easy to state and easy to get slightly wrong, so it helps to write down what happens to one position. Each non-special position is a target with probability \(p\); a target is shown to the model in one of three ways:

\[ \begin{aligned} \Pr[i \in \mathcal{M}] &= p = 0.15, \qquad \E\,|\mathcal{M}| = n\,p, \\ \Pr[\tilde{x}_i = \text{[MASK]}] &= 0.8\,p = 0.12, \qquad \Pr[\tilde{x}_i \text{ random}] = \Pr[\tilde{x}_i = x_i,\ i \in \mathcal{M}] = 0.1\,p = 0.015 \end{aligned} \tag{33.2} \]
Read the symbols
\(p\)
the mask rate: the chance that one position becomes a prediction target
\(\mathcal{M}\)
the set of target positions of this sequence; the loss is computed only there
\(n\)
the number of positions that may be chosen (special tokens never are)
\(\tilde{x}_i\)
the token the model actually sees at position \(i\); \(x_i\) is the original token

So only 12% of the input positions show [MASK], and 3% of the positions are targets that look like normal words. For a 512-token sequence, \(512 \times 0.15 = 76.8\): about 77 targets. Hugging Face’s DataCollatorForLanguageModeling flips an independent coin for every position, as in (33.2), so the number of targets varies from sequence to sequence; for a 12-word sentence there is a \(0.85^{12} \approx 14\%\) chance of no target at all. [Certain] The original BERT script instead picks exactly \(\operatorname{round}(0.15\,n)\) positions (at least 1, at most 20 for length 128). [Likely] Figure 2 runs the coin-flip version on the sentence pair from the BERT paper.

epoch 1
each card, top to bottom: position \(i\) · original token \(x_i\) · coin \(u_i\) (red tick = \(p\)) · model input \(\tilde{x}_i\) · label \(y_i\)
[MASK] (80% of targets)random token (10%)unchanged but predicted (10%)
Algorithm: 80/10/10 masking, one position at a time
    Figure 2. Masked language modeling data, built live in your browser for “[CLS] the man went to the store [SEP] he bought a gallon of milk [SEP]”. For every position the player draws a coin \(u_i\); if \(u_i \lt p\) the position becomes a target and a second number \(r\) decides between [MASK], a random word and no change, as in formula (33.2). The random numbers come from a seeded generator, so every epoch gives the same picture each time you load the page.
    How to read Figure 2

    Press Play or Step. The outlined card is the current position; the bar under its word fills to \(u_i\), and the red tick marks \(p\). A bar shorter than the tick means “chosen”. The live panel substitutes the numbers into the test. In epoch 1 at \(p = 0.15\), three positions are chosen: “man” (\(u = 0.072\), then \(r = 0.962 \ge 0.9\), so it stays “man” but is still a target), “gallon” (\(r = 0.813\): replaced by the random word “house”) and “of” (\(r = 0.399 \lt 0.8\): [MASK]). The model must predict all three, even “man”, which it can see. Press “New mask (next epoch)”: epoch 2 happens to choose no position at all, so this sequence adds nothing to the loss in that epoch. The last step also runs 10,000 fresh masks of the sentence and counts: about 15% of positions are chosen, and of those about 80/10/10 go to the three cases. Raise \(p\) to 0.5 to see many targets in one sentence.

    Task 2: next-sentence prediction (NSP)

    For each training example, BERT takes a span A from a document. For B, it takes the real next span 50% of the time (label IsNext) and a random span from another document 50% of the time (label NotNext). [Certain] The final vector of [CLS] goes through a small layer and a two-way softmax to predict the label. The idea: tasks such as question answering and natural language inference need relations between two texts, and MLM alone does not train that directly.

    The total loss is the sum of the two (the paper: “the sum of the mean masked LM likelihood and the mean next sentence prediction likelihood”):

    \[ \mathcal{L} = \underbrace{-\frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \log p_\theta(x_i \mid \tilde{x})}_{\mathcal{L}_{\text{MLM}}} \; \underbrace{- \log p_\theta(y \mid \tilde{x})}_{\mathcal{L}_{\text{NSP}}}, \qquad p_\theta(y \mid \tilde{x}) = \softmax\!\big(W_{\text{nsp}} \tanh(W_p \mathbf{h}_{\text{CLS}} + \mathbf{b}_p)\big)_y \tag{33.3} \]
    Read the symbols
    \(\mathcal{M}\)
    the set of chosen (masked) positions, about 15% of the tokens
    \(\tilde{x}\)
    the corrupted input (with [MASK] and random tokens), both sentences
    \(y\)
    the NSP label: IsNext or NotNext
    \(\mathbf{h}_{\text{CLS}}\)
    the last-layer output vector at position 0, the [CLS] position
    \(W_p, \mathbf{b}_p\)
    the pooler: an \(H \times H\) layer with tanh, trained by NSP. Hugging Face’s pooler_output is this vector
    \(W_{\text{nsp}}\)
    a \(2 \times H\) matrix: the NSP classifier

    The real recipe

    BERT was trained on BooksCorpus (800M words) and English Wikipedia (2,500M words), with batches of 256 sequences of up to 512 tokens, for 1,000,000 steps: about 40 epochs over the 3.3 billion words. [Certain] Optimizer: Adam with learning rate \(10^{-4}\), \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), weight decay 0.01, 10,000 warmup steps then linear decay; dropout 0.1; GELU activations. [Certain] Because attention cost grows with the square of the length, 90% of the steps used length 128 and only the last 10% used length 512 “to learn the positional embeddings”. [Certain] Each model trained for 4 days, BERT-base on 16 TPU chips and BERT-large on 64. [Certain]

    ModelLayers \(L\)Hidden \(H\)HeadsParameters
    BERT-base1276812110M
    BERT-large241,02416340M

    The NSP task turned out to be easy: the final model reaches 97–98% accuracy on it. [Certain] The paper also warns that the [CLS] vector “is not a meaningful sentence representation without fine-tuning, since it was trained with NSP”. [Certain] Remember that sentence; it comes back in the pooling section.

    RoBERTa: the same model, trained better

    RoBERTa (“a Robustly optimized BERT approach”, Liu et al., 2019) asked a simple question: is BERT’s design good, or was it under-trained? It kept the architecture of BERT-large (24 layers, \(H = 1024\), 16 heads, 355M parameters with its larger vocabulary) and changed only the training. [Certain] Each change was tested separately.

    Change 1: dynamic masking

    The original BERT code chose the masks once, when the data was prepared. To get some variety, the data was copied 10 times with 10 different masks. Over 40 epochs, each sequence was therefore seen with the same mask 4 times. [Certain] RoBERTa makes a new mask every time a sequence is fed to the model. This costs nothing, and it matters more when you train longer: with a static mask, the model would see the same puzzle again and again. Figure 3 lets you count.

    [MASK] (80%)random token (10%)unchanged but predicted (10%)
    Figure 3. One training sequence of 20 tokens, seen once per epoch. Each row is one epoch; each colored cell is a position the model must predict (15% of 20 = 3 positions per row). With a static mask the rows repeat; with dynamic masking every row is new. The masks come from a seeded random generator, so the picture is the same every time.
    How to read Figure 3

    With BERT’s 10 copies, at 40 epochs you see exactly 10 different patterns, each repeated 4 times, and some positions are never predicted at all. With one static copy, the model sees one pattern 40 times. With dynamic masking every row is a new pattern, and after 40 epochs 19 of the 20 positions have been a target. Move the slider down to 5 epochs: then the three modes are much closer. That is why RoBERTa found that dynamic masking “becomes crucial when pretraining for more steps or with larger datasets”.

    On BERT-base-sized runs, RoBERTa measured static and dynamic masking as close (SQuAD 2.0 F1 78.3 vs 78.7, MNLI 84.3 vs 84.0, SST-2 92.5 vs 92.9), “comparable or slightly better”, and kept dynamic masking because it is also simpler. [Certain] So dynamic masking is not a big win by itself; it is the change that makes longer training safe.

    Algorithm 1: dynamic 80/10/10 masking, run inside the data loader
    1. input: token IDs \(t_{1..n}\), a flag for special tokens, mask rate \(p = 0.15\), vocabulary size \(V\)
    2. for each position \(i\) that is not a special token: choose it with probability \(p\)
    3. for each chosen \(i\): set label \(y_i \leftarrow t_i\); draw \(r \sim U(0,1)\)
    4.   if \(r \lt 0.8\): \(t_i \leftarrow\) [MASK]; else if \(r \lt 0.9\): \(t_i \leftarrow\) a random token; else keep \(t_i\)
    5. for each position not chosen: \(y_i \leftarrow -100\) // ignored by the loss
    6. return corrupted IDs and labels // called again next epoch: a new mask (RoBERTa)

    Change 2: drop next-sentence prediction, pack full sentences

    The BERT paper had reported that removing NSP hurts (MNLI 84.4 → 83.9, QNLI 88.4 → 84.9, SQuAD F1 88.5 → 87.9 for BERT-base). [Certain] RoBERTa noticed that this test removed the NSP loss but probably kept the input format of two short segments. It compared four formats, all at BERT-base size, 1M steps, batch 256: [Certain]

    Input formatNSP lossSQuAD 2.0 (F1)MNLI-m (acc.)
    SEGMENT-PAIR: two segments, up to 512 tokens together (BERT)yes78.784.0
    SENTENCE-PAIR: two single natural sentencesyes76.282.9
    FULL-SENTENCES: fill 512 tokens with contiguous sentences, may cross documentsno79.184.7
    DOC-SENTENCES: the same, but never cross a document boundaryno79.784.7

    Two lessons. Short inputs hurt (the model cannot learn long-range relations). And without NSP, with long packed inputs, results are equal or better. RoBERTa used FULL-SENTENCES because DOC-SENTENCES gives variable batch sizes. [Certain] This is the same packing idea you saw in session 18. The third mode of Figure 5, in the ALBERT section below, packs a toy corpus this way next to the NSP and SOP recipes.

    Change 3: bigger batches

    Training for 1M steps with 256 sequences costs about the same as 125K steps with 2,048 sequences or 31K steps with 8,192 sequences: the number of sequences seen is the same.

    \[ B_1 S_1 \approx B_2 S_2: \qquad 256 \times 10^6 = 2048 \times 125{,}000 \approx 8192 \times 31{,}000 \tag{33.4} \]
    Read the symbols
    \(B\)
    batch size in sequences (each up to 512 tokens)
    \(S\)
    number of optimizer steps

    With the learning rate tuned for each size (\(10^{-4}\), \(7 \times 10^{-4}\), \(10^{-3}\)), the 2K batch gave the best MLM perplexity (3.68 vs 3.99 for 256) and the best MNLI accuracy (85.2 vs 84.7). [Certain] Large batches are also easier to spread over many GPUs. The final RoBERTa used 8K sequences per batch: \(8192 \times 512 \approx 4.2\) million tokens per step, 32 times BERT’s batch.

    Change 4: more data, longer training, byte-level BPE

    RoBERTa collected 160GB of text: BooksCorpus + Wikipedia (16GB, BERT’s data), CC-News (76GB), OpenWebText (38GB) and Stories (31GB). [Certain] It trained for up to 500K steps at 8K batch, which is about 16 times as many sequences as BERT. Final settings: peak learning rate \(4 \times 10^{-4}\) (large) or \(6 \times 10^{-4}\) (base), Adam \(\beta_2 = 0.98\) (for stability with large batches), \(\varepsilon = 10^{-6}\), linear decay. [Certain] It replaced BERT’s 30K WordPiece vocabulary with a 50K byte-level BPE (as in GPT-2), which never produces an unknown token and adds about 15M (base) and 20M (large) parameters. [Certain] Each step of the recipe helped RoBERTa-large on MNLI (dev): [Certain]

    RoBERTa-large setupDataSteps (batch 8K)MNLI-m
    BERT-large (for comparison; batch 256, 1M steps)13GB—86.6
    RoBERTa, Books + Wiki16GB100K89.0
    + more data160GB100K89.3
    + train longer160GB300K90.0
    + train even longer160GB500K90.2

    The big lesson for this whole track: before you change an architecture, make sure the baseline is trained well. The same lesson returns in session 35, where the amount of data per parameter turns out to matter a lot.

    ALBERT: fewer parameters

    ALBERT (“A Lite BERT”, Lan et al., 2019) attacks the parameter count. It makes two architecture changes and one change to the sentence task.

    Idea 1: factorized embedding

    In BERT the token table has size \(V \times H\): one \(H\)-dimensional row per vocabulary entry. If you want a wider model (bigger \(H\)), the table grows with it. But the table only needs to hold context-free word meaning, while the hidden states must hold everything the model knows about the token in context. ALBERT gives each token a small vector of size \(E\) and then projects it up to \(H\) with one shared matrix:

    \[ \underbrace{V \times H}_{\text{BERT}} \quad \longrightarrow \quad \underbrace{V \times E \;+\; E \times H}_{\text{ALBERT}}, \qquad E \ll H \tag{33.5} \]
    Read the symbols
    \(V\)
    vocabulary size (30,000 SentencePiece tokens in ALBERT)
    \(E\)
    the embedding size; ALBERT uses \(E = 128\) for every model size
    \(H\)
    the hidden size of the transformer layers (768 to 4,096)

    This is a low-rank factorization: the full \(V \times H\) table is replaced by the product of a \(V \times E\) and an \(E \times H\) matrix, so its rank is at most \(E\). It is the same trick as LoRA, used here for the embedding table (session 57).

    Worked example: the embedding table of a BERT-large-sized model

    BERT style, \(V = 30{,}000\), \(H = 1{,}024\): \(30{,}000 \times 1{,}024 = 30{,}720{,}000\) parameters.

    ALBERT style, \(E = 128\): \(30{,}000 \times 128 + 128 \times 1{,}024 = 3{,}840{,}000 + 131{,}072 = 3{,}971{,}072\). That is 7.7 times smaller. At \(H = 4{,}096\) (ALBERT-xxlarge) the saving is bigger: \(122{,}880{,}000\) against \(4{,}364{,}288\).

    At \(H = 768\) the saving is \(23{,}040{,}000\) against \(3{,}938{,}304\), 5.9 times. The larger the model, the more the factorization saves.

    Idea 2: cross-layer parameter sharing

    BERT-base has 12 layers, each with its own weights. ALBERT uses one set of layer weights and applies it 12 times (or 24 times). The input goes through the same layer again and again, like a loop. One layer of width \(H\) with FFN width \(F\) has

    \[ P_{\text{layer}} = \underbrace{4(H^2 + H)}_{Q,K,V,O} + \underbrace{2HF + F + H}_{\text{FFN}} + \underbrace{4H}_{\text{2 LayerNorms}}, \qquad P_{\text{total}} = P_{\text{emb}} + \begin{cases} L \cdot P_{\text{layer}} & \text{BERT} \\ P_{\text{layer}} & \text{ALBERT (all shared)} \end{cases} \tag{33.6} \]

    For \(H = 768\), \(F = 3{,}072\): \(P_{\text{layer}} = 7{,}087{,}872\). Sharing it across 12 layers removes \(11 \times 7{,}087{,}872 \approx 78\)M parameters. ALBERT also tested sharing only the attention or only the FFN, on an ALBERT-base model: [Certain]

    ALBERT-base variantParams, \(E = 768\)Avg. scoreParams, \(E = 128\)Avg. score
    not shared (BERT style)108M82.389M81.6
    shared attention only83M81.664M81.7
    shared FFN only57M79.538M80.2
    all shared (ALBERT)31M79.812M80.1

    “Avg.” is the paper’s average over SQuAD 1.1, SQuAD 2.0, MNLI, SST-2 and RACE. Our formula (33.6) reproduces every parameter count in this table to within 1M (Figure 4). Sharing everything costs 1.5 to 2.5 points at base size; sharing the FFN hurts more than sharing attention. The paper chose all-shared because it allows much wider models for the same parameter budget.

    Figure 4. A parameter calculator for BERT-like encoders, using formulas (33.5) and (33.6) plus the pooler (\(H^2 + H\)), 512 positions and 2 segments. The top bar is what you store; the bottom bar is how many weights each token is multiplied with in one forward pass, which sets the compute.
    How to read Figure 4

    Click BERT-base: 109,482,240 stored parameters, the “110M” of the paper. Click ALBERT-base: about 11.7M stored, yet the weights used per token stay at about 85M, the same as BERT-base, because the shared layer is still applied 12 times. Click ALBERT-xxlarge (\(H = 4096\), 12 layers): about 223M stored, fewer than BERT-large’s 335M, but about 2.4 billion weights used per token, 8 times BERT-large’s 302M. That is why ALBERT-xxlarge is about 3 times slower to train than BERT-large. Our count for xxlarge is a bit below the paper’s 235M; the paper does not list exactly what it counts. [Likely]

    Fewer parameters is not less compute

    This is the most important lesson from ALBERT, and a classic interview trap. Sharing reduces memory for weights (and the size of the file you download), and it reduces communication between GPUs during training. It does not reduce the FLOPs: a 12-layer ALBERT does 12 layers of work per token. The paper measured training throughput relative to BERT-large: ALBERT-large is about 1.7 times faster, but ALBERT-xxlarge is about 3 times slower. [Certain] ALBERT-xxlarge (235M parameters, about 70% of BERT-large) was clearly better on the benchmarks (for example MNLI 88.0 vs 86.6 and RACE 82.3 vs 73.9 on dev), at a much higher compute cost. [Certain]

    Idea 3: sentence-order prediction (SOP) instead of NSP

    Why did NSP not help? ALBERT’s answer: it is too easy. A NotNext pair has B taken from a different document, so B is usually about a different topic. The model can solve NSP by checking “same topic or not”, which MLM already teaches, and never needs to learn how sentences connect. ALBERT’s sentence-order prediction keeps both segments from the same place in the same document and only swaps their order for negatives. Now the topic is the same in both cases, and only the logic of the order tells them apart.

    cookingfootballweather
      Figure 5. Three ways to build pretraining inputs from the same toy corpus of 6 documents (4 sentences each, 3 topics), computed live. NSP takes the negative B from another document; SOP swaps two consecutive sentences; RoBERTa drops the sentence task and packs whole sentences into long sequences. After each NSP or SOP example, a “topic shortcut” that never reads the order guesses the label: it predicts a negative exactly when A and B have different topics.
      How to read Figure 5

      Each NSP or SOP example takes three steps: pick sentence A (blue outline), flip a coin to build B (orange outline) and the label, then let the topic shortcut guess. On NSP the shortcut is right on 7 of the first 8 examples; its one miss is a NotNext pair whose random B (“Add the pasta and some salt.”) happens to come from the other cooking document. With 3 topics, a random other document has a different topic 4 times in 5, so the expected accuracy is \(0.5 \cdot 1 + 0.5 \cdot \tfrac{4}{5} = 0.9\); “Simulate 1,000 examples” gives 906 of 1,000. Switch to SOP: both sentences always come from the same document, so the shortcut always says “in order” and gets 4 of 8, and 510 of 1,000 in the simulation: chance. To solve SOP, a model must learn how sentences connect (“Then …” cannot come first). The third mode is RoBERTa’s FULL-SENTENCES: sentences are packed in order into sequences of at most 20 words (a toy stand-in for 512 tokens), with a separator where a document ends. No label and no [CLS] loss; the 24 sentences fill 8 sequences, and the model sees long, natural context for MLM.

      The experiment is clean: [Certain]

      Trained withAccuracy on NSP taskAccuracy on SOP taskAvg. downstream
      MLM only52.453.379.0
      MLM + NSP90.552.079.2
      MLM + SOP78.986.580.1

      A model trained on NSP is at chance (52.0%) on SOP: it learned topic, not order. A model trained on SOP solves NSP reasonably well (78.9%), and its downstream average is the best. [Certain]

      Other ALBERT choices

      ALBERT masks n-grams of up to 3 whole words (as in SpanBERT), with shorter spans more likely:

      \[ p(n) = \frac{1/n}{\sum_{k=1}^{N} 1/k}, \qquad N = 3: \quad p(1) = \tfrac{6}{11} \approx 0.545,\; p(2) = \tfrac{3}{11} \approx 0.273,\; p(3) = \tfrac{2}{11} \approx 0.182 \tag{33.7} \]

      It trains with the LAMB optimizer (a variant of Adam made for very large batches) at batch 4,096, and its final models remove dropout, which improved results. [Certain]

      From token vectors to one vector: [CLS] or mean pooling

      An encoder returns one vector per token: a matrix \(\mathbf{h}_0, \dots, \mathbf{h}_{n-1}\), each of size \(H\). Token tasks (named entities, extractive question answering) put a small layer on each \(\mathbf{h}_i\). Whole-text tasks (sentiment, topic, spam) need one vector \(\mathbf{z}\) for the whole input. This step is called pooling. There are two common ways.

      [CLS] pooling takes the output at position 0. Because every layer is bidirectional, position 0 has attended to every token, so it can hold a summary, and fine-tuning teaches it to. “Bidirectional” is a statement about the attention mask. In every layer, the output at position \(i\) is a weighted average of value vectors, and the mask decides which positions may get a weight:

      \[ \begin{aligned} s_{ij} &= \frac{\mathbf{q}_i^\top \mathbf{k}_j}{\sqrt{d}}, \qquad a_{ij} = \frac{\exp(s_{ij} + M_{ij})}{\sum_{k} \exp(s_{ik} + M_{ik})}, \qquad \mathbf{o}_i = \sum_j a_{ij} \mathbf{v}_j, \\ M_{ij} &= \begin{cases} 0 & \text{if } m_j = 1 \text{ (encoder)}, \text{ or } m_j = 1 \text{ and } j \le i \text{ (decoder)} \\ -\infty & \text{otherwise} \end{cases} \end{aligned} \tag{33.8} \]
      Read the symbols
      \(\mathbf{q}_i, \mathbf{k}_j, \mathbf{v}_j\)
      the query of position \(i\) and the key and value of position \(j\), each of size \(d\)
      \(M_{ij}\)
      the attention mask: 0 keeps the pair, \(-\infty\) makes \(\exp(\cdot) = 0\), so the weight is exactly 0
      \(m_j\)
      the padding mask from the tokenizer: 1 for a real token, 0 for [PAD]
      \(a_{ij}\)
      how much position \(i\) reads from position \(j\); each row sums to 1

      Row 0 of this matrix is the [CLS] row. In an encoder it may read every real token. With a causal (decoder) mask, row 0 may read only itself, so the vector at position 0 is the same whatever the sentence says, and a classifier on it can only guess. This is why decoder models are pooled at the last token instead. Figure 6 computes (33.8) row by row in one toy attention layer and then puts the classification head (33.11) on \(\mathbf{o}_0\).

      Algorithm: masked attention, then [CLS] pooling
        Figure 6. One attention layer, formula (33.8), on “[CLS] the film was great [SEP]” plus padding, computed live with 4-number inputs \(\mathbf{x}_j\), \(W_K = W_V = I\) and a fixed \(W_Q\) that lets the [CLS] query look for opinion words. Left: the attention matrix; row \(i\) is the query, column \(j\) the key; grey cells are masked (\(-\infty\)); a filled cell shows the weight \(a_{ij}\). Right: the current row’s scores and weights, then the output \(\mathbf{o}_i\). The last step applies the head \(W = \begin{pmatrix} 2 & 0 & 0 & 0 \\ -2 & 0 & 0 & 0 \end{pmatrix}\), \(\mathbf{b} = \mathbf{0}\), to \(\mathbf{z} = \mathbf{o}_0\).
        How to read Figure 6

        Press Play. Row 0 ([CLS]) is computed in three steps: scores, mask, softmax and output. Its query is \(\mathbf{q}_0 = (0, 0, 0, 4)\), so it scores each key by its 4th number; “great” gets \(s_{0,4} = 4 \times 0.9 / 2 = 1.8\) and, after softmax, weight 0.500. The other rows follow, one per step. In the last step the head reads \(\mathbf{z} = \mathbf{o}_0 = (0.779, 0.040, 0.174, 0.506)\) and gives p(positive) = 0.958. Click “awful”: p(positive) = 0.053. Now choose “Decoder: causal”: row 0 may only read itself, \(\mathbf{o}_0 = \mathbf{x}_0 = (0, 1, 0, 0)\), and p(positive) = 0.500 for both words. The [CLS] vector no longer knows the sentence. “Bug: no padding mask” lets the queries read the [PAD] keys too: with 3 pads, the weight on “great” falls to 0.299 and p(positive) drops from 0.958 to 0.710, although the text did not change. With the correct mask (first mode), the slider changes nothing in row 0. The pad rows themselves still get outputs; they are simply never read.

        BERT puts its pooler (tanh layer) on top of \(\mathbf{h}_0\):

        \[ \mathbf{z}_{\text{CLS}} = \mathbf{h}_0 \qquad \text{(BERT’s pooler: } \mathbf{z} = \tanh(W_p \mathbf{h}_0 + \mathbf{b}_p)\text{)} \tag{33.9} \]

        Mean pooling averages the outputs of all real tokens. The word “real” is important: padding positions must be left out, using the attention mask \(m_i\) (1 for a real token, 0 for padding):

        \[ \mathbf{z}_{\text{mean}} = \frac{\sum_{i=0}^{n-1} m_i \, \mathbf{h}_i}{\sum_{i=0}^{n-1} m_i} \tag{33.10} \]
        Read the symbols
        \(\mathbf{h}_i\)
        the last-layer output vector at position \(i\)
        \(m_i\)
        the attention mask: 1 for a real token (including [CLS] and [SEP] in the usual implementation), 0 for padding
        \(n\)
        the padded length of the batch

        The classification head

        For \(K\) classes, the head is one linear layer followed by softmax, trained with cross-entropy. The BERT paper says: “The only new parameters introduced during fine-tuning are classification layer weights \(W \in \R^{K \times H}\).” [Certain]

        \[ \mathbf{p} = \softmax(W \mathbf{z} + \mathbf{b}), \qquad \mathcal{L} = -\log p_y, \qquad W \in \R^{K \times H},\; \mathbf{b} \in \R^{K} \tag{33.11} \]

        With \(H = 768\) and \(K = 2\), the head has \(2 \times 768 + 2 = 1{,}538\) parameters, against 110M in the encoder. In fine-tuning all 110M are updated together with the head, for a few epochs at a small learning rate; the paper suggests batch 16 or 32, learning rate \(5\times10^{-5}\), \(3\times10^{-5}\) or \(2\times10^{-5}\), and 2 to 4 epochs. [Certain] In a linear probe the encoder is frozen and only the head is trained; this is cheaper but usually weaker. [Likely] (Hugging Face’s BertForSequenceClassification uses the pooler output; RobertaForSequenceClassification applies its own dense + tanh layer to the <s> vector, then the output layer. [Certain])

        Figure 7. Three pooling methods on the same toy outputs (\(H = 4\)) for “[CLS] the film was great [SEP]”, followed by the same 2-class head \(W\). Each column of colored cells is one token’s output vector (darker = larger). Padding outputs are not zero: the model computes some vector at every position.
        How to read Figure 7

        At 0 padding all methods behave well. Add padding: [CLS] pooling and masked mean pooling (formula 33.10) do not change, because the mask removes the padded positions. The naive mean (h.mean(dim=1)) averages the padding vectors in, and the predicted class flips from positive to negative at 4 padding tokens. In a real batch, the amount of padding depends on the other texts in the batch, so a naive mean gives a different answer for the same review depending on its batch neighbours.

        Worked example: the numbers in Figure 7

        Head: \(W = \begin{pmatrix} 1.2 & -0.8 & 0.6 & 0 \\ -1.2 & 0.8 & -0.6 & 0 \end{pmatrix}\), \(\mathbf{b} = (0.1, -0.1)\). Row 1 is “positive”.

        [CLS] pooling: \(\mathbf{z} = \mathbf{h}_0 = (0.9, -0.2, 0.4, 0.1)\). Logit 1 \(= 1.08 + 0.16 + 0.24 + 0.1 = 1.58\), logit 2 \(= -1.58\). \(p_{\text{pos}} = e^{1.58} / (e^{1.58} + e^{-1.58}) = 0.959\).

        Mean pooling over the 6 real tokens: \(\mathbf{z} = (0.533, 0.033, 0.233, 0.150)\), logits \(\pm 0.853\), \(p_{\text{pos}} = 0.846\).

        Naive mean with 4 padding vectors \((-0.8, 0.9, -0.5, 0.6)\) included: \(\mathbf{z} = (0, 0.38, -0.06, 0.33)\), logits \(\mp 0.24\), \(p_{\text{pos}} = 0.382\). The review is now “negative”, only because of padding.

        Which one is better?

        For fine-tuned classification, both work; [CLS] is the BERT convention, and the difference is usually small. [Likely] For sentence embeddings (vectors you compare with cosine similarity, with no task head), the answer is different, and the next section shows it with numbers. ModernBERT’s config sets mean pooling as its default for classification (classifier_pooling: "mean"). [Certain]

        Sentence embeddings: why raw BERT is not enough, and Sentence-BERT

        Suppose you want semantic search: find the sentences most similar to a query. BERT can compare two sentences very well if you feed them together, as [CLS] A [SEP] B [SEP], and put a head on [CLS]. This is called a cross-encoder. But it needs one forward pass per pair. For \(n\) sentences, finding the most similar pair needs

        \[ \binom{n}{2} = \frac{n(n-1)}{2}, \qquad n = 10{,}000: \quad 49{,}995{,}000 \text{ forward passes} \tag{33.12} \]

        Sentence-BERT (Reimers and Gurevych, 2019) measured this at about 65 hours with BERT. [Certain] The alternative is a bi-encoder: encode each sentence alone into one vector (10,000 passes), then compare vectors with cosine similarity, which is a cheap dot product. SBERT reports about 5 seconds for the 10,000 embeddings and about 0.01 seconds for the cosine similarities. [Certain]

        The problem: raw BERT vectors are poor at this. On seven semantic similarity benchmarks (Spearman correlation × 100, higher is better), SBERT found: [Certain]

        Sentence vectorAverage over 7 STS tasks
        BERT [CLS] vector, no fine-tuning29.19
        Mean of BERT output vectors, no fine-tuning54.81
        Mean of GloVe word vectors (2014, no context)61.32
        SBERT-NLI-base (mean pooling, fine-tuned)74.89

        Raw [CLS] is far worse than averaging old static word vectors. This matches the BERT paper’s warning: [CLS] was trained for NSP, not to be a similarity space. Raw BERT vectors also crowd into a narrow cone of the space (all cosines high), which makes cosine similarity less informative (session 11). [Likely]

        The SBERT fix: fine-tune in a siamese setup

        SBERT runs the same BERT on sentence A and on sentence B separately (a “siamese” network: two copies with shared weights), mean-pools each to \(\mathbf{u}\) and \(\mathbf{v}\), and trains on natural-language-inference pairs (SNLI 570K + MultiNLI 430K pairs, labels entailment / contradiction / neutral) with this head: [Certain]

        \[ \mathbf{o} = \softmax\!\big( W_t \, [\,\mathbf{u};\, \mathbf{v};\, |\mathbf{u} - \mathbf{v}|\,] \big), \qquad W_t \in \R^{3 \times 3H} \tag{33.13} \]
        Read the symbols
        \(\mathbf{u}, \mathbf{v}\)
        the pooled vectors of the two sentences, each of size \(H\)
        \([\,\cdot\,;\,\cdot\,;\,\cdot\,]\)
        concatenation into one vector of size \(3H\)
        \(|\mathbf{u} - \mathbf{v}|\)
        the element-wise absolute difference
        \(W_t\)
        the training head; it is thrown away after training

        After training, only the encoder is kept, and similarity is the cosine:

        \[ \operatorname{sim}(a, b) = \cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u}^\top \mathbf{v}}{\lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert} \tag{33.14} \]

        SBERT trained 1 epoch with batch 16 and learning rate \(2\times10^{-5}\). [Certain] Its ablation compared poolings (on the STS benchmark dev set, Spearman × 100): when trained on NLI, MEAN 80.78, CLS 79.80, MAX 79.07; when trained directly on STS data, MEAN 87.44, CLS 86.62, MAX 69.92. [Certain] So after fine-tuning, CLS and mean are close; mean is a safe default. The paper calls \(|\mathbf{u} - \mathbf{v}|\) the most important part of the head input. [Certain] Modern embedding models (E5, BGE and others) replace the NLI classifier with a contrastive loss over many negatives, as in session 60 and SimCSE; the bi-encoder plan is the same. [Likely]

        Algorithm 2: fine-tune an encoder for classification, then serve it
        1. input: pretrained encoder (BERT/RoBERTa/ALBERT), labeled pairs \((\text{text}, y)\), \(K\) classes
        2. add head \(W \in \R^{K \times H}\), \(\mathbf{b} \in \R^K\), randomly initialized
        3. for 2 to 4 epochs, batch 16–32, AdamW with lr \(2\text{–}5 \times 10^{-5}\), warmup then linear decay:
        4.   tokenize with the model’s own tokenizer: special tokens, segments, padding, attention mask
        5.   \(H \leftarrow \operatorname{Encoder}(\text{ids}, \text{segments}, \text{mask})\) // one pass, no causal mask
        6.   \(\mathbf{z} \leftarrow\) [CLS] pooling (33.9) or masked mean (33.10)
        7.   \(\mathcal{L} \leftarrow -\log \softmax(W\mathbf{z} + \mathbf{b})_y\) (33.11); backward through head and encoder; step
        8. serve: one forward pass per text, take \(\arg\max\) of \(\mathbf{p}\); no generation loop

        The code: a tiny BERT/ALBERT, pooling, a head, and dynamic masking

        This script builds a small BERT-like encoder from PyTorch parts, with a switch for ALBERT’s factorized embedding and shared layer. It checks that [CLS] and masked mean pooling do not change when padding is added, that the naive mean does, and it runs one MLM step with dynamic masking. It needs only torch and was run with PyTorch 2 on CPU; the printed values are shown in the comments.

        import torch, torch.nn as nn, torch.nn.functional as F
        torch.manual_seed(0)
        CLS, SEP, MASK, PAD = 101, 102, 103, 0          # bert-base-uncased special ids
        
        class TinyEncoder(nn.Module):
            """BERT-like encoder. E < H gives ALBERT's factorized embedding; share=True gives ALBERT's shared layers."""
            def __init__(self, V=30522, H=64, E=64, L=4, heads=4, F_=256, P=512, share=False, n_cls=2):
                super().__init__()
                self.tok = nn.Embedding(V, E, padding_idx=PAD)   # token table
                self.seg = nn.Embedding(2, E)                    # sentence A / sentence B
                self.pos = nn.Embedding(P, E)                    # learned absolute positions
                self.ln  = nn.LayerNorm(E)
                self.proj = nn.Linear(E, H) if E != H else nn.Identity()   # ALBERT: E -> H, formula (33.5)
                mk = lambda: nn.TransformerEncoderLayer(H, heads, F_, dropout=0.0, activation="gelu",
                                                        batch_first=True, norm_first=False)  # Post-LN like BERT
                self.layers = nn.ModuleList([mk()] if share else [mk() for _ in range(L)])
                self.L, self.share = L, share
                self.mlm = nn.Linear(H, V)                       # MLM head (BERT also ties it to self.tok)
                self.cls_head = nn.Linear(H, n_cls)              # classification head: one linear layer
        
            def forward(self, ids, seg, attn):                   # attn: 1 = real token, 0 = padding
                pos = torch.arange(ids.shape[1]).unsqueeze(0)
                x = self.proj(self.ln(self.tok(ids) + self.seg(seg) + self.pos(pos)))   # formula (33.1)
                for i in range(self.L):
                    layer = self.layers[0] if self.share else self.layers[i]          # same weights L times
                    x = layer(x, src_key_padding_mask=(attn == 0))
                return x                                         # (B, T, H): one vector per token
        
        def pool(h, attn, how):
            if how == "cls":  return h[:, 0]                     # formula (33.9)
            m = attn.unsqueeze(-1).float()                       # formula (33.10): masked mean
            return (h * m).sum(1) / m.sum(1).clamp(min=1)
        
        def dynamic_mask(ids, special, V, p=0.15):
            """Fresh 80/10/10 mask every call (RoBERTa), Algorithm 1. Labels -100 = no loss."""
            ids = ids.clone(); labels = ids.clone()
            chosen = (torch.rand(ids.shape) < p) & ~special
            labels[~chosen] = -100
            r = torch.rand(ids.shape)
            ids[chosen & (r < 0.8)] = MASK                                          # 80%: [MASK]
            rnd = chosen & (r >= 0.8) & (r < 0.9)                                   # 10%: random token
            ids[rnd] = torch.randint(1000, V, (int(rnd.sum()),))
            return ids, labels                                                      # other 10%: unchanged
        
        # [CLS] my dog is cute [SEP] he likes playing [SEP]  (real bert-base-uncased ids), then 3 pads
        ids  = torch.tensor([[CLS, 2026, 3899, 2003, 10140, SEP, 2002, 7777, 2652, SEP, PAD, PAD, PAD]])
        seg  = torch.tensor([[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0]])
        attn = (ids != PAD).long()
        
        bert   = TinyEncoder().eval()
        albert = TinyEncoder(E=16, share=True).eval()
        count = lambda m: sum(p.numel() for p in m.parameters())
        print("params  bert-like:", count(bert), "  albert-like:", count(albert))   # 4170428 vs 2531740
        
        with torch.no_grad():
            h = bert(ids, seg, attn)
            h_short = bert(ids[:, :10], seg[:, :10], attn[:, :10])                 # same text, no padding
            for how in ["cls", "mean"]:
                a, b = pool(h, attn, how), pool(h_short, attn[:, :10], how)
                print(how, tuple(a.shape), "padding changes it by", f"{(a - b).abs().max().item():.1e}")  # ~1e-7
            naive = h.mean(1)                                                       # WRONG: averages the pads too
            print("naive mean differs by", round((naive - pool(h_short, attn[:, :10], "mean")).abs().max().item(), 3))  # 0.304
            probs = F.softmax(bert.cls_head(pool(h, attn, "cls")), -1)              # formula (33.11)
            print("class probabilities", [round(v, 3) for v in probs[0].tolist()])   # untrained head: [0.867, 0.133]
        
        special = (ids == CLS) | (ids == SEP) | (ids == PAD)
        torch.manual_seed(1); x1, y1 = dynamic_mask(ids, special, 30522, p=0.3)
        torch.manual_seed(2); x2, y2 = dynamic_mask(ids, special, 30522, p=0.3)
        print("epoch 1 targets at", (y1[0] != -100).nonzero().flatten().tolist())   # [1, 4]
        print("epoch 2 targets at", (y2[0] != -100).nonzero().flatten().tolist())   # [7]: a new mask
        
        # one MLM training step (positions with label -100 are ignored by cross_entropy)
        bert.train(); opt = torch.optim.AdamW(bert.parameters(), lr=1e-4, weight_decay=0.01)
        logits = bert.mlm(bert(x1, seg, attn))
        loss = F.cross_entropy(logits.view(-1, logits.shape[-1]), y1.view(-1), ignore_index=-100)
        loss.backward(); opt.step()
        print("MLM loss", round(loss.item(), 2), " ln(V) =", round(torch.log(torch.tensor(30522.)).item(), 2))  # 9.85 vs 10.33

        Notes. The mask rate is 0.3 in the demo only so that a 13-token example has some targets. The first MLM loss (9.85) is close to \(\ln V = 10.33\), the loss of a uniform guess, as expected for random weights (session 3). The ALBERT-like model has fewer parameters mostly because its token table is \(30{,}522 \times 16\) instead of \(\times 64\). With the Hugging Face library you would load AutoModel.from_pretrained("bert-base-uncased") and get the same three pieces: last_hidden_state (all \(\mathbf{h}_i\)), pooler_output (the tanh pooler), and the attention mask from the tokenizer.

        Where this is used

        ApplicationWhat the encoder doesPooling / head
        Sentiment, intent, spam, toxicity, topic classificationFine-tuned BERT/RoBERTa reads each text once[CLS] or mean, linear head (33.11)
        Named-entity recognition, taggingOne label per tokenNo pooling; a linear head on every \(\mathbf{h}_i\)
        Extractive question answering (SQuAD)Question as A, passage as B; predict start and end positionsTwo vectors scored against every passage token
        Semantic search, RAG retrievalBi-encoder (SBERT-style) embeds queries and documents separately (session 152)Mean pooling + cosine (33.14)
        Reranking the top results of a searchCross-encoder reads query and document together[CLS] + 1-output head (relevance score)
        Text-to-video retrievalThe text side of a dual encoder maps the query into the same space as video clip vectors (session 208)Pooled text vector, cosine to video vectors
        Video key-moment and chapter detection from transcriptsClassify each transcript window (“highlight / not”) or score it against a queryPer-window [CLS] or mean, linear head
        Video-RAG over speech transcripts and captionsEmbed time-stamped transcript chunks; retrieve the chunks for a question (session 210)Mean pooling + cosine
        Near-duplicate detection of video titles and descriptionsEmbed metadata text; very similar vectors suggest re-uploads, used next to visual fingerprints (session 209)Mean pooling + cosine threshold
        Vision encoders (ViT, VideoMAE)The same encoder idea for patches: a [CLS] token or mean pooling, and masked prediction for pretraining (session 164)[CLS] (ViT) or mean of patch tokens

        Later encoders kept the plan and improved the parts: ELECTRA trains the encoder to detect replaced tokens at every position instead of predicting 15% of them; DeBERTa separates content and position in attention; ModernBERT brings RoPE, longer context (8,192) and faster attention. [Certain]

        Common mistakes

        • Mean pooling without the attention mask. h.mean(dim=1) includes padding vectors, so the result depends on the batch (Figure 7). Always use formula (33.10).
        • Using the raw [CLS] vector of a pretrained (not fine-tuned) BERT as a sentence embedding. It scores 29.19 on STS, worse than averaged GloVe. Use a model trained for embeddings (SBERT, E5 and similar).
        • Forgetting token_type_ids for sentence pairs in BERT, or passing them to RoBERTa, whose segment table has only one row. Let the tokenizer build the pair: tok(a, b).
        • Using the wrong special tokens. RoBERTa expects <s>, </s></s>, </s>; BERT expects [CLS], [SEP]. Mixing tokenizers and models gives wrong IDs with no error message.
        • Feeding more than 512 tokens. BERT and RoBERTa have only 512 position rows. Truncate, or split long texts into overlapping windows and pool the window results.
        • Reading ALBERT’s small parameter count as “fast”. ALBERT-xxlarge has fewer parameters than BERT-large and is about 3 times slower to train.
        • Fine-tuning with a pretraining-sized learning rate. \(10^{-4}\) or more for full fine-tuning of BERT often destroys the pretrained features; the paper’s range is \(2\)–\(5 \times 10^{-5}\). [Likely]
        • Cross-encoder for first-stage search. It needs one pass per (query, document) pair. Use a bi-encoder to retrieve, a cross-encoder only to rerank the top few.

        Interview questions

        What is the [CLS] token, and why can it summarize the whole input?basic
        [CLS] is a special token placed at position 0 of every BERT input. It carries no word meaning. Because BERT’s attention is bidirectional, the vector at position 0 attends to every token in every layer, so after 12 layers it can collect information from the whole input. Pretraining (NSP) and fine-tuning (a classification head on it) teach it to hold a summary that is useful for the task. Without fine-tuning it is not a good general sentence vector.
        Describe the BERT input format and the three embeddings.basic
        [CLS] A [SEP] B [SEP] (or [CLS] A [SEP] for one text). Each position’s input is the sum of a token embedding (30,522 × 768 in BERT-base), a segment embedding (A or B, 2 × 768) and a learned position embedding (512 × 768), followed by LayerNorm. The segment embedding lets one model handle single-text and pair tasks such as NLI and question answering.
        What is next-sentence prediction, and why did RoBERTa remove it?core
        NSP is a binary task on [CLS]: is B the real next segment after A (50%) or a random segment from another document (50%)? RoBERTa compared input formats and found that training on long packed inputs (FULL-SENTENCES or DOC-SENTENCES) without NSP matched or beat the BERT format with NSP (for example MNLI 84.7 vs 84.0). The earlier BERT ablation that showed NSP helping probably kept the short two-segment inputs. ALBERT later explained why NSP is weak: random negatives differ in topic, so the task is mostly topic detection, which MLM already covers.
        What is the difference between static and dynamic masking, and when does it matter?core
        Static masking picks the masked positions once in preprocessing; BERT made 10 copies, so over 40 epochs each sequence was seen 4 times with each of 10 masks. Dynamic masking draws a new mask each time a sequence is loaded. At BERT’s training length the difference was small (SQuAD 2.0 78.3 vs 78.7); it matters more with longer training or more epochs, where a static mask would make the model see the same puzzles many times and leave some positions never predicted.
        List the RoBERTa changes. Which ones matter most?core
        Dynamic masking; no NSP with full-sentence packing; larger batches (8K sequences) with a higher learning rate and \(\beta_2 = 0.98\); 10 times more data (160GB vs 16GB); longer training (500K steps at 8K batch, about 16 times more sequences than BERT); byte-level BPE with 50K tokens. The architecture is unchanged. Data and training length gave the largest gains in its table (MNLI 89.0 → 90.2 for large as data and steps grew), and the overall gain over BERT-large was large (86.6 → 90.2). The lesson: BERT was under-trained.
        How does ALBERT reduce parameters? Give the numbers for the embedding.core
        Two ways. (1) Factorized embedding: replace the \(V \times H\) table with \(V \times E + E \times H\), \(E = 128\). For \(V = 30{,}000\), \(H = 1{,}024\): 30.7M becomes 3.97M. (2) Cross-layer sharing: one set of layer weights applied \(L\) times, so the layers cost \(P_{\text{layer}}\) instead of \(L \cdot P_{\text{layer}}\). ALBERT-base has 12M parameters against BERT-base’s 108M (in the ALBERT paper’s count).
        ALBERT-xxlarge has fewer parameters than BERT-large. Is it faster?advanced
        No. Compute per token depends on the weights each token is multiplied with, and a shared layer applied 12 times still costs 12 layers. ALBERT-xxlarge has \(H = 4096\), so each layer is about 16 times larger than a BERT-large layer; with 12 layers it uses about 2.4B weights per token versus about 0.3B for BERT-large. The paper measured it as about 3 times slower to train than BERT-large (ALBERT-large is about 1.7 times faster). Sharing saves memory and communication, not FLOPs.
        [CLS] pooling or mean pooling? What can go wrong with mean pooling?core
        For fine-tuned classification both work and differ little. For sentence embeddings, mean pooling is the safer default: SBERT found MEAN slightly above CLS (80.78 vs 79.80 after NLI training) and raw CLS without fine-tuning is very poor. The classic bug is averaging over padding: the mean must use the attention mask, \(\sum m_i \mathbf{h}_i / \sum m_i\); otherwise the embedding of a text changes with the other texts in its batch.
        Why are raw BERT embeddings bad for semantic similarity, and how does Sentence-BERT fix it?advanced
        BERT was never trained so that cosine similarity between pooled vectors means “similar meaning”: MLM trains token predictions and NSP trains [CLS] for a binary task. Raw vectors also crowd into a narrow region where all cosines are high. SBERT fine-tunes BERT as a siamese bi-encoder on NLI pairs with a softmax over \([\mathbf{u}; \mathbf{v}; |\mathbf{u}-\mathbf{v}|]\), which shapes the space so that cosine works (STS average 74.89 vs 54.81 for raw mean and 29.19 for raw CLS). It also makes search cheap: \(n\) passes plus dot products instead of \(n(n-1)/2\) cross-encoder passes.
        What does a classification head on BERT look like, and how many parameters does it add?basic
        One linear layer from the pooled vector to \(K\) logits, then softmax and cross-entropy: \(\mathbf{p} = \softmax(W\mathbf{z} + \mathbf{b})\), \(W \in \R^{K \times H}\). For \(H = 768\), \(K = 2\): 1,538 parameters. (BERT’s pretrained tanh pooler adds \(768^2 + 768 = 590{,}592\) more, but it already exists.) During fine-tuning all encoder weights are usually updated too, for 2 to 4 epochs at a learning rate around \(2\text{–}5\times10^{-5}\).
        Why does BERT mask with 80/10/10 rather than always using [MASK]?core
        [MASK] never appears at fine-tuning time. If the model only had to predict at [MASK] positions, it could treat all normal tokens as “not my problem” and not build good vectors for them. Replacing 10% with random tokens and leaving 10% unchanged means any token might be a target, so the model must keep a good representation of every position. The cost is a small amount of noise in the inputs.
        Would you use BERT or a decoder LLM for a 50-class ticket classifier with 1 million requests per day?advanced
        A fine-tuned encoder is usually the better choice when you have labeled data: one forward pass of about 110–350M parameters per ticket, no generation, a fixed label set, and easy calibration of the softmax scores. A decoder LLM can work zero-shot or with few examples and handles changing labels, but costs much more per request and its outputs must be parsed. A common pattern is to use an LLM to label data, then fine-tune an encoder for serving. [Likely]

        Exercises

        1. Write the input IDs and segment IDs that bert-base-uncased produces for the single sentence “My dog is cute”. Use Figure 1 to check.
        2. BERT made 10 static copies and trained 40 epochs. If you train 100 epochs with the same 10 copies, how many times does the model see each mask pattern?
        3. Compute the embedding parameters for \(V = 50{,}265\) and \(H = 1{,}024\) (RoBERTa-large), and with ALBERT’s factorization at \(E = 128\).
        4. Using formula (33.6), how many layer parameters does a 24-layer encoder with \(H = 1024\), \(F = 4096\) have with no sharing, and with all layers shared?
        5. A batch has two texts: one with 6 real tokens and one with 10. With naive mean pooling, how many padding vectors are averaged into the first text’s embedding? What fraction of its embedding comes from padding?
        6. How many cross-encoder passes are needed to compare one query against 1 million documents? How many encoder passes does a bi-encoder need for the query, once the documents are indexed?
        7. In formula (33.7) with \(N = 3\), what is the expected span length in words?
        Answerscheck
        1. IDs: 101, 2026, 3899, 2003, 10140, 102 ([CLS] my dog is cute [SEP]); segment IDs: 0, 0, 0, 0, 0, 0. 2. \(100 / 10 = 10\) times each. 3. \(50{,}265 \times 1{,}024 = 51{,}471{,}360\). Factorized: \(50{,}265 \times 128 + 128 \times 1{,}024 = 6{,}433{,}920 + 131{,}072 = 6{,}564{,}992\), about 7.8 times smaller. 4. \(P_{\text{layer}} = 4(1024^2 + 1024) + 2 \cdot 1024 \cdot 4096 + 4096 + 1024 + 4 \cdot 1024 = 12{,}596{,}224\). No sharing: \(24 \times 12{,}596{,}224 = 302{,}309{,}376\). All shared: \(12{,}596{,}224\). 5. The batch is padded to 10, so 4 padding vectors; \(4/10 = 40\%\) of the average comes from padding. 6. One million cross-encoder passes (one per pair). The bi-encoder needs 1 pass for the query, then 1 million dot products (or an approximate nearest-neighbour search). 7. \(1 \cdot \tfrac{6}{11} + 2 \cdot \tfrac{3}{11} + 3 \cdot \tfrac{2}{11} = \tfrac{18}{11} \approx 1.64\) words.

        Key takeaways

        • BERT’s input is [CLS] A [SEP] B [SEP]; each position’s vector is token + segment + position embedding (33.1). It was pretrained with MLM plus next-sentence prediction (33.3).
        • RoBERTa changed only training: dynamic masking, no NSP with full-sentence packing, big batches (33.4), 10 times more data, longer training, byte-level BPE. MNLI went from 86.6 (BERT-large) to 90.2.
        • ALBERT factorizes the embedding (33.5) and shares one layer across all depths (33.6): 12M parameters for a base model. Fewer parameters is not less compute: ALBERT-xxlarge is about 3 times slower than BERT-large.
        • MLM picks each position with probability 0.15; of those, 80% become [MASK], 10% a random token, 10% stay unchanged (33.2), so only 12% of the inputs show [MASK] and every position might be a target.
        • NSP is mostly topic detection; sentence-order prediction keeps the topic fixed and tests coherence.
        • “Bidirectional” is the attention mask (33.8): the [CLS] row may read every real token. Under a causal mask position 0 reads only itself, so it cannot summarize the text.
        • Whole-text tasks need pooling: [CLS] (33.9) or masked mean (33.10), then one linear layer + softmax (33.11). Never average over padding.
        • Raw BERT vectors are poor for cosine similarity; Sentence-BERT’s siamese fine-tuning (33.13) makes bi-encoder search both good and fast (33.12, 33.14).

        Key formulas

        \[ \mathbf{x}_i = \operatorname{LN}\big(E_{\text{tok}}[t_i] + E_{\text{seg}}[s_i] + E_{\text{pos}}[i]\big), \qquad V H \to V E + E H, \qquad \mathbf{z} = \frac{\sum_i m_i \mathbf{h}_i}{\sum_i m_i}, \qquad \mathbf{p} = \softmax(W\mathbf{z} + \mathbf{b}) \]

        References for this session

        1. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
        2. Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach
        3. Lan et al. (2019). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
        4. Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
        5. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 9: Masked Language Models
        6. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
        7. Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach
        8. Lan et al. (2019). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
        9. Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
        10. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 9: Masked Language Models
        11. Joshi, Chen, Liu, Weld, Zettlemoyer, Levy (2019). SpanBERT: Improving Pre-training by Representing and Predicting Spans
        12. Clark, Luong, Le, Manning (2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
        13. He, Liu, Gao, Chen (2020). DeBERTa: Decoding-enhanced BERT with Disentangled Attention
        14. Warner et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
        15. Gao et al. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings
        16. Vaswani et al. (2017). Attention Is All You Need