Tokenization, Embeddings, and the First Language-Model Forward Pass

0. What we will actually build

A language model cannot multiply the string “The cat sleeps” by a weight matrix. The problem is not just converting letters into integers: we need a reproducible segmentation, a finite vocabulary, a learned vector for each symbol, and a mechanism for making those vectors depend on context.

By the end you should be able to implement a small BPE trainer, explain how WordPiece and Unigram differ, trace shapes from input_ids to output logits, calculate embedding memory, and diagnose multilingual tokenization issues. We assume basic Python and linear algebra; every new tokenizer-specific concept is explained here.

One subtle distinction: a token may be a word, a word fragment, a punctuation mark, a byte sequence, or a special marker. A token ID is its discrete index. An embedding is a vector of trainable floating-point numbers associated with that ID. These are not interchangeable concepts.

1. Why not just use words or characters?

1.1 Word-level tokenization

Suppose our training corpus contains “low”, “lower”, and “new”. A word vocabulary assigns an ID to each complete word. This keeps sequences short, but makes unknown words painful: “newest” may map to [UNK]. A vocabulary of all possible words and spellings is impractical, especially for names, code, inflections, and multiple languages.

1.2 Character-level tokenization

Characters avoid most out-of-vocabulary failures, but lengthen the sequence: “unbelievable” becomes many individual symbols. Self-attention's dense score matrix grows quadratically with sequence length, so longer tokenizations can increase compute and KV-cache demand. Unicode code points, grapheme clusters, and bytes are also not the same thing: a visible emoji may contain multiple code points, and a code point may occupy several UTF-8 bytes.

1.3 Byte-level tokenization

UTF-8 represents text as bytes in the range 0–255. A pure byte model has a tiny base alphabet and can represent arbitrary byte streams; it typically produces long sequences. Byte-level BPE instead starts from bytes and merges frequently co-occurring byte sequences into a larger learned vocabulary. A byte sequence that is one token may not be a complete Unicode character; decoding individual token IDs as text can therefore be misleading. Decode the full byte stream.

1.4 Subword tokenization

Subword models strike a compromise between word and character extremes. They reuse recurring fragments, so an unseen compound can be composed from known pieces. But the way fragments are learned and selected depends on the algorithm.

Critical distinction: SentencePiece is a tokenizer library/framework, not a synonym for Unigram. It supports Unigram and BPE (among other modes) and commonly uses the visible space marker ▁. Likewise, “byte-level” describes the underlying symbol alphabet, while BPE describes the merge algorithm; they can be combined.

2. A tokenizer is a pipeline, not just one algorithm

Hugging Face's tokenizer architecture separates normalization, pre-tokenization, the tokenization model, post-processing for markers, and decoding. This matters because two tokenizers described as “BPE” can give different outputs on exactly the same sentence.

Consider “Café” written as a single é versus e followed by a combining accent. They can look identical but have different Unicode sequences. NFC normalization may unify them; lowercasing and accent removal can change information. A production tokenizer's round-trip behavior therefore depends on the full normalization and decoder settings, not just the subword method.

Pre-tokenization can split punctuation or keep preceding spaces attached to words. For GPT-2-style byte-level BPE, the string “ cat” may be a different learned token from “cat”. In WordPiece, ##ing conventionally means an inside-word continuation. In SentencePiece, ▁cat includes the beginning-of-word/space marker. These visible markers are representation conventions, not literal words.

3. BPE from first principles — and an interactive trainer

BPE repeatedly finds the most frequent adjacent pair of existing symbols in the training corpus, adds the pair to its vocabulary, and replaces occurrences with the merged unit. In the toy character-level example below, begin with a finite corpus and a character alphabet; at each step count pair occurrences.

Here \(\mathcal C\) is the corpus, \(c(w)\) its multiplicity, and \(\#_w(a,b)\) counts adjacent appearances of the pair within the current segmentation of a word. We do not merge across word boundaries in this toy. Real implementations can use a pre-tokenizer and byte-level initial alphabet. During inference, learned merge priorities/ranks are applied to a new input; we do not recompute frequencies from that input.

Think of “low” initially as l · o · w. If l + o is a frequent pair, its merge creates lo · w; a later merge may create low. In a larger corpus, other merges compete at every step. The following trainer performs the actual pair counting and merging in-browser (its initial alphabet is Unicode characters, not production byte-level BPE).

3.1 What is actually stored after BPE training?

At minimum: the base alphabet, the learned ordered merges (or their ranks), the token-to-ID vocabulary, any pre-tokenizer/normalizer configuration, and special-token definitions. A BPE model cannot safely borrow a different vocabulary just because both use the same algorithm. The model's embedding table is trained for one exact mapping.

3.2 Byte-level BPE and OpenAI's tiktoken

In a byte-level BPE implementation such as tiktoken, the initial units are bytes. This is valuable for arbitrary text, including names and scripts unseen during tokenizer training. Byte-level BPE can be lossless in the bytes-to-bytes sense, while a text-oriented normalizer can separately make its input transformation lossy. Always test decode(encode(text)) with the actual tokenizer rather than assuming identical semantics across tokenizers.

4. WordPiece: a different subword model

WordPiece's inference-time segmentation typically uses greedy longest-match-first over its learned vocabulary. For example, with a hypothetical vocabulary that includes play, ##ing, ##er, and ##s, the word “playing” can become play + ##ing. The ## prefix means the piece may continue an existing word.

If no valid continuation can cover the input, a classical WordPiece tokenizer may return [UNK] for that word; byte fallback or hybrid conventions change this behavior. WordPiece training is also not the same as BPE's raw most-frequent-pair criterion; implementations commonly favor pairs associated more strongly than their individual frequencies would suggest. The details can vary by implementation, so do not infer a training algorithm solely from the presence of ## in displayed tokens.

5. Unigram language-model tokenization

BPE learns a sequence of merges. Unigram instead starts with a candidate vocabulary of many pieces, assigns each a probability, and chooses a segmentation that best explains the string under a piece-probability model. Low-utility pieces can be pruned during training; Expectation-Maximization-style estimation and pruning are common.

Because several segmentations can have nonzero probability, Unigram can sample alternative decompositions for data augmentation (subword regularization). For a toy input newest, the options could include new + est, new + e + st, or smaller fragments; their scores are sums of log probabilities, not merge-count heuristics.

5.1 Deriving Viterbi decoding

Define \(D(i)\) as the best log-score for the first \(i\) characters, with \(D(0)=0\). For each candidate piece \(s\) that ends at position \(i\), choose the best predecessor:

Backpointers recover the optimal segmentation. The complexity is approximately \(O(nL)\) when each position tests candidate piece lengths up to \(L\), subject to the actual vocabulary lookup data structure.

6. Special tokens, padding, masks, and chat structure

After segmentation, post-processing may insert markers. Examples include [CLS]/[SEP] for BERT, beginning/end markers for autoregressive models, and role delimiters for instruction-tuned chat models. The exact marker strings and IDs are model-specific. A displayed chat template is part of input formatting; its delimiters cannot safely be guessed.

A padded batch may have two tensors: input_ids ∈ ℤ[B,T] and attention_mask ∈ {0,1}[B,T]. Conventionally, 1 marks non-padding positions and 0 marks padding. This padding mask is distinct from a causal attention mask, which prevents position \(t\) from seeing future positions. Models may combine multiple masks internally.

For decoder-only language modeling, a typical input can be [BOS] The cat sleeps [EOS]; whether the tokenizer automatically adds those markers, and whether the model ties EOS and PAD, depends on its configuration. Tokenizer vocabulary and the model's embedding rows must stay aligned — adding a new token normally requires resizing and training the corresponding embedding row.

7. How discrete IDs become continuous embeddings

Here is the missing bridge. The tokenizer returns IDs such as [2, 4, 1]. The embedding layer owns a trainable matrix \(E\in\mathbb R^{V\times d}\), with one row per vocabulary ID and \(d\) floating-point coordinates per row. Embedding lookup selects the corresponding rows:

The ID itself is categorical — ID 200 is not twice as meaningful as ID 100. Its numerical value is just an index. The vector coordinates initially have no human-readable semantics; training updates them so the network can make useful predictions.

7.1 Lookup is equivalent to one-hot matrix multiplication

Let \(O\in\{0,1\}^{T\times V}\) be a one-hot representation of the ID sequence (one 1 per row). Then \(X=OE\). Implementations use direct row gathering instead of materializing enormous one-hot matrices. Embeddings are therefore an efficient learned linear mapping from discrete symbols into a continuous space.

7.2 Are the vectors “semantic” already?

No, not in isolation. An input embedding is a context-independent lookup: the same token ID retrieves the same vector every time. Contextual hidden states \(H^{(\ell)}\) are produced after transformer layers, attention, and nonlinear transformations. The token “bank” in “river bank” and “bank account” may share its starting row but end with very different contextual vectors.

Not every model adds a position table \(P\). Modern decoder-only models commonly apply rotary position embeddings (RoPE) to query/key vectors inside attention instead. See our older deep dive on RoPE for the rotation derivation. The important lesson here is that token embedding and positional information are separate mechanisms.

8. From contextual vectors to logits and probabilities

Once the transformer has turned input vectors into contextual hidden states \(H\in\mathbb R^{B\times T\times d}\), the language-model head maps every position to one score for every token in the vocabulary:

For a given position, softmax turns scores into probabilities. An autoregressive model predicts the next token using only the prefix:

Many implementations tie input and output embeddings (\(W_{\text{out}}=E\), possibly with architecture-specific details). This saves parameters and couples input/output representations; other models keep them separate. The gradient from next-token prediction flows backward through the output head and transformer to the input embeddings. So the rows of \(E\) are learned — tokenizer IDs themselves remain fixed.

8.1 Context length and token fertility

Fertility is the number of tokens used per unit of text, often tokens per word or tokens per character. A tokenizer that uses 1.8× as many tokens for a language or domain can hit the context limit sooner, consume more attention computation, and need more cache at inference. Compare languages using a clearly defined unit: whitespace-defined “words” are not equally natural in every language. Token/Unicode-character and token/UTF-8-byte measurements can complement word-based metrics.

9. Interactive lab: memory and sequence-budget trade-offs

Research prompt: compare a 32k versus a 128k vocabulary at the same \(d\), then separately measure real tokenizer fertility on English, Persian, code, and emoji. Do not assume bigger vocabulary always improves total system cost: shorter sequences can offset some matrix growth, depending on workload and architecture.

10. Reproducible implementations: interleaved code and explanations

10.1 Inspect real WordPiece, BPE, and Unigram tokenizers

The following example loads three released pretrained tokenizers, prints exact tokens/IDs, performs a round-trip, and compares multilingual fertility. It needs pip install transformers sentencepiece torch and internet access to download the tokenizer files. The displayed outputs are produced by the code when you run it — they are not fabricated token IDs.

from transformers import AutoTokenizer
texts = [
    "The unbelievable player is playing.",
    "مدل‌های بینایی و زبان چگونه کار می‌کنند؟",
    "👩‍💻 AI + vision = 2026",
]
models = {
    "WordPiece / BERT": "bert-base-uncased",
    "byte-level BPE / GPT-2": "gpt2",
    "SentencePiece / mT5": "google/mt5-small",
}
for name, checkpoint in models.items():
    tokenizer = AutoTokenizer.from_pretrained(checkpoint)
    print("\n", name, "| vocab =", len(tokenizer))
    for text in texts:
        ids = tokenizer(text, add_special_tokens=False).input_ids
        tokens = tokenizer.convert_ids_to_tokens(ids)
        decoded = tokenizer.decode(ids, clean_up_tokenization_spaces=False)
        print("text:", text)
        print("IDs:", ids)
        print("tokens:", tokens)
        print("round trip:", decoded)
        print("tokens per char:", len(ids) / max(1, len(text)))

Interpretation: BERT's lowercase normalization may prevent exact round-trip. GPT-2's byte-level tokens may display a byte-to-visible-character mapping rather than literal user text. SentencePiece is a framework; the mT5 checkpoint is a specific SentencePiece configuration. Changes in tokenizer files or versions should be recorded in a reproducible comparison.

10.2 Train a real tiny BPE tokenizer

Now replace the conceptual JavaScript demo with a Hugging Face tokenizers training pipeline. This demonstration uses word-level pre-tokenization and a finite toy corpus, not the byte-level conventions of GPT-2:

from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders
corpus = [
    "low low lower lowest",
    "newer newest lower",
    "wide wider widest",
] * 100
tok = Tokenizer(models.BPE(unk_token="[UNK]"))
tok.pre_tokenizer = pre_tokenizers.Whitespace()
tok.decoder = decoders.BPEDecoder()
trainer = trainers.BpeTrainer(vocab_size=65, special_tokens=["[UNK]"])
tok.train_from_iterator(corpus, trainer=trainer)
out = tok.encode("lowest wider")
print("tokens:", out.tokens)
print("IDs:", out.ids)
print("decoded:", tok.decode(out.ids))
print("vocab size:", tok.get_vocab_size())

Why multiply corpus lines? Only to give the toy dataset repeated counts; frequencies and the requested vocabulary cap determine which merges get learned. Observe how the trained merges change when you replace repeated “low” with many “wide” examples. This experiment makes the training objective concrete. Important: with Whitespace(), the decoder may not restore the original spaces; use compatible whitespace-aware pre-tokenization and decoding when exact text round-trip is required.

10.3 Build embeddings, a tiny context model, and a real cross-entropy loss

We use an untied output head here to make the gradient experiment easy to understand. There is no pretrained transformer and this toy network will not generate meaningful language: it verifies the tensor mechanics of embedding lookup, contextual mixing, output vocabulary scores, and parameter optimization.

import torch
import torch.nn as nn
torch.manual_seed(7)

V, d = 8, 4
ids = torch.tensor([[1, 3, 5, 2]])  # [B=1, T=4]
embedding = nn.Embedding(V, d)
mix = nn.Linear(d, d, bias=False)
head = nn.Linear(d, V, bias=False)

x = embedding(ids)          # [1, 4, 4], row lookup
h = torch.tanh(mix(x))      # [1, 4, 4], tiny non-Transformer layer
logits = head(h)            # [1, 4, 8], one logit per vocab ID
targets = ids[:, 1:]        # [1, 3]: predict next ID
loss = nn.functional.cross_entropy(
    logits[:, :-1, :].reshape(-1, V), targets.reshape(-1)
)
loss.backward()

assert x.shape == (1, 4, d)
assert logits.shape == (1, 4, V)
print("loss:", round(loss.item(), 4))
print("nonzero input embedding gradient rows:",
    (embedding.weight.grad.abs().sum(dim=1) > 0).nonzero().flatten().tolist())
# With an untied output head only input rows 1,3,5 receive nonzero
# gradient here; the last token ID 2 has no prediction loss.

Why is row 2 absent from this gradient? The last position's logits are excluded by logits[:, :-1]; row 2 occurs only there, so this simplified independent-per-position model has no loss path from that row. In a real transformer with causal attention, earlier token embeddings influence later representations, but future tokens still cannot affect earlier predictions. If you tie the output head to \(E\), the softmax output path can create gradients on many vocabulary rows even if those tokens were not present in the input.

10.4 The actual embedding matrix in a pretrained Hugging Face model

from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "sshleifer/tiny-gpt2"  # tiny checkpoint, only for inspection
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()

ids = tokenizer("Tokens become vectors.", return_tensors="pt").input_ids
E = model.get_input_embeddings()
X = E(ids)
with torch.no_grad():
    out = model(input_ids=ids, output_hidden_states=True)
print("input_ids shape:", tuple(ids.shape))
print("embedding table:", tuple(E.weight.shape))
print("input embeddings:", tuple(X.shape))
print("last hidden state:", tuple(out.hidden_states[-1].shape))
print("vocabulary logits:", tuple(out.logits.shape))

The checkpoint is intentionally small to make inspection cheap; it is not a quality benchmark. Compare the input embedding vectors against the final hidden states for the same token in two different sentences. You will see why static lookup vectors and contextual representations cannot be treated as synonyms.

11. Research exercises: predictions, ablations, and failures

1. Vocabulary ablation: train BPE vocabularies of 100, 300, and 1,000 units on the same larger corpus. Record average tokens per sentence and OOV/fallback behavior. Repeat on held-out text.
2. Cross-language comparison: compare English, Persian, code, and emoji using token counts per UTF-8 byte and per Unicode code point. Inspect how zero-width joiners, diacritics, Persian half-space, and mixed RTL/LTR strings affect results.
3. Normalization ablation: change case, Unicode form, and accents, then record whether the original string can be reconstructed. Distinguish tokenizer correctness from irreversibility introduced by normalization.
4. Embedding experiment: initialize \(E\) randomly, train a toy next-token task, and track cosine similarity for selected rows over optimizer steps. State clearly why similarity does not automatically equal semantic equivalence.
5. Compute budget: combine actual token lengths with measured attention latency, peak VRAM, and batch sizes. Compare a 32k and 128k tokenizer-model pair fairly — never swap the tokenizer while keeping incompatible embedding weights.

12. Check your understanding (answers included)

13. Sources and further reading (with purpose)

This chapter is an independently written expansion. The uploaded Vision Language Models: Building VLMs with Hugging Face (O'Reilly, 2026) introduces tokenizers in Chapter 1 and directly demonstrates tokenizer → input_ids → embed_tokens in Chapter 3's NanoVLM example. Its relevant illustrations are around book pages 38–39 (language preprocessing), 101–103 (text embeddings and image/text concatenation), and 105 (cross-entropy loss). It does not itself supply a full comparison of BPE/WordPiece/Unigram; that material is added here from tokenizer-specific sources.

• Hugging Face Tokenizers — Components: normalization, pre-tokenization, BPE, WordPiece, Unigram and post-processing.
• Hugging Face LLM course — Normalization and pre-tokenization: concrete pipeline examples and differences.
• SentencePiece documentation: framework, Unigram/BPE modes, normalization and subword sampling.
• OpenAI tiktoken: practical fast byte-level BPE and _educational merge visualizations.
• Original research note — Rotary Positional Embeddings: derivation and PyTorch reference for the positional mechanism.
• Original research note — ControlNet: example of the long-form intuition → architecture → math → code style used for this course rewrite.