Embeddings: turning token IDs into vectors
A tokenizer gives the model a list of integers. The first layer of every LLM turns each integer into a vector by picking one row of a big learned table. This session shows why that lookup is the same as multiplying a one-hot vector by a matrix, how training makes words that appear in similar contexts get similar vectors (word2vec), what cosine similarity can and cannot tell you, and how much memory the table costs in real models.
What you will learn
- Why a token ID cannot be fed to a network as a number, and why one-hot vectors are not enough
- The embedding layer: one-hot times a matrix equals a table lookup, in the forward and in the backward pass
- The distributional idea behind word2vec: skip-gram, negative sampling, and its link to PMI
- Cosine similarity, nearest neighbors and analogies, and where this geometry fails (polysemy, antonyms, bias, anisotropy)
- Embedding size and parameter count, |V| × d, with real numbers for GPT-2, Llama and Qwen
- Weight tying between the input embedding and the output layer
- How images and video frames enter the same vector space in a VLM
Why this matters
A tokenizer turns text into a list of integers. “The cat sat” might become [791, 8415, 7731]. A neural network cannot do anything useful with these integers directly. The very first layer of every LLM has one job: turn each integer into a vector of real numbers that the rest of the network can compute with.
That layer is called the embedding layer, and the vectors it produces are called embeddings. An embedding is a list of \(d\) numbers that stands for one token. The layer is very simple, a big table with one row per token, but it carries several ideas that come back again and again in this course:
- Lookup is matrix multiplication. Picking row \(i\) of a table is the same as multiplying a one-hot vector by a matrix. This is why the table can be trained with backpropagation like any other layer.
- Meaning comes from context. Training pushes tokens that appear in similar contexts toward similar vectors. This idea, made famous by word2vec, is the root of semantic search, retrieval and CLIP.
- Geometry has limits. Cosine similarity between embeddings is useful, but it mixes up opposites, ignores the many meanings of one word, and copies social bias from the data.
- The table is big. It has \(|V| \times d\) numbers. In Llama 3 8B the input table alone has 525 million parameters. In small models the embedding tables can be more than a quarter of all weights.
Interviewers like this topic because it looks easy and hides real questions: “Why is an embedding layer the same as a linear layer?”, “Which rows get a gradient?”, “How many parameters are in the embeddings of Llama 3 8B?”, “What is weight tying?”, “Why are all cosine similarities high in some models?”. By the end of this session you can answer each one with a formula and a number. In Session 10 we built the tokenizer that produces the IDs; here we pick up those IDs and give them meaning.
From an ID to a vector
Why not use the ID itself?
The simplest idea is to feed the integer as one input number. This fails for a clear reason: the network would treat the IDs as amounts. Token 8415 would be “twice as much” as token 4207, and tokens with close IDs would look alike. But IDs are given by the tokenizer in the order it merged pieces. Their size means nothing. A linear layer on a single number can only scale it, so the model would see a meaningless order.
One-hot vectors
The fix is to give each token its own direction. A one-hot vector for token \(i\) has length \(|V|\) (the vocabulary size), a 1 in position \(i\), and 0 everywhere else:
Now no token is “bigger” than another. But one-hot vectors have two problems. First, they are huge: with \(|V| = 128{,}256\) (Llama 3) each token would be a vector of 128,256 numbers, almost all zero. Second, every pair of different tokens is perpendicular: \(\mathbf{e}_i \cdot \mathbf{e}_j = 0\) for \(i \ne j\). So “cat” is exactly as far from “dog” as it is from “carburetor”. One-hot vectors carry no notion of similarity at all.
The embedding matrix
We fix both problems by multiplying the one-hot vector by a learned matrix \(E\) with \(|V|\) rows and \(d\) columns, where \(d\) is much smaller than \(|V|\) (for example \(d = 4096\)). Write the one-hot vector as a row vector. Then:
- Read the symbols
- \(|V|\)
- the vocabulary size: how many different tokens exist
- \(d\)
- the embedding size (also called the model width or hidden size)
- \(E\)
- the embedding matrix: one row of \(d\) numbers for each token. Its numbers are learned during training.
- \(\mathbf{e}_i^\top E\)
- a one-hot row vector times the matrix. All rows of \(E\) are multiplied by 0 except row \(i\), which is multiplied by 1.
- \(E_{i,:}\)
- row \(i\) of \(E\): the embedding of token \(i\)
So multiplying by a one-hot vector just selects one row. That is why we call the layer a lookup table: in code, we never build the one-hot vector; we read row \(i\) directly. For a sequence of \(T\) tokens, stack the one-hot rows into a matrix \(O \in \R^{T \times |V|}\):
Each row of \(X\) is the embedding of one token in the sequence. If the same token appears twice, the same row is copied twice.
Let the vocabulary be the (ID 0), cat (1), sat (2), on (3), mat (4), . (5), with \(d = 4\). Row 1 of \(E\) is \((0.8, 0.5, -0.2, 0.1)\). The one-hot vector for cat is \((0, 1, 0, 0, 0, 0)\). The product is \(0 \cdot E_{0,:} + 1 \cdot E_{1,:} + 0 \cdot E_{2,:} + \dots = (0.8, 0.5, -0.2, 0.1)\): exactly row 1.
Cost: the full product needs \(|V| \cdot d = 24\) multiplications and about as many additions, all but 4 of them with a zero. The lookup needs no arithmetic at all, only a copy of 4 numbers. For Llama 3 8B the one-hot product would cost \(2 \times 128{,}256 \times 4096 \approx 1.05\) billion FLOPs per token, about 6.5% of the whole forward pass (about \(2 \times 8.03\) billion FLOPs). The lookup costs nothing but one memory read of 4,096 numbers.
How to read Figure 1
Step through the loop. The right panel writes each line of the algorithm with the real numbers: for “cat” (ID 1), step \(j = 0\) adds \(0 \cdot (0.1, -0.3, 0.2, 0.0)\), step \(j = 1\) adds \(1 \cdot (0.8, 0.5, -0.2, 0.1)\), and the four other steps add zeros. After 6 rows, \(\mathbf{x} = (0.8, 0.5, -0.2, 0.1)\), exactly row 1. The counter shows 24 multiply-adds, of which only 4 were not with a zero; the lookup gives the same vector for free. Click a token at any moment: the loop is recomputed for that token. Click the two copies of “the”: they give the same vector, because the embedding layer does not know where a token is in the sentence. Position is added later (session 13).
Training the table
At the start, \(E\) is random. Every row is learned with backpropagation, together with the rest of the network. To see how, use formula (11.3) and the chain rule from session 4. Let \(\mathbf{g}_t = \partial \mathcal{L} / \partial X_{t,:}\) be the gradient that flows back into the embedding of position \(t\). Then the gradient for row \(j\) of the table is:
- Read the symbols
- \(\mathcal{L}\)
- the training loss (for an LLM, the next-token cross-entropy)
- \(\mathbf{g}_t\)
- the gradient of the loss with respect to the embedding at position \(t\), a vector of \(d\) numbers
- \(t : \mathrm{id}_t = j\)
- all positions in the batch where token \(j\) appears
Two facts follow, and both matter in practice:
- Only rows of tokens in the batch get a gradient. All other rows get exactly zero. A batch of a few thousand tokens touches only a small part of a 128,256-row table.
- Repeated tokens add up. If “the” appears five times, its row receives the sum of five gradients. Frequent tokens are updated often and strongly; rare tokens are updated rarely.
In matrix form, \(\partial \mathcal{L}/\partial E = O^\top G\), where \(G\) stacks the \(\mathbf{g}_t\). This is the usual gradient of a linear layer (input transposed times output gradient), with the one-hot matrix as input. PyTorch computes it with a scatter-add, never with the dense product.
The sentence “the cat sat on the mat” has IDs \((0, 1, 2, 3, 0, 4)\). Token 0 appears at positions 1 and 5. Suppose the gradients arriving there are \(\mathbf{g}_1 = (0.2, -0.1, 0, 0.3)\) and \(\mathbf{g}_5 = (-0.5, 0.4, 0.1, 0.1)\). Then the gradient of row 0 is \(\mathbf{g}_1 + \mathbf{g}_5 = (-0.3, 0.3, 0.1, 0.4)\). Row 5 (the token “.”) is not in the sentence, so its gradient is \((0, 0, 0, 0)\).
Figure 2 runs the backward pass of the embedding layer on this sentence, one position at a time, and then takes one gradient step. Watch which rows of the table change.
How to read Figure 2
Each step adds one row of \(G\) into the row of \(\nabla E\) that belongs to that token. At \(t = 5\) the second “the” lands on row 0 again, so row 0 becomes \(\mathbf{g}_1 + \mathbf{g}_5 = (-0.3, 0.3, 0.1, 0.4)\). Row 5 (“.”) never receives anything, and after the SGD step it is the only row of \(E\) that did not change. Now tick Tied output layer. The same table is also used as the output layer (formula 11.14) to predict the token after “mat”, which is “.”. Two extra steps compute the logits \(z_j = E_{j,:} \cdot \mathbf{h}\), the softmax \(p\), and the output-side gradient \((p_j - y_j)\,\mathbf{h}\) of formula (11.15) for every row. With the fixed hidden vector \(\mathbf{h} = (0.5, -0.3, -0.4, 0.8)\), \(p_{\text{.}} = 0.305\), and row 5 now gets the gradient \((-0.348, 0.209, 0.278, -0.556)\): with \(\eta = 0.5\) it moves from \((-0.1, 0, -0.6, 0.9)\) to \((0.074, -0.104, -0.739, 1.178)\). With tying, all 6 rows change.
Practical details
- Initialization. PyTorch’s
nn.Embeddingstarts every entry from a standard normal distribution, \(\mathcal{N}(0, 1)\). GPT-style code such as nanoGPT uses a much smaller standard deviation, 0.02. [Certain] The original transformer multiplies embeddings by \(\sqrt{d}\) before adding position signals. [Certain] - The optimizer still moves unused rows. With Adam, a row whose gradient is zero this step still moves if its momentum from earlier steps is not zero, and AdamW’s weight decay shrinks every row at every step. A dense optimizer updates the whole table every step. [Certain]
- Rare tokens stay close to their random start. A token that almost never appears in the training text almost never receives a gradient. When such a token shows up at test time, the model may behave strangely; people call these “glitch tokens”. [Likely]
- Adding new tokens. When you add special tokens for a chat template (session 9), you add rows to \(E\). A common choice is to start them at the mean of the existing rows instead of random noise, so they do not look like extreme outliers to the first layer. [Likely]
Where meaning comes from: word2vec
Training an LLM to predict the next token gives its embedding table structure for free. But the cleanest way to see why embeddings become meaningful is an older and smaller model: word2vec (Mikolov et al., 2013). It trains only embeddings, with no deep network at all.
The idea behind it is the distributional hypothesis: words that appear in similar contexts have similar meanings. You never see a definition of “dog”, but you see “the dog eats”, “I feed the dog”, “the dog sleeps”. “Cat” appears in the same kind of sentences. If a model must use a word’s vector to predict its neighbors, then “cat” and “dog” need similar vectors, because they have similar neighbors.
Skip-gram: predict the context from the word
Slide a window over the text. For each center word \(c\), every word within \(m\) positions of it is a context word \(o\). With \(m = 2\), the sentence “the cat eats fish” gives pairs like (cat, the), (cat, eats), (cat, fish). Skip-gram learns two tables: an input vector \(\mathbf{v}_w\) for each word used as a center, and an output vector \(\mathbf{u}_w\) for each word used as a context. The probability of seeing \(o\) near \(c\) is a softmax over the whole vocabulary:
A large dot product means “these two words often appear together”. Training maximizes \(\log p(o \mid c)\) over all pairs. The problem is the sum in the denominator: it runs over the whole vocabulary (word2vec’s released Google News vectors cover 3 million words and phrases [Certain]), for every pair.
Negative sampling
The fix is negative sampling (SGNS: skip-gram with negative sampling). Instead of a softmax over all words, we solve a small yes-or-no problem: “Is \(o\) a real neighbor of \(c\)?” For each real pair we also draw \(k\) random negative words \(n_1, \dots, n_k\) that probably are not neighbors. The loss pushes the real pair’s score up and the fake pairs’ scores down:
- Read the symbols
- \(\sigma(s)\)
- the sigmoid \(1/(1 + e^{-s})\): turns a score into a probability between 0 and 1
- \(\mathbf{v}_c\)
- the input (center) vector of word \(c\). After training, these are the word embeddings people use.
- \(\mathbf{u}_o, \mathbf{u}_{n_i}\)
- output (context) vectors of the real neighbor and of the negative words
- \(k\)
- the number of negatives per real pair. The paper suggests 5 to 20 for small datasets and 2 to 5 for large ones. [Certain]
Now each pair costs \(k + 1\) dot products instead of \(|V|\). The gradient with respect to the center vector has the same “prediction minus target” shape as softmax cross-entropy in session 2:
Read it like this: the center vector moves toward its real neighbor (the first term has a negative factor, and we step against the gradient) and away from the negatives. The step is large when the model is wrong and small when it is already right.
Negatives are not drawn uniformly. They come from the word frequencies raised to the power \(3/4\):
The power \(3/4\) flattens the distribution a little: very common words like “the” are drawn less often than their raw frequency, and rare words more often. With counts 100, 10 and 1, the raw probabilities are 0.901, 0.090 and 0.009; after the power they become 0.827, 0.147 and 0.026. The paper found \(3/4\) worked better than both the raw counts and the uniform distribution. [Certain] word2vec also randomly drops very frequent words from the training text (subsampling), so that “the” does not dominate the pairs.
Take \(d = 2\), center vector \(\mathbf{v}_c = (0.5, -0.2)\), real neighbor \(\mathbf{u}_o = (0.4, 0.1)\), one negative \(\mathbf{u}_n = (-0.3, 0.6)\).
Scores: \(\mathbf{u}_o \cdot \mathbf{v}_c = 0.20 - 0.02 = 0.18\), so \(\sigma(0.18) = 0.545\). \(\mathbf{u}_n \cdot \mathbf{v}_c = -0.15 - 0.12 = -0.27\), so \(\sigma(0.27) = 0.567\). Loss (11.6): \(-\log 0.545 - \log 0.567 = 1.174\).
Gradient (11.7): \((0.545 - 1)(0.4, 0.1) + \sigma(-0.27)\,(-0.3, 0.6) = (-0.312, 0.214)\). One step with learning rate 0.5 gives \(\mathbf{v}_c = (0.656, -0.307)\). The new scores are \(0.232\) (real, up) and \(-0.381\) (negative, down), and the loss drops to \(1.105\).
Figure 3 runs exactly this algorithm on a toy corpus of 12 short sentences: half about a cat and a dog, half about a car and a bus. The embeddings have only \(d = 2\) numbers, so we can draw them.
How to read Figure 3
At 0 epochs the vectors are random: “cat” and “dog” point in almost opposite directions (cosine about −0.98). Move the slider or press Play. In the first few epochs all words move together, because they all share the very common context “the”. Then the two topics separate: “cat” and “dog” end up pointing the same way, “car” and “bus” too, while “cat” and “car” become nearly perpendicular. No one told the model which words are animals. It found this only from which words share neighbors. The loss in the readout goes down and then shakes, because the negatives are random at every step.
CBOW, GloVe and fastText
word2vec has a second variant, CBOW (continuous bag of words), which predicts the center word from the average of its context vectors. Two later methods are worth knowing by name:
- GloVe (Pennington et al., 2014) first counts how often each pair of words appears together in the whole corpus, then fits vectors so that their dot product matches the log of that count.
- fastText (Bojanowski et al., 2017) builds a word’s vector as the sum of vectors of its character n-grams (pieces like “
<ca”, “cat”, “at>”). It can make a vector for a word it never saw. This is the same goal that subword tokenizers serve in LLMs (session 8).
What SGNS really computes
Levy and Goldberg (2014) showed a neat link between the counting view and the prediction view. At the best solution, the SGNS score of a word–context pair equals the pointwise mutual information (PMI) of the pair, shifted by \(\log k\):
PMI measures how much more often two words appear together than they would by chance. So SGNS is secretly a low-rank factorization of a (shifted) PMI table: it squeezes a \(|V| \times |V|\) table of co-occurrence statistics into two thin \(|V| \times d\) matrices. [Certain] This tells you what embeddings can and cannot hold: they hold co-occurrence statistics, not definitions.
Cosine similarity and the geometry of meaning
Once we have embeddings, we compare them with cosine similarity, the cosine of the angle between two vectors:
It is 1 for the same direction, 0 for perpendicular vectors and −1 for opposite ones. We use the angle and not the plain dot product because embedding lengths often track word frequency, not meaning: frequent words tend to have different lengths from rare ones. Dividing by the lengths removes that effect. For vectors of length 1 the cosine and the dot product are the same number.
Finding the nearest neighbors of a word is a loop over the table: score every row against the query, keep the \(k\) best. Figure 4 runs this loop on 11 hand-made 2-dimensional word vectors, so that each cosine is simply the cosine of the angle you see. The vectors of the frequent words “the” and “and” are long, like real frequent tokens often are.
How to read Figure 4
With Cosine and query “cat”, the loop finds dog (0.990), kitten (0.978) and puppy (0.951): the words with the smallest angle. Switch to Dot product and run again: now “the” wins with 1.472, and “and” enters the top 3, only because their arrows are long. This is why we divide by the lengths. With Tied LM head the query is a hidden vector \(\mathbf{h}\) (angle 35°, length 2). The scores are logits, and the frequent words win again: \(p(\text{the}) = 0.299\), \(p(\text{dog}) = 0.180\), \(p(\text{cat}) = 0.155\). For an output layer that is useful: a long vector acts like a high prior for a frequent token. Make \(\lVert\mathbf{h}\rVert\) larger and the softmax gets sharper: at length 2.4, \(p(\text{the}) = 0.344\); at length 0.5, only 0.136. Turn \(\mathbf{h}\) to 150°: “and” still wins (0.221) just ahead of “bus” (0.197), because it is long. At 165° the vehicle words take the top 3.
Two things people do with cosine:
- Nearest neighbors. Rank all words by cosine to a query word. In good embeddings the top neighbors of “frog” are other frogs and toads, and the top neighbors of “Paris” are other capitals or French cities.
- Analogies. The famous example: \(\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}}\) is close to \(\mathbf{v}_{\text{queen}}\). The idea is that the difference “man → woman” is roughly the same direction for many word pairs.
- Read the symbols
- \(a, b, c\)
- the three given words, as in “\(b\) is to \(a\) as \(c\) is to ?”. For \(\text{king} - \text{man} + \text{woman}\): \(a\) = king, \(b\) = man, \(c\) = woman.
- \(w \notin \{a, b, c\}\)
- the answer may not be one of the three input words. This small rule is important, as Figure 5 shows.
- \(\hat{w}\)
- the predicted answer: the word whose vector has the largest cosine with the target point
In the hand-made table of Figure 5 each word has 5 features (royal, male minus female, person, food, young). king \(= (0.9, 0.8, 0.9, 0, 0.1)\) and queen \(= (0.9, -0.8, 0.9, 0, 0.1)\). The dot product is \(0.81 - 0.64 + 0.81 + 0 + 0.01 = 0.99\). Both lengths are \(\sqrt{0.81 + 0.64 + 0.81 + 0.01} = \sqrt{2.27} = 1.507\). So \(\cos = 0.99 / 2.27 = 0.436\). The target point \(\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}}\) is \((0.9, -0.8, 0.9, 0, 0.1)\), which is exactly queen: cosine 1.
How to read Figure 5
With no noise the vectors were built so that the analogy is perfect, and queen wins with cosine 1. Real embeddings are not like this. Raise the noise to 0.30: queen still wins when the input words are excluded, but uncheck the box and the top answer becomes “woman”, one of the inputs. This is a known weakness of analogy tests: the target point is often closest to one of the words you started from, and the famous results depend on the exclusion rule. Try \(\text{prince} - \text{boy} + \text{girl}\), or build your own.
The limits of this geometry
Cosine similarity between word embeddings is useful, but it is easy to read too much into it. Here are the main limits.
- One vector per word, many meanings. “Bank” (river) and “bank” (money) get one static vector, a blend of both meanings weighted by how often each appears. This is the main reason we need contextual embeddings, where the vector depends on the sentence. That is the subject of the next session, session 12.
- Similar is not the same as synonym. “Hot” and “cold” appear in the same contexts (“the water is ___”), so their cosine is high even though they mean opposite things. Distributional similarity measures “used in the same places”, which mixes synonyms, antonyms and related words. [Likely]
- Frequency effects. Very frequent and very rare words behave differently: rare words get few updates and noisy vectors, and some words (“hubs”) appear among the nearest neighbors of very many other words. [Likely]
- Social bias. Embeddings copy the statistics of their text, including stereotypes. Bolukbasi et al. (2016) found that word2vec vectors trained on news text complete “man is to computer programmer as woman is to ?” with “homemaker”. [Certain] Any system built on these vectors (search, ranking, hiring tools) can inherit such patterns.
- Anisotropy: all vectors in a narrow cone. In many trained models the embeddings share a large common direction, so almost every pair has a high positive cosine. Then “0.8” does not mean “very similar”; it may be the average. This was reported for static word vectors (Mu and Viswanath, 2018), for the input and output embeddings of generation models (Gao et al., 2019), and for the hidden states of BERT and GPT-2 (Ethayarajh, 2019). [Certain]
We can see the effect of a common direction with a short calculation. Let every vector be \(\mathbf{x} = \mathbf{z} + \mathbf{m}\), where \(\mathbf{z}\) is a random vector with \(d\) independent entries of variance 1 (so \(\lVert \mathbf{z} \rVert^2 \approx d\)) and \(\mathbf{m}\) is a fixed shared vector with \(\lVert \mathbf{m} \rVert^2 = \alpha^2 d\). Random parts of different vectors are nearly perpendicular in high dimension, so \(\mathbf{x} \cdot \mathbf{x}' \approx \alpha^2 d\) and \(\lVert \mathbf{x} \rVert^2 \approx (1 + \alpha^2)\, d\). Therefore:
With \(\alpha = 1\) (the shared part as long as the random part), unrelated vectors already have cosine about 0.5. With \(\alpha = 2\), about 0.8. The simple fix is centering: subtract the mean vector of the collection before computing cosines. “All-but-the-top” goes one step further and also removes the few strongest shared directions. [Certain]
How to read Figure 6
At \(\alpha = 0\) the cosines are spread around 0, as expected for unrelated random vectors. Increase \(\alpha\): the whole histogram slides to the right and becomes narrow, and its mean follows the dashed line \(\alpha^2/(1 + \alpha^2)\). The vectors have not become “more similar” in any useful sense; they all just share one direction. Now tick centering: the histogram jumps back to 0 for every \(\alpha\), because subtracting the mean removes the shared vector (for 40 vectors the mean pair cosine after centering is about \(-1/39 = -0.026\)). When you see high cosines everywhere in a real model, check this first.
Embedding size and parameter count
The embedding matrix has one row of \(d\) numbers per token, so its size is simple:
Why two tables? At the end of an LLM, the last hidden vector \(\mathbf{h} \in \R^d\) must become one score (logit) per token. That is done by another \(|V| \times d\) matrix, the output embedding or “LM head”. Many models keep it separate; others reuse the input table, which is called weight tying:
- Read the symbols
- \(\mathbf{h}\)
- the final hidden vector at one position, after all transformer blocks
- \(E\)
- the same matrix as in formula (11.2), used a second time
- \(z_j\)
- the logit of token \(j\): the dot product of token \(j\)’s embedding with \(\mathbf{h}\). Softmax over \(\mathbf{z}\) gives the next-token probabilities.
Tying makes a nice symmetry: a token’s input vector is also the direction the model must point to in order to predict that token. Press and Wolf (2017) showed that tying lowers perplexity of language models while removing a whole matrix of parameters. [Certain] It also means every row of \(E\) gets a gradient at every step through the output side, because every logit is in the softmax. With cross-entropy, the output-side gradient of row \(j\) for one position is:
- Read the symbols
- \(p_j\)
- the predicted probability of token \(j\) as the next token
- \(y_j\)
- 1 if token \(j\) is the true next token, else 0
- \(\mathbf{h}\)
- the final hidden vector at that position, as in (11.14)
The true token’s row moves toward \(\mathbf{h}\) (its factor \(p_j - 1\) is negative), and every other row moves a little away from \(\mathbf{h}\), in proportion to its probability. So even a token that never appears in the batch gets a small gradient, which helps rare tokens. Tick “Tied output layer” in Figure 2 to see this: row 5 (“.”) is not in the input sentence, but it gets the largest gradient of all, because “.” is the true next token. And the “Tied LM head” mode of Figure 4 shows the forward side: the logits are one dot-product search over the rows of \(E\).
The cost: one matrix must serve two roles, which large models with plenty of parameters often prefer not to do.
| Model | \(|V|\) | \(d\) | Tied? | Embedding parameters | Share of all parameters |
|---|---|---|---|---|---|
| GPT-2 small | 50,257 | 768 | yes | 38.6 M | 31.0% of 124.4 M |
| Llama 2 7B | 32,000 | 4,096 | no | 2 × 131.1 M = 262.1 M | 3.9% of 6.74 B |
| Llama 3 8B | 128,256 | 4,096 | no | 2 × 525.3 M = 1.05 B | 13.1% of 8.03 B |
| Llama 3.2 1B | 128,256 | 2,048 | yes | 262.7 M | 21.3% of 1.24 B |
| Qwen2.5-0.5B | 151,936 | 896 | yes | 136.1 M | 27.6% of 0.49 B |
Sizes from the public model configurations; the totals match a full count of all layers. [Certain for GPT-2 and the Llama models; Likely for Qwen2.5-0.5B.] Three lessons:
- Vocabulary growth is expensive. Llama 2 to Llama 3 kept \(d = 4096\) but grew \(|V|\) four times, so the embedding parameters grew four times, from 262 M to 1.05 B.
- Small models pay the most. The transformer blocks grow like \(d^2\) per layer, but the embeddings grow only like \(d\). For a small \(d\), a large vocabulary is a big share of the model. This is why small models almost always tie their embeddings.
- Embedding parameters are cheap in compute. The input lookup costs no FLOPs. This is why scaling-law papers often count “non-embedding parameters” separately (session 25). The output layer is different: \(E\mathbf{h}\) costs \(2|V|d\) FLOPs per token, and the logits tensor of shape \((B, T, |V|)\) can take a lot of memory.
How to read Figure 7
The colored part of the bar is the embedding tables, the grey part is everything else. Load Llama 3 8B and move \(|V|\) down to 32,000: the embeddings shrink from 1.05 B to 262 M parameters, which is the Llama 2 tokenizer size. Load Qwen2.5-0.5B and untie: the model gains 136 M parameters and the embeddings become 43% of it. The readout also shows the memory of one table in bfloat16 and the FLOPs of the output layer per token, \(2|V|d\).
How to choose \(d\) and \(|V|\)
In an LLM the embedding size is not chosen alone: it is the width \(d\) of the whole residual stream, so it is set by the size of the model (768 for GPT-2 small, 4,096 for 7B to 8B models). The vocabulary size is a trade-off from session 8: a larger vocabulary makes sequences shorter (fewer tokens per sentence, so faster attention and longer effective context), but costs \(2|V|d\) more parameters in untied models and a bigger softmax at the output. For classic word2vec, \(d\) between 100 and 300 was common; the released Google News vectors use 300. [Certain]
Beyond text: images and video in the same space
A VLM must put images into the same \(d\)-dimensional space as text tokens, so that the transformer can read them side by side. Images have no vocabulary, so there is no table to look up. Instead:
- Patch embedding. The Vision Transformer cuts an image into small squares (patches), for example \(16 \times 16\) pixels. A \(224 \times 224\) image gives \(14 \times 14 = 196\) patches, each with \(16 \cdot 16 \cdot 3 = 768\) numbers. One linear layer maps each patch to a \(d\)-dimensional vector (session 164). [Certain] This is the image version of formula (11.2), except the input is a real-valued vector instead of a one-hot one.
- Projection into the LLM. The vision encoder’s output vectors go through a small projector (a linear layer or an MLP) that maps them to the LLM’s width \(d\). From then on they are treated like token embeddings.
- Video. A video is a sequence of frames, so it becomes many visual embeddings, often grouped by time. This is why token counts for video grow so fast, and why video systems compress or select frames.
The word2vec idea also returns at full scale: CLIP (session 166) trains an image encoder and a text encoder so that matching image–caption pairs have high cosine and non-matching pairs have low cosine. It is the same “pull real pairs together, push random pairs apart” as negative sampling (11.6), with images in place of center words and captions in place of context words.
Where this is used
| Application | What the embeddings do | Link |
|---|---|---|
| Every LLM and VLM | Input lookup (11.2) turns token IDs into vectors; the output embedding (11.14) turns the last hidden vector into logits | session 12 |
| Semantic search and RAG | Documents and queries become vectors; ranking by cosine (11.10) finds related text, not only matching words | session 150 |
| Text-to-video retrieval | A caption and every clip are embedded in one shared space; the clips with the highest cosine are returned | session 208 |
| Video fingerprinting and near-duplicate detection | Each frame or clip gets a descriptor vector; two videos match when many descriptors have cosine above a threshold. Centering (as in Figure 6) helps the threshold work. | session 209 |
| Action recognition and key-moment detection | Clip embeddings are compared with text embeddings of action names (“a person jumps”) for zero-shot labels and moment search | session 210 |
| Recommendation | Item IDs (videos, products) are looked up in an embedding table exactly like tokens; items watched by the same users get similar vectors, the word2vec idea applied to watch histories | session 151 |
| Adding new tokens in fine-tuning | New rows of \(E\) for special tokens or a new language; initialized from the mean row and trained | session 57 |
Pseudo-code
- input: token IDs \(\mathrm{id}_1, \dots, \mathrm{id}_T\), table \(E \in \R^{|V| \times d}\)
- forward: for \(t = 1\) to \(T\) do \(X_{t,:} \leftarrow E_{\mathrm{id}_t,:}\) // a copy, no FLOPs, formula (11.3)
- backward: receive \(G = \partial \mathcal{L}/\partial X \in \R^{T \times d}\)
- \(\nabla E \leftarrow\) zeros\((|V|, d)\)
- for \(t = 1\) to \(T\) do
- \(\nabla E_{\mathrm{id}_t,:} \leftarrow \nabla E_{\mathrm{id}_t,:} + G_{t,:}\) // scatter-add: repeated IDs sum, formula (11.4)
- return \(\nabla E\) // rows of unseen tokens stay exactly 0
- input: corpus, window \(m\), negatives \(k\), learning rate \(\eta\), tables \(\mathbf{v}, \mathbf{u}\)
- build \(P_n(w) \propto \mathrm{count}(w)^{3/4}\) // formula (11.8)
- for each position with center word \(c\) do
- for each context word \(o\) within \(m\) positions do
- draw \(n_1, \dots, n_k \sim P_n\)
- \(\mathbf{g} \leftarrow \mathbf{0}\)
- for \((w, y)\) in \(\{(o, 1), (n_1, 0), \dots, (n_k, 0)\}\) do
- \(e \leftarrow \sigma(\mathbf{u}_w \cdot \mathbf{v}_c) - y\) // prediction minus target
- \(\mathbf{g} \leftarrow \mathbf{g} + e\,\mathbf{u}_w\); \(\ \mathbf{u}_w \leftarrow \mathbf{u}_w - \eta\, e\, \mathbf{v}_c\)
- \(\mathbf{v}_c \leftarrow \mathbf{v}_c - \eta\, \mathbf{g}\) // formula (11.7)
- return \(\mathbf{v}\) // the word embeddings
- input: embeddings \(\mathbf{v}_w\), words \(a, b, c\), optional mean vector \(\mathbf{m}\)
- (optional) \(\mathbf{v}_w \leftarrow \mathbf{v}_w - \mathbf{m}\) for all \(w\) // centering fixes anisotropy, formula (11.12)
- \(\hat{\mathbf{v}}_w \leftarrow \mathbf{v}_w / \lVert \mathbf{v}_w \rVert\) for all \(w\) // now dot product = cosine
- \(\mathbf{q} \leftarrow \hat{\mathbf{v}}_a - \hat{\mathbf{v}}_b + \hat{\mathbf{v}}_c\)
- \(s_w \leftarrow \hat{\mathbf{v}}_w \cdot \mathbf{q} / \lVert \mathbf{q} \rVert\) for all \(w \notin \{a, b, c\}\)
- return \(\argmax_w s_w\) // formula (11.11)
The same ideas in PyTorch. Each block checks one claim from this session:
import torch import torch.nn as nn import torch.nn.functional as F torch.manual_seed(0) # 1) lookup == one-hot @ E, formulas (11.2) and (11.3) V, d = 6, 4 emb = nn.Embedding(V, d) ids = torch.tensor([0, 1, 2, 3, 0, 4]) # "the cat sat on the mat" onehot = F.one_hot(ids, V).float() # (6, 6) print(torch.allclose(emb(ids), onehot @ emb.weight)) # True # 2) only used rows get a gradient, repeated IDs add up, formula (11.4) G = torch.randn(len(ids), d) # pretend upstream gradient (emb(ids) * G).sum().backward() print(emb.weight.grad[5]) # all zeros: "." is not in the batch print(torch.allclose(emb.weight.grad[0], G[0] + G[4])) # True # 3) the SGNS loss of formula (11.6) on the worked example def sgns_loss(v_c, u_o, u_neg): # u_neg: (k, d) pos = F.logsigmoid(u_o @ v_c) neg = F.logsigmoid(-(u_neg @ v_c)).sum() return -(pos + neg) v_c = torch.tensor([0.5, -0.2], requires_grad=True) loss = sgns_loss(v_c, torch.tensor([0.4, 0.1]), torch.tensor([[-0.3, 0.6]])) loss.backward() print(round(loss.item(), 4), v_c.grad) # 1.1744, tensor([-0.3119, 0.2142]) # 4) a tiny skip-gram (full softmax, formula 11.5) on the toy corpus of Figure 3 S = ["the cat eats fish", "the dog eats meat", "the cat sleeps at home", "the dog sleeps at home", "i feed the cat", "i feed the dog", "the car needs fuel", "the bus needs fuel", "the car drives on the road", "the bus drives on the road", "i park the car", "i park the bus"] words = sorted({w for s in S for w in s.split()}) ix = {w: i for i, w in enumerate(words)} pairs = [(ix[t[i]], ix[t[j]]) for t in (s.split() for s in S) for i in range(len(t)) for j in range(max(0, i - 2), min(len(t), i + 3)) if j != i] c, o = torch.tensor(pairs).T # 144 pairs v_in, u_out = nn.Embedding(len(words), 8), nn.Embedding(len(words), 8) with torch.no_grad(): u_out.weight.mul_(0.1) opt = torch.optim.Adam(list(v_in.parameters()) + list(u_out.parameters()), lr=0.05) for step in range(500): logits = v_in(c) @ u_out.weight.T # (144, |V|): u_w . v_c for all w loss = F.cross_entropy(logits, o) opt.zero_grad(); loss.backward(); opt.step() E = F.normalize(v_in.weight.detach(), dim=1) # unit rows: dot product = cosine cs = lambda a, b: round((E[ix[a]] @ E[ix[b]]).item(), 3) print("cat-dog", cs("cat", "dog"), "car-bus", cs("car", "bus"), "cat-car", cs("cat", "car")) # cat-dog and car-bus come out clearly higher than cat-car; exact values depend on the seed # 5) parameter count |V| x d without allocating memory, formula (11.13) big = nn.Embedding(128256, 4096, device="meta") # Llama 3 vocabulary and width print(big.weight.numel()) # 525336576 # 6) weight tying, formula (11.14): the LM head reuses the same tensor tok_emb = nn.Embedding(V, d) lm_head = nn.Linear(d, V, bias=False) lm_head.weight = tok_emb.weight # both (V, d): one tensor, two uses h = torch.randn(d) print(torch.allclose(lm_head(h), tok_emb.weight @ h)) # True
Common mistakes
- Feeding token IDs as numbers. A linear layer on raw IDs treats them as amounts and makes neighbors in ID order look similar. Always go through an embedding table (or one-hot).
- Changing the tokenizer without changing the table. Adding tokens without
resize_token_embeddings(or the equivalent) gives index-out-of-range errors, or worse, silently reuses the wrong rows if the table was larger. And a tied output layer must be resized together with the input table. - Random init for new tokens. New rows drawn from \(\mathcal{N}(0, 1)\) can be far larger than trained rows (which may have a much smaller scale), and the model’s first steps go wild. Start new rows at the mean of the existing rows, or at a similar scale.
- Using the dot product when the model was trained with cosine (or the reverse). Embedding lengths often reflect frequency. Match the similarity used in training, and normalize if in doubt.
- Reading raw cosine values as absolute meaning. “0.8 means similar” is not true in an anisotropic space where random pairs already score 0.8 (Figure 6). Compare against a baseline of random pairs, or center first.
- Trusting analogy demos. The classic results exclude the input words (11.11). Without that rule, the nearest word is often one of the inputs (Figure 5). Analogies are a probe, not a proof that the space is linear.
- Forgetting the output side when counting. An untied model has two \(|V| \times d\) matrices. For Llama 3 8B that is 1.05 B parameters, not 525 M.
- Weight decay on the embedding table without thinking. With AdamW, rows of rare tokens shrink at every step but are seldom pushed back by a gradient. Many training codes exclude embeddings (and norms) from weight decay. [Likely]
Interview questions
What is an embedding layer, and why is it the same as multiplying a one-hot vector by a matrix?basic
Why not feed token IDs as numbers, or use one-hot vectors directly as the model input?basic
Which rows of the embedding table get a gradient in one step, and does that mean only those rows change?core
sparse=True with SparseAdam) update only the touched rows. With tied weights, every row also gets a gradient through the output softmax.Explain skip-gram with negative sampling. Why negative sampling instead of the full softmax?core
Why are negatives sampled from \(\mathrm{count}^{3/4}\)?core
Why do vector analogies like \(\text{king} - \text{man} + \text{woman} \approx \text{queen}\) work, and when do they fail?core
All cosine similarities in your embedding model are between 0.6 and 0.9. What is going on and what do you do?advanced
How many parameters are in Llama 3 8B’s embeddings? Why is the share larger in small models?core
What is weight tying? What are its benefits and costs?core
How does an image become “tokens” for an LLM if there is no vocabulary?core
You add 10 new special tokens before fine-tuning. How do you set up their embeddings?advanced
What is the difference between a static word embedding and a contextual embedding?basic
Exercises
- A model has \(|V| = 32{,}000\) and \(d = 4{,}096\). How many embedding parameters does it have with tied and with untied embeddings? How many megabytes is one table in bfloat16?
- A batch contains the token IDs \((3, 1, 3, 3)\) and upstream gradients \(\mathbf{g}_1, \dots, \mathbf{g}_4\). Which rows of \(E\) get a non-zero gradient, and what is the gradient of row 3?
- Compute the cosine between \(\mathbf{a} = (1, 2, 2)\) and \(\mathbf{b} = (2, 1, 2)\).
- In SGNS, a real pair has score \(\mathbf{u}_o \cdot \mathbf{v}_c = 0\). What is its loss term \(-\log \sigma(0)\), and what is the factor \(\sigma(0) - 1\) in the gradient (11.7)?
- In Figure 6, what mean cosine do you expect at \(\alpha = 1\) and at \(\alpha = 3\)? What happens after centering?
- A 2B-parameter model uses \(|V| = 256{,}000\) and \(d = 2{,}048\) with tied embeddings. How many embedding parameters is that, and roughly what share of the model?
- Why does the one-hot product for Llama 3 8B cost about 1.05 GFLOPs per token? Compare it with the total forward cost of about \(2N\).
Answerscheck
Key takeaways
- An embedding layer is a learned table \(E \in \R^{|V| \times d}\). Looking up row \(i\) equals multiplying the one-hot vector \(\mathbf{e}_i\) by \(E\) (11.2), so the table trains with ordinary backpropagation.
- The gradient of a row is the sum of the gradients at the positions where that token appears (11.4); tokens not in the batch get zero, so rare tokens learn slowly.
- word2vec makes words with similar contexts get similar vectors: skip-gram (11.5), made cheap by negative sampling (11.6–11.8). At the optimum it factorizes a shifted PMI table (11.9).
- Cosine similarity (11.10) gives nearest neighbors and analogies (11.11), but it mixes antonyms with synonyms, gives one vector to words with many meanings, copies bias, and can be inflated by a shared direction (11.12). Center before trusting raw values.
- Embeddings cost \(|V| \cdot d\) parameters per table (11.13): 525 M per table for Llama 3 8B, over a quarter of the weights in small models. Weight tying (11.14) reuses the input table as the output layer, so every row also gets an output-side gradient (11.15).
- Images and video enter the same space through patch embeddings and a projector, and CLIP uses the same “pull real pairs together” idea as negative sampling.
Key formulas
References for this session
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 5: Embeddings
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 3: Words and Word Vectors
- Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space
- Mikolov, Sutskever, Chen, Corrado, Dean (2013). Distributed Representations of Words and Phrases and their Compositionality
- Pennington, Socher, Manning (2014). GloVe: Global Vectors for Word Representation
- Levy & Goldberg (2014). Neural Word Embedding as Implicit Matrix Factorization
- Bojanowski, Grave, Joulin, Mikolov (2017). Enriching Word Vectors with Subword Information
- Press & Wolf (2017). Using the Output Embedding to Improve Language Models
- Vaswani et al. (2017). Attention Is All You Need
- Bolukbasi, Chang, Zou, Saligrama, Kalai (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
- Mu & Viswanath (2018). All-but-the-Top: Simple and Effective Postprocessing for Word Representations
- Ethayarajh (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
- Gao, He, Tan, Qin, Wang, Liu (2019). Representation Degeneration Problem in Training Natural Language Generation Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale