LLM & VLM courseBlog
11Track 1: Foundations · session 11 of 15

Embeddings: turning token IDs into vectors

A tokenizer gives the model a list of integers. The first layer of every LLM turns each integer into a vector by picking one row of a big learned table. This session shows why that lookup is the same as multiplying a one-hot vector by a matrix, how training makes words that appear in similar contexts get similar vectors (word2vec), what cosine similarity can and cannot tell you, and how much memory the table costs in real models.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 2, 4 and 8 (vectors and softmax, gradients, tokens)

What you will learn

  • Why a token ID cannot be fed to a network as a number, and why one-hot vectors are not enough
  • The embedding layer: one-hot times a matrix equals a table lookup, in the forward and in the backward pass
  • The distributional idea behind word2vec: skip-gram, negative sampling, and its link to PMI
  • Cosine similarity, nearest neighbors and analogies, and where this geometry fails (polysemy, antonyms, bias, anisotropy)
  • Embedding size and parameter count, |V| × d, with real numbers for GPT-2, Llama and Qwen
  • Weight tying between the input embedding and the output layer
  • How images and video frames enter the same vector space in a VLM

Why this matters

A tokenizer turns text into a list of integers. “The cat sat” might become [791, 8415, 7731]. A neural network cannot do anything useful with these integers directly. The very first layer of every LLM has one job: turn each integer into a vector of real numbers that the rest of the network can compute with.

That layer is called the embedding layer, and the vectors it produces are called embeddings. An embedding is a list of \(d\) numbers that stands for one token. The layer is very simple, a big table with one row per token, but it carries several ideas that come back again and again in this course:

  • Lookup is matrix multiplication. Picking row \(i\) of a table is the same as multiplying a one-hot vector by a matrix. This is why the table can be trained with backpropagation like any other layer.
  • Meaning comes from context. Training pushes tokens that appear in similar contexts toward similar vectors. This idea, made famous by word2vec, is the root of semantic search, retrieval and CLIP.
  • Geometry has limits. Cosine similarity between embeddings is useful, but it mixes up opposites, ignores the many meanings of one word, and copies social bias from the data.
  • The table is big. It has \(|V| \times d\) numbers. In Llama 3 8B the input table alone has 525 million parameters. In small models the embedding tables can be more than a quarter of all weights.

Interviewers like this topic because it looks easy and hides real questions: “Why is an embedding layer the same as a linear layer?”, “Which rows get a gradient?”, “How many parameters are in the embeddings of Llama 3 8B?”, “What is weight tying?”, “Why are all cosine similarities high in some models?”. By the end of this session you can answer each one with a formula and a number. In Session 10 we built the tokenizer that produces the IDs; here we pick up those IDs and give them meaning.

From an ID to a vector

Why not use the ID itself?

The simplest idea is to feed the integer as one input number. This fails for a clear reason: the network would treat the IDs as amounts. Token 8415 would be “twice as much” as token 4207, and tokens with close IDs would look alike. But IDs are given by the tokenizer in the order it merged pieces. Their size means nothing. A linear layer on a single number can only scale it, so the model would see a meaningless order.

One-hot vectors

The fix is to give each token its own direction. A one-hot vector for token \(i\) has length \(|V|\) (the vocabulary size), a 1 in position \(i\), and 0 everywhere else:

\[ \mathbf{e}_i = (0, \dots, 0, \underbrace{1}_{\text{position } i}, 0, \dots, 0) \in \R^{|V|} \tag{11.1} \]

Now no token is “bigger” than another. But one-hot vectors have two problems. First, they are huge: with \(|V| = 128{,}256\) (Llama 3) each token would be a vector of 128,256 numbers, almost all zero. Second, every pair of different tokens is perpendicular: \(\mathbf{e}_i \cdot \mathbf{e}_j = 0\) for \(i \ne j\). So “cat” is exactly as far from “dog” as it is from “carburetor”. One-hot vectors carry no notion of similarity at all.

The embedding matrix

We fix both problems by multiplying the one-hot vector by a learned matrix \(E\) with \(|V|\) rows and \(d\) columns, where \(d\) is much smaller than \(|V|\) (for example \(d = 4096\)). Write the one-hot vector as a row vector. Then:

\[ \mathbf{x} = \mathbf{e}_i^\top E = E_{i,:} \in \R^{d}, \qquad E \in \R^{|V| \times d} \tag{11.2} \]
Read the symbols
\(|V|\)
the vocabulary size: how many different tokens exist
\(d\)
the embedding size (also called the model width or hidden size)
\(E\)
the embedding matrix: one row of \(d\) numbers for each token. Its numbers are learned during training.
\(\mathbf{e}_i^\top E\)
a one-hot row vector times the matrix. All rows of \(E\) are multiplied by 0 except row \(i\), which is multiplied by 1.
\(E_{i,:}\)
row \(i\) of \(E\): the embedding of token \(i\)

So multiplying by a one-hot vector just selects one row. That is why we call the layer a lookup table: in code, we never build the one-hot vector; we read row \(i\) directly. For a sequence of \(T\) tokens, stack the one-hot rows into a matrix \(O \in \R^{T \times |V|}\):

\[ X = O\,E \in \R^{T \times d}, \qquad X_{t,:} = E_{\mathrm{id}_t,:} \tag{11.3} \]

Each row of \(X\) is the embedding of one token in the sequence. If the same token appears twice, the same row is copied twice.

Worked example: a 6-token vocabulary

Let the vocabulary be the (ID 0), cat (1), sat (2), on (3), mat (4), . (5), with \(d = 4\). Row 1 of \(E\) is \((0.8, 0.5, -0.2, 0.1)\). The one-hot vector for cat is \((0, 1, 0, 0, 0, 0)\). The product is \(0 \cdot E_{0,:} + 1 \cdot E_{1,:} + 0 \cdot E_{2,:} + \dots = (0.8, 0.5, -0.2, 0.1)\): exactly row 1.

Cost: the full product needs \(|V| \cdot d = 24\) multiplications and about as many additions, all but 4 of them with a zero. The lookup needs no arithmetic at all, only a copy of 4 numbers. For Llama 3 8B the one-hot product would cost \(2 \times 128{,}256 \times 4096 \approx 1.05\) billion FLOPs per token, about 6.5% of the whole forward pass (about \(2 \times 8.03\) billion FLOPs). The lookup costs nothing but one memory read of 4,096 numbers.

Algorithm: one-hot product, then lookup
    Figure 1. One-hot times matrix equals lookup, computed one row at a time. Pick a token of the sentence “the cat sat on the mat .” and press Play. The loop walks over the 6 rows of \(E\): the row in use is outlined in blue, and it is added to \(\mathbf{x}\) with the weight \(e_j\) from the one-hot vector. Only the token’s own row (green) has weight 1. The last steps read the same row directly, with no arithmetic.
    How to read Figure 1

    Step through the loop. The right panel writes each line of the algorithm with the real numbers: for “cat” (ID 1), step \(j = 0\) adds \(0 \cdot (0.1, -0.3, 0.2, 0.0)\), step \(j = 1\) adds \(1 \cdot (0.8, 0.5, -0.2, 0.1)\), and the four other steps add zeros. After 6 rows, \(\mathbf{x} = (0.8, 0.5, -0.2, 0.1)\), exactly row 1. The counter shows 24 multiply-adds, of which only 4 were not with a zero; the lookup gives the same vector for free. Click a token at any moment: the loop is recomputed for that token. Click the two copies of “the”: they give the same vector, because the embedding layer does not know where a token is in the sentence. Position is added later (session 13).

    Training the table

    At the start, \(E\) is random. Every row is learned with backpropagation, together with the rest of the network. To see how, use formula (11.3) and the chain rule from session 4. Let \(\mathbf{g}_t = \partial \mathcal{L} / \partial X_{t,:}\) be the gradient that flows back into the embedding of position \(t\). Then the gradient for row \(j\) of the table is:

    \[ \frac{\partial \mathcal{L}}{\partial E_{j,:}} = \sum_{t\,:\,\mathrm{id}_t = j} \mathbf{g}_t \tag{11.4} \]
    Read the symbols
    \(\mathcal{L}\)
    the training loss (for an LLM, the next-token cross-entropy)
    \(\mathbf{g}_t\)
    the gradient of the loss with respect to the embedding at position \(t\), a vector of \(d\) numbers
    \(t : \mathrm{id}_t = j\)
    all positions in the batch where token \(j\) appears

    Two facts follow, and both matter in practice:

    1. Only rows of tokens in the batch get a gradient. All other rows get exactly zero. A batch of a few thousand tokens touches only a small part of a 128,256-row table.
    2. Repeated tokens add up. If “the” appears five times, its row receives the sum of five gradients. Frequent tokens are updated often and strongly; rare tokens are updated rarely.

    In matrix form, \(\partial \mathcal{L}/\partial E = O^\top G\), where \(G\) stacks the \(\mathbf{g}_t\). This is the usual gradient of a linear layer (input transposed times output gradient), with the one-hot matrix as input. PyTorch computes it with a scatter-add, never with the dense product.

    Worked example: gradient of a repeated token

    The sentence “the cat sat on the mat” has IDs \((0, 1, 2, 3, 0, 4)\). Token 0 appears at positions 1 and 5. Suppose the gradients arriving there are \(\mathbf{g}_1 = (0.2, -0.1, 0, 0.3)\) and \(\mathbf{g}_5 = (-0.5, 0.4, 0.1, 0.1)\). Then the gradient of row 0 is \(\mathbf{g}_1 + \mathbf{g}_5 = (-0.3, 0.3, 0.1, 0.4)\). Row 5 (the token “.”) is not in the sentence, so its gradient is \((0, 0, 0, 0)\).

    Figure 2 runs the backward pass of the embedding layer on this sentence, one position at a time, and then takes one gradient step. Watch which rows of the table change.

    Algorithm: backward pass and one SGD step
      Figure 2. The backward pass of an embedding layer, live. Left: the upstream gradients \(\mathbf{g}_t\) for the 6 positions of “the cat sat on the mat” (\(\mathbf{g}_1\) and \(\mathbf{g}_5\) are the ones of the worked example; the others are fixed made-up values). Middle: \(\nabla E\), filled by scatter-add (formula 11.4); orange cells are positive, blue cells negative, and the color strength follows the size. Right: the table \(E\) of Figure 1, then \(E - \eta \nabla E\) after the step; green outlines mark the cells that changed.
      How to read Figure 2

      Each step adds one row of \(G\) into the row of \(\nabla E\) that belongs to that token. At \(t = 5\) the second “the” lands on row 0 again, so row 0 becomes \(\mathbf{g}_1 + \mathbf{g}_5 = (-0.3, 0.3, 0.1, 0.4)\). Row 5 (“.”) never receives anything, and after the SGD step it is the only row of \(E\) that did not change. Now tick Tied output layer. The same table is also used as the output layer (formula 11.14) to predict the token after “mat”, which is “.”. Two extra steps compute the logits \(z_j = E_{j,:} \cdot \mathbf{h}\), the softmax \(p\), and the output-side gradient \((p_j - y_j)\,\mathbf{h}\) of formula (11.15) for every row. With the fixed hidden vector \(\mathbf{h} = (0.5, -0.3, -0.4, 0.8)\), \(p_{\text{.}} = 0.305\), and row 5 now gets the gradient \((-0.348, 0.209, 0.278, -0.556)\): with \(\eta = 0.5\) it moves from \((-0.1, 0, -0.6, 0.9)\) to \((0.074, -0.104, -0.739, 1.178)\). With tying, all 6 rows change.

      Practical details

      • Initialization. PyTorch’s nn.Embedding starts every entry from a standard normal distribution, \(\mathcal{N}(0, 1)\). GPT-style code such as nanoGPT uses a much smaller standard deviation, 0.02. [Certain] The original transformer multiplies embeddings by \(\sqrt{d}\) before adding position signals. [Certain]
      • The optimizer still moves unused rows. With Adam, a row whose gradient is zero this step still moves if its momentum from earlier steps is not zero, and AdamW’s weight decay shrinks every row at every step. A dense optimizer updates the whole table every step. [Certain]
      • Rare tokens stay close to their random start. A token that almost never appears in the training text almost never receives a gradient. When such a token shows up at test time, the model may behave strangely; people call these “glitch tokens”. [Likely]
      • Adding new tokens. When you add special tokens for a chat template (session 9), you add rows to \(E\). A common choice is to start them at the mean of the existing rows instead of random noise, so they do not look like extreme outliers to the first layer. [Likely]

      Where meaning comes from: word2vec

      Training an LLM to predict the next token gives its embedding table structure for free. But the cleanest way to see why embeddings become meaningful is an older and smaller model: word2vec (Mikolov et al., 2013). It trains only embeddings, with no deep network at all.

      The idea behind it is the distributional hypothesis: words that appear in similar contexts have similar meanings. You never see a definition of “dog”, but you see “the dog eats”, “I feed the dog”, “the dog sleeps”. “Cat” appears in the same kind of sentences. If a model must use a word’s vector to predict its neighbors, then “cat” and “dog” need similar vectors, because they have similar neighbors.

      Skip-gram: predict the context from the word

      Slide a window over the text. For each center word \(c\), every word within \(m\) positions of it is a context word \(o\). With \(m = 2\), the sentence “the cat eats fish” gives pairs like (cat, the), (cat, eats), (cat, fish). Skip-gram learns two tables: an input vector \(\mathbf{v}_w\) for each word used as a center, and an output vector \(\mathbf{u}_w\) for each word used as a context. The probability of seeing \(o\) near \(c\) is a softmax over the whole vocabulary:

      \[ p(o \mid c) = \frac{\exp(\mathbf{u}_o \cdot \mathbf{v}_c)}{\sum_{w \in V} \exp(\mathbf{u}_w \cdot \mathbf{v}_c)} \tag{11.5} \]

      A large dot product means “these two words often appear together”. Training maximizes \(\log p(o \mid c)\) over all pairs. The problem is the sum in the denominator: it runs over the whole vocabulary (word2vec’s released Google News vectors cover 3 million words and phrases [Certain]), for every pair.

      Negative sampling

      The fix is negative sampling (SGNS: skip-gram with negative sampling). Instead of a softmax over all words, we solve a small yes-or-no problem: “Is \(o\) a real neighbor of \(c\)?” For each real pair we also draw \(k\) random negative words \(n_1, \dots, n_k\) that probably are not neighbors. The loss pushes the real pair’s score up and the fake pairs’ scores down:

      \[ \mathcal{L}(c, o) = -\log \sigma(\mathbf{u}_o \cdot \mathbf{v}_c) \;-\; \sum_{i=1}^{k} \log \sigma(-\mathbf{u}_{n_i} \cdot \mathbf{v}_c) \tag{11.6} \]
      Read the symbols
      \(\sigma(s)\)
      the sigmoid \(1/(1 + e^{-s})\): turns a score into a probability between 0 and 1
      \(\mathbf{v}_c\)
      the input (center) vector of word \(c\). After training, these are the word embeddings people use.
      \(\mathbf{u}_o, \mathbf{u}_{n_i}\)
      output (context) vectors of the real neighbor and of the negative words
      \(k\)
      the number of negatives per real pair. The paper suggests 5 to 20 for small datasets and 2 to 5 for large ones. [Certain]

      Now each pair costs \(k + 1\) dot products instead of \(|V|\). The gradient with respect to the center vector has the same “prediction minus target” shape as softmax cross-entropy in session 2:

      \[ \frac{\partial \mathcal{L}}{\partial \mathbf{v}_c} = \big(\sigma(\mathbf{u}_o \cdot \mathbf{v}_c) - 1\big)\,\mathbf{u}_o + \sum_{i=1}^{k} \sigma(\mathbf{u}_{n_i} \cdot \mathbf{v}_c)\, \mathbf{u}_{n_i} \tag{11.7} \]

      Read it like this: the center vector moves toward its real neighbor (the first term has a negative factor, and we step against the gradient) and away from the negatives. The step is large when the model is wrong and small when it is already right.

      Negatives are not drawn uniformly. They come from the word frequencies raised to the power \(3/4\):

      \[ P_n(w) = \frac{\mathrm{count}(w)^{3/4}}{\sum_{w'} \mathrm{count}(w')^{3/4}} \tag{11.8} \]

      The power \(3/4\) flattens the distribution a little: very common words like “the” are drawn less often than their raw frequency, and rare words more often. With counts 100, 10 and 1, the raw probabilities are 0.901, 0.090 and 0.009; after the power they become 0.827, 0.147 and 0.026. The paper found \(3/4\) worked better than both the raw counts and the uniform distribution. [Certain] word2vec also randomly drops very frequent words from the training text (subsampling), so that “the” does not dominate the pairs.

      Worked example: one SGNS step by hand

      Take \(d = 2\), center vector \(\mathbf{v}_c = (0.5, -0.2)\), real neighbor \(\mathbf{u}_o = (0.4, 0.1)\), one negative \(\mathbf{u}_n = (-0.3, 0.6)\).

      Scores: \(\mathbf{u}_o \cdot \mathbf{v}_c = 0.20 - 0.02 = 0.18\), so \(\sigma(0.18) = 0.545\). \(\mathbf{u}_n \cdot \mathbf{v}_c = -0.15 - 0.12 = -0.27\), so \(\sigma(0.27) = 0.567\). Loss (11.6): \(-\log 0.545 - \log 0.567 = 1.174\).

      Gradient (11.7): \((0.545 - 1)(0.4, 0.1) + \sigma(-0.27)\,(-0.3, 0.6) = (-0.312, 0.214)\). One step with learning rate 0.5 gives \(\mathbf{v}_c = (0.656, -0.307)\). The new scores are \(0.232\) (real, up) and \(-0.381\) (negative, down), and the loss drops to \(1.105\).

      Figure 3 runs exactly this algorithm on a toy corpus of 12 short sentences: half about a cat and a dog, half about a car and a bus. The embeddings have only \(d = 2\) numbers, so we can draw them.

      Figure 3. Skip-gram with negative sampling, trained live in your browser. Corpus: 12 sentences such as “the cat eats fish”, “i feed the dog”, “the car needs fuel”, “i park the bus”; 19 words, 144 (center, context) pairs with a window of 2, \(k = 3\) negatives, learning rate 0.1, fixed random seed. Blue: animal words. Orange: vehicle words. Grey: all other words. Each point is an input vector \(\mathbf{v}_w\) (hover a point to see its word); the arrows and labels show the four nouns.
      How to read Figure 3

      At 0 epochs the vectors are random: “cat” and “dog” point in almost opposite directions (cosine about −0.98). Move the slider or press Play. In the first few epochs all words move together, because they all share the very common context “the”. Then the two topics separate: “cat” and “dog” end up pointing the same way, “car” and “bus” too, while “cat” and “car” become nearly perpendicular. No one told the model which words are animals. It found this only from which words share neighbors. The loss in the readout goes down and then shakes, because the negatives are random at every step.

      CBOW, GloVe and fastText

      word2vec has a second variant, CBOW (continuous bag of words), which predicts the center word from the average of its context vectors. Two later methods are worth knowing by name:

      • GloVe (Pennington et al., 2014) first counts how often each pair of words appears together in the whole corpus, then fits vectors so that their dot product matches the log of that count.
      • fastText (Bojanowski et al., 2017) builds a word’s vector as the sum of vectors of its character n-grams (pieces like “<ca”, “cat”, “at>”). It can make a vector for a word it never saw. This is the same goal that subword tokenizers serve in LLMs (session 8).

      What SGNS really computes

      Levy and Goldberg (2014) showed a neat link between the counting view and the prediction view. At the best solution, the SGNS score of a word–context pair equals the pointwise mutual information (PMI) of the pair, shifted by \(\log k\):

      \[ \mathbf{u}_o \cdot \mathbf{v}_c \approx \operatorname{PMI}(c, o) - \log k, \qquad \operatorname{PMI}(c, o) = \log \frac{p(c, o)}{p(c)\, p(o)} \tag{11.9} \]

      PMI measures how much more often two words appear together than they would by chance. So SGNS is secretly a low-rank factorization of a (shifted) PMI table: it squeezes a \(|V| \times |V|\) table of co-occurrence statistics into two thin \(|V| \times d\) matrices. [Certain] This tells you what embeddings can and cannot hold: they hold co-occurrence statistics, not definitions.

      Cosine similarity and the geometry of meaning

      Once we have embeddings, we compare them with cosine similarity, the cosine of the angle between two vectors:

      \[ \cos(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert} \in [-1, 1] \tag{11.10} \]

      It is 1 for the same direction, 0 for perpendicular vectors and −1 for opposite ones. We use the angle and not the plain dot product because embedding lengths often track word frequency, not meaning: frequent words tend to have different lengths from rare ones. Dividing by the lengths removes that effect. For vectors of length 1 the cosine and the dot product are the same number.

      Finding the nearest neighbors of a word is a loop over the table: score every row against the query, keep the \(k\) best. Figure 4 runs this loop on 11 hand-made 2-dimensional word vectors, so that each cosine is simply the cosine of the angle you see. The vectors of the frequent words “the” and “and” are long, like real frequent tokens often are.

      Algorithm: top-k search over the rows of E
        Figure 4. Nearest-neighbor search, step by step. Blue arrow: the query. Orange arrow: the word being scored now, with its angle to the query (cosine) or its shadow on the query direction (dot product). Green: the current top \(k\). The table lists every score in the order the loop computes it. “Tied LM head” uses a free hidden vector \(\mathbf{h}\) as the query and the dot product as the score, as the output layer of formula (11.14) does, and then turns the scores into probabilities with softmax.
        How to read Figure 4

        With Cosine and query “cat”, the loop finds dog (0.990), kitten (0.978) and puppy (0.951): the words with the smallest angle. Switch to Dot product and run again: now “the” wins with 1.472, and “and” enters the top 3, only because their arrows are long. This is why we divide by the lengths. With Tied LM head the query is a hidden vector \(\mathbf{h}\) (angle 35°, length 2). The scores are logits, and the frequent words win again: \(p(\text{the}) = 0.299\), \(p(\text{dog}) = 0.180\), \(p(\text{cat}) = 0.155\). For an output layer that is useful: a long vector acts like a high prior for a frequent token. Make \(\lVert\mathbf{h}\rVert\) larger and the softmax gets sharper: at length 2.4, \(p(\text{the}) = 0.344\); at length 0.5, only 0.136. Turn \(\mathbf{h}\) to 150°: “and” still wins (0.221) just ahead of “bus” (0.197), because it is long. At 165° the vehicle words take the top 3.

        Two things people do with cosine:

        • Nearest neighbors. Rank all words by cosine to a query word. In good embeddings the top neighbors of “frog” are other frogs and toads, and the top neighbors of “Paris” are other capitals or French cities.
        • Analogies. The famous example: \(\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}}\) is close to \(\mathbf{v}_{\text{queen}}\). The idea is that the difference “man → woman” is roughly the same direction for many word pairs.
        \[ \hat{w} = \argmax_{w \notin \{a, b, c\}} \cos\!\big(\mathbf{v}_w,\ \mathbf{v}_a - \mathbf{v}_b + \mathbf{v}_c\big) \tag{11.11} \]
        Read the symbols
        \(a, b, c\)
        the three given words, as in “\(b\) is to \(a\) as \(c\) is to ?”. For \(\text{king} - \text{man} + \text{woman}\): \(a\) = king, \(b\) = man, \(c\) = woman.
        \(w \notin \{a, b, c\}\)
        the answer may not be one of the three input words. This small rule is important, as Figure 5 shows.
        \(\hat{w}\)
        the predicted answer: the word whose vector has the largest cosine with the target point
        Worked example: cosine by hand

        In the hand-made table of Figure 5 each word has 5 features (royal, male minus female, person, food, young). king \(= (0.9, 0.8, 0.9, 0, 0.1)\) and queen \(= (0.9, -0.8, 0.9, 0, 0.1)\). The dot product is \(0.81 - 0.64 + 0.81 + 0 + 0.01 = 0.99\). Both lengths are \(\sqrt{0.81 + 0.64 + 0.81 + 0.01} = \sqrt{2.27} = 1.507\). So \(\cos = 0.99 / 2.27 = 0.436\). The target point \(\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}}\) is \((0.9, -0.8, 0.9, 0, 0.1)\), which is exactly queen: cosine 1.

        −+
        Figure 5. Analogies by vector arithmetic, formula (11.11), on 10 hand-made 5-feature word vectors (royal, male minus female, person, food, young). The bars rank all candidate words by cosine with the target point \(\mathbf{v}_a - \mathbf{v}_b + \mathbf{v}_c\). The noise slider adds seeded Gaussian noise with standard deviation \(\sigma\) to every feature, to imitate real, imperfect embeddings.
        How to read Figure 5

        With no noise the vectors were built so that the analogy is perfect, and queen wins with cosine 1. Real embeddings are not like this. Raise the noise to 0.30: queen still wins when the input words are excluded, but uncheck the box and the top answer becomes “woman”, one of the inputs. This is a known weakness of analogy tests: the target point is often closest to one of the words you started from, and the famous results depend on the exclusion rule. Try \(\text{prince} - \text{boy} + \text{girl}\), or build your own.

        The limits of this geometry

        Cosine similarity between word embeddings is useful, but it is easy to read too much into it. Here are the main limits.

        • One vector per word, many meanings. “Bank” (river) and “bank” (money) get one static vector, a blend of both meanings weighted by how often each appears. This is the main reason we need contextual embeddings, where the vector depends on the sentence. That is the subject of the next session, session 12.
        • Similar is not the same as synonym. “Hot” and “cold” appear in the same contexts (“the water is ___”), so their cosine is high even though they mean opposite things. Distributional similarity measures “used in the same places”, which mixes synonyms, antonyms and related words. [Likely]
        • Frequency effects. Very frequent and very rare words behave differently: rare words get few updates and noisy vectors, and some words (“hubs”) appear among the nearest neighbors of very many other words. [Likely]
        • Social bias. Embeddings copy the statistics of their text, including stereotypes. Bolukbasi et al. (2016) found that word2vec vectors trained on news text complete “man is to computer programmer as woman is to ?” with “homemaker”. [Certain] Any system built on these vectors (search, ranking, hiring tools) can inherit such patterns.
        • Anisotropy: all vectors in a narrow cone. In many trained models the embeddings share a large common direction, so almost every pair has a high positive cosine. Then “0.8” does not mean “very similar”; it may be the average. This was reported for static word vectors (Mu and Viswanath, 2018), for the input and output embeddings of generation models (Gao et al., 2019), and for the hidden states of BERT and GPT-2 (Ethayarajh, 2019). [Certain]

        We can see the effect of a common direction with a short calculation. Let every vector be \(\mathbf{x} = \mathbf{z} + \mathbf{m}\), where \(\mathbf{z}\) is a random vector with \(d\) independent entries of variance 1 (so \(\lVert \mathbf{z} \rVert^2 \approx d\)) and \(\mathbf{m}\) is a fixed shared vector with \(\lVert \mathbf{m} \rVert^2 = \alpha^2 d\). Random parts of different vectors are nearly perpendicular in high dimension, so \(\mathbf{x} \cdot \mathbf{x}' \approx \alpha^2 d\) and \(\lVert \mathbf{x} \rVert^2 \approx (1 + \alpha^2)\, d\). Therefore:

        \[ \cos(\mathbf{x}, \mathbf{x}') \approx \frac{\alpha^2}{1 + \alpha^2} \tag{11.12} \]

        With \(\alpha = 1\) (the shared part as long as the random part), unrelated vectors already have cosine about 0.5. With \(\alpha = 2\), about 0.8. The simple fix is centering: subtract the mean vector of the collection before computing cosines. “All-but-the-top” goes one step further and also removes the few strongest shared directions. [Certain]

        Figure 6. Anisotropy. 40 random vectors in \(d = 64\) (seeded), each plus the same shared vector of length \(\alpha \sqrt{d}\). The histogram shows the cosine of all 780 pairs. The dashed line is the prediction of formula (11.12).
        How to read Figure 6

        At \(\alpha = 0\) the cosines are spread around 0, as expected for unrelated random vectors. Increase \(\alpha\): the whole histogram slides to the right and becomes narrow, and its mean follows the dashed line \(\alpha^2/(1 + \alpha^2)\). The vectors have not become “more similar” in any useful sense; they all just share one direction. Now tick centering: the histogram jumps back to 0 for every \(\alpha\), because subtracting the mean removes the shared vector (for 40 vectors the mean pair cosine after centering is about \(-1/39 = -0.026\)). When you see high cosines everywhere in a real model, check this first.

        Embedding size and parameter count

        The embedding matrix has one row of \(d\) numbers per token, so its size is simple:

        \[ N_{\text{emb}} = |V| \cdot d \quad \text{(one table)}, \qquad N_{\text{emb}} = 2\,|V| \cdot d \quad \text{(separate input and output tables)} \tag{11.13} \]

        Why two tables? At the end of an LLM, the last hidden vector \(\mathbf{h} \in \R^d\) must become one score (logit) per token. That is done by another \(|V| \times d\) matrix, the output embedding or “LM head”. Many models keep it separate; others reuse the input table, which is called weight tying:

        \[ \mathbf{z} = E\, \mathbf{h} \in \R^{|V|}, \qquad z_j = E_{j,:} \cdot \mathbf{h} \tag{11.14} \]
        Read the symbols
        \(\mathbf{h}\)
        the final hidden vector at one position, after all transformer blocks
        \(E\)
        the same matrix as in formula (11.2), used a second time
        \(z_j\)
        the logit of token \(j\): the dot product of token \(j\)’s embedding with \(\mathbf{h}\). Softmax over \(\mathbf{z}\) gives the next-token probabilities.

        Tying makes a nice symmetry: a token’s input vector is also the direction the model must point to in order to predict that token. Press and Wolf (2017) showed that tying lowers perplexity of language models while removing a whole matrix of parameters. [Certain] It also means every row of \(E\) gets a gradient at every step through the output side, because every logit is in the softmax. With cross-entropy, the output-side gradient of row \(j\) for one position is:

        \[ \frac{\partial \mathcal{L}}{\partial E_{j,:}}\bigg|_{\text{output}} = (p_j - y_j)\,\mathbf{h}, \qquad \mathbf{p} = \softmax(E\,\mathbf{h}) \tag{11.15} \]
        Read the symbols
        \(p_j\)
        the predicted probability of token \(j\) as the next token
        \(y_j\)
        1 if token \(j\) is the true next token, else 0
        \(\mathbf{h}\)
        the final hidden vector at that position, as in (11.14)

        The true token’s row moves toward \(\mathbf{h}\) (its factor \(p_j - 1\) is negative), and every other row moves a little away from \(\mathbf{h}\), in proportion to its probability. So even a token that never appears in the batch gets a small gradient, which helps rare tokens. Tick “Tied output layer” in Figure 2 to see this: row 5 (“.”) is not in the input sentence, but it gets the largest gradient of all, because “.” is the true next token. And the “Tied LM head” mode of Figure 4 shows the forward side: the logits are one dot-product search over the rows of \(E\).

        The cost: one matrix must serve two roles, which large models with plenty of parameters often prefer not to do.

        Model\(|V|\)\(d\)Tied?Embedding parametersShare of all parameters
        GPT-2 small50,257768yes38.6 M31.0% of 124.4 M
        Llama 2 7B32,0004,096no2 × 131.1 M = 262.1 M3.9% of 6.74 B
        Llama 3 8B128,2564,096no2 × 525.3 M = 1.05 B13.1% of 8.03 B
        Llama 3.2 1B128,2562,048yes262.7 M21.3% of 1.24 B
        Qwen2.5-0.5B151,936896yes136.1 M27.6% of 0.49 B

        Sizes from the public model configurations; the totals match a full count of all layers. [Certain for GPT-2 and the Llama models; Likely for Qwen2.5-0.5B.] Three lessons:

        • Vocabulary growth is expensive. Llama 2 to Llama 3 kept \(d = 4096\) but grew \(|V|\) four times, so the embedding parameters grew four times, from 262 M to 1.05 B.
        • Small models pay the most. The transformer blocks grow like \(d^2\) per layer, but the embeddings grow only like \(d\). For a small \(d\), a large vocabulary is a big share of the model. This is why small models almost always tie their embeddings.
        • Embedding parameters are cheap in compute. The input lookup costs no FLOPs. This is why scaling-law papers often count “non-embedding parameters” separately (session 25). The output layer is different: \(E\mathbf{h}\) costs \(2|V|d\) FLOPs per token, and the logits tensor of shape \((B, T, |V|)\) can take a lot of memory.
        Figure 7. An embedding parameter calculator. A preset loads a real model’s \(|V|\), \(d\), tying, and the number of all its other (non-embedding) parameters. Change \(|V|\), \(d\) or tying to see what would happen to that model; the non-embedding part is kept fixed.
        How to read Figure 7

        The colored part of the bar is the embedding tables, the grey part is everything else. Load Llama 3 8B and move \(|V|\) down to 32,000: the embeddings shrink from 1.05 B to 262 M parameters, which is the Llama 2 tokenizer size. Load Qwen2.5-0.5B and untie: the model gains 136 M parameters and the embeddings become 43% of it. The readout also shows the memory of one table in bfloat16 and the FLOPs of the output layer per token, \(2|V|d\).

        How to choose \(d\) and \(|V|\)

        In an LLM the embedding size is not chosen alone: it is the width \(d\) of the whole residual stream, so it is set by the size of the model (768 for GPT-2 small, 4,096 for 7B to 8B models). The vocabulary size is a trade-off from session 8: a larger vocabulary makes sequences shorter (fewer tokens per sentence, so faster attention and longer effective context), but costs \(2|V|d\) more parameters in untied models and a bigger softmax at the output. For classic word2vec, \(d\) between 100 and 300 was common; the released Google News vectors use 300. [Certain]

        Beyond text: images and video in the same space

        A VLM must put images into the same \(d\)-dimensional space as text tokens, so that the transformer can read them side by side. Images have no vocabulary, so there is no table to look up. Instead:

        1. Patch embedding. The Vision Transformer cuts an image into small squares (patches), for example \(16 \times 16\) pixels. A \(224 \times 224\) image gives \(14 \times 14 = 196\) patches, each with \(16 \cdot 16 \cdot 3 = 768\) numbers. One linear layer maps each patch to a \(d\)-dimensional vector (session 164). [Certain] This is the image version of formula (11.2), except the input is a real-valued vector instead of a one-hot one.
        2. Projection into the LLM. The vision encoder’s output vectors go through a small projector (a linear layer or an MLP) that maps them to the LLM’s width \(d\). From then on they are treated like token embeddings.
        3. Video. A video is a sequence of frames, so it becomes many visual embeddings, often grouped by time. This is why token counts for video grow so fast, and why video systems compress or select frames.

        The word2vec idea also returns at full scale: CLIP (session 166) trains an image encoder and a text encoder so that matching image–caption pairs have high cosine and non-matching pairs have low cosine. It is the same “pull real pairs together, push random pairs apart” as negative sampling (11.6), with images in place of center words and captions in place of context words.

        Where this is used

        ApplicationWhat the embeddings doLink
        Every LLM and VLMInput lookup (11.2) turns token IDs into vectors; the output embedding (11.14) turns the last hidden vector into logitssession 12
        Semantic search and RAGDocuments and queries become vectors; ranking by cosine (11.10) finds related text, not only matching wordssession 150
        Text-to-video retrievalA caption and every clip are embedded in one shared space; the clips with the highest cosine are returnedsession 208
        Video fingerprinting and near-duplicate detectionEach frame or clip gets a descriptor vector; two videos match when many descriptors have cosine above a threshold. Centering (as in Figure 6) helps the threshold work.session 209
        Action recognition and key-moment detectionClip embeddings are compared with text embeddings of action names (“a person jumps”) for zero-shot labels and moment searchsession 210
        RecommendationItem IDs (videos, products) are looked up in an embedding table exactly like tokens; items watched by the same users get similar vectors, the word2vec idea applied to watch historiessession 151
        Adding new tokens in fine-tuningNew rows of \(E\) for special tokens or a new language; initialized from the mean row and trainedsession 57

        Pseudo-code

        Algorithm 1: embedding layer, forward and backward
        1. input: token IDs \(\mathrm{id}_1, \dots, \mathrm{id}_T\), table \(E \in \R^{|V| \times d}\)
        2. forward: for \(t = 1\) to \(T\) do \(X_{t,:} \leftarrow E_{\mathrm{id}_t,:}\) // a copy, no FLOPs, formula (11.3)
        3. backward: receive \(G = \partial \mathcal{L}/\partial X \in \R^{T \times d}\)
        4. \(\nabla E \leftarrow\) zeros\((|V|, d)\)
        5. for \(t = 1\) to \(T\) do
        6. \(\nabla E_{\mathrm{id}_t,:} \leftarrow \nabla E_{\mathrm{id}_t,:} + G_{t,:}\) // scatter-add: repeated IDs sum, formula (11.4)
        7. return \(\nabla E\) // rows of unseen tokens stay exactly 0
        Algorithm 2: skip-gram with negative sampling (one epoch)
        1. input: corpus, window \(m\), negatives \(k\), learning rate \(\eta\), tables \(\mathbf{v}, \mathbf{u}\)
        2. build \(P_n(w) \propto \mathrm{count}(w)^{3/4}\) // formula (11.8)
        3. for each position with center word \(c\) do
        4. for each context word \(o\) within \(m\) positions do
        5. draw \(n_1, \dots, n_k \sim P_n\)
        6. \(\mathbf{g} \leftarrow \mathbf{0}\)
        7. for \((w, y)\) in \(\{(o, 1), (n_1, 0), \dots, (n_k, 0)\}\) do
        8. \(e \leftarrow \sigma(\mathbf{u}_w \cdot \mathbf{v}_c) - y\) // prediction minus target
        9. \(\mathbf{g} \leftarrow \mathbf{g} + e\,\mathbf{u}_w\); \(\ \mathbf{u}_w \leftarrow \mathbf{u}_w - \eta\, e\, \mathbf{v}_c\)
        10. \(\mathbf{v}_c \leftarrow \mathbf{v}_c - \eta\, \mathbf{g}\) // formula (11.7)
        11. return \(\mathbf{v}\) // the word embeddings
        Algorithm 3: analogy and nearest neighbors with cosine
        1. input: embeddings \(\mathbf{v}_w\), words \(a, b, c\), optional mean vector \(\mathbf{m}\)
        2. (optional) \(\mathbf{v}_w \leftarrow \mathbf{v}_w - \mathbf{m}\) for all \(w\) // centering fixes anisotropy, formula (11.12)
        3. \(\hat{\mathbf{v}}_w \leftarrow \mathbf{v}_w / \lVert \mathbf{v}_w \rVert\) for all \(w\) // now dot product = cosine
        4. \(\mathbf{q} \leftarrow \hat{\mathbf{v}}_a - \hat{\mathbf{v}}_b + \hat{\mathbf{v}}_c\)
        5. \(s_w \leftarrow \hat{\mathbf{v}}_w \cdot \mathbf{q} / \lVert \mathbf{q} \rVert\) for all \(w \notin \{a, b, c\}\)
        6. return \(\argmax_w s_w\) // formula (11.11)

        The same ideas in PyTorch. Each block checks one claim from this session:

        import torch
        import torch.nn as nn
        import torch.nn.functional as F
        torch.manual_seed(0)
        
        # 1) lookup == one-hot @ E, formulas (11.2) and (11.3)
        V, d = 6, 4
        emb = nn.Embedding(V, d)
        ids = torch.tensor([0, 1, 2, 3, 0, 4])               # "the cat sat on the mat"
        onehot = F.one_hot(ids, V).float()                   # (6, 6)
        print(torch.allclose(emb(ids), onehot @ emb.weight)) # True
        
        # 2) only used rows get a gradient, repeated IDs add up, formula (11.4)
        G = torch.randn(len(ids), d)                         # pretend upstream gradient
        (emb(ids) * G).sum().backward()
        print(emb.weight.grad[5])                            # all zeros: "." is not in the batch
        print(torch.allclose(emb.weight.grad[0], G[0] + G[4]))   # True
        
        # 3) the SGNS loss of formula (11.6) on the worked example
        def sgns_loss(v_c, u_o, u_neg):                     # u_neg: (k, d)
            pos = F.logsigmoid(u_o @ v_c)
            neg = F.logsigmoid(-(u_neg @ v_c)).sum()
            return -(pos + neg)
        v_c = torch.tensor([0.5, -0.2], requires_grad=True)
        loss = sgns_loss(v_c, torch.tensor([0.4, 0.1]), torch.tensor([[-0.3, 0.6]]))
        loss.backward()
        print(round(loss.item(), 4), v_c.grad)               # 1.1744, tensor([-0.3119, 0.2142])
        
        # 4) a tiny skip-gram (full softmax, formula 11.5) on the toy corpus of Figure 3
        S = ["the cat eats fish", "the dog eats meat", "the cat sleeps at home", "the dog sleeps at home",
             "i feed the cat", "i feed the dog", "the car needs fuel", "the bus needs fuel",
             "the car drives on the road", "the bus drives on the road", "i park the car", "i park the bus"]
        words = sorted({w for s in S for w in s.split()})
        ix = {w: i for i, w in enumerate(words)}
        pairs = [(ix[t[i]], ix[t[j]]) for t in (s.split() for s in S)
                 for i in range(len(t)) for j in range(max(0, i - 2), min(len(t), i + 3)) if j != i]
        c, o = torch.tensor(pairs).T                         # 144 pairs
        v_in, u_out = nn.Embedding(len(words), 8), nn.Embedding(len(words), 8)
        with torch.no_grad(): u_out.weight.mul_(0.1)
        opt = torch.optim.Adam(list(v_in.parameters()) + list(u_out.parameters()), lr=0.05)
        for step in range(500):
            logits = v_in(c) @ u_out.weight.T                # (144, |V|): u_w . v_c for all w
            loss = F.cross_entropy(logits, o)
            opt.zero_grad(); loss.backward(); opt.step()
        E = F.normalize(v_in.weight.detach(), dim=1)         # unit rows: dot product = cosine
        cs = lambda a, b: round((E[ix[a]] @ E[ix[b]]).item(), 3)
        print("cat-dog", cs("cat", "dog"), "car-bus", cs("car", "bus"), "cat-car", cs("cat", "car"))
        # cat-dog and car-bus come out clearly higher than cat-car; exact values depend on the seed
        
        # 5) parameter count |V| x d without allocating memory, formula (11.13)
        big = nn.Embedding(128256, 4096, device="meta")    # Llama 3 vocabulary and width
        print(big.weight.numel())                            # 525336576
        
        # 6) weight tying, formula (11.14): the LM head reuses the same tensor
        tok_emb = nn.Embedding(V, d)
        lm_head = nn.Linear(d, V, bias=False)
        lm_head.weight = tok_emb.weight                      # both (V, d): one tensor, two uses
        h = torch.randn(d)
        print(torch.allclose(lm_head(h), tok_emb.weight @ h))    # True

        Common mistakes

        • Feeding token IDs as numbers. A linear layer on raw IDs treats them as amounts and makes neighbors in ID order look similar. Always go through an embedding table (or one-hot).
        • Changing the tokenizer without changing the table. Adding tokens without resize_token_embeddings (or the equivalent) gives index-out-of-range errors, or worse, silently reuses the wrong rows if the table was larger. And a tied output layer must be resized together with the input table.
        • Random init for new tokens. New rows drawn from \(\mathcal{N}(0, 1)\) can be far larger than trained rows (which may have a much smaller scale), and the model’s first steps go wild. Start new rows at the mean of the existing rows, or at a similar scale.
        • Using the dot product when the model was trained with cosine (or the reverse). Embedding lengths often reflect frequency. Match the similarity used in training, and normalize if in doubt.
        • Reading raw cosine values as absolute meaning. “0.8 means similar” is not true in an anisotropic space where random pairs already score 0.8 (Figure 6). Compare against a baseline of random pairs, or center first.
        • Trusting analogy demos. The classic results exclude the input words (11.11). Without that rule, the nearest word is often one of the inputs (Figure 5). Analogies are a probe, not a proof that the space is linear.
        • Forgetting the output side when counting. An untied model has two \(|V| \times d\) matrices. For Llama 3 8B that is 1.05 B parameters, not 525 M.
        • Weight decay on the embedding table without thinking. With AdamW, rows of rare tokens shrink at every step but are seldom pushed back by a gradient. Many training codes exclude embeddings (and norms) from weight decay. [Likely]

        Interview questions

        What is an embedding layer, and why is it the same as multiplying a one-hot vector by a matrix?basic
        It is a learned table \(E \in \R^{|V| \times d}\) with one row per token; the layer returns row \(\mathrm{id}\) for each token ID. A one-hot row vector \(\mathbf{e}_i^\top\) times \(E\) multiplies every row by 0 except row \(i\), so \(\mathbf{e}_i^\top E = E_{i,:}\) (formula 11.2). So the embedding layer is a bias-free linear layer on one-hot inputs, implemented as an indexed read that skips the \(|V| \cdot d\) multiplications by zero.
        Why not feed token IDs as numbers, or use one-hot vectors directly as the model input?basic
        IDs are arbitrary labels; as numbers they impose a fake order and scale. One-hot vectors fix that but are huge (\(|V|\) up to 100k to 250k), and all pairs are perpendicular, so no similarity can be expressed and nothing learned about one token helps another. A dense \(d\)-dimensional embedding is small and lets similar tokens share directions, which lets the model generalize.
        Which rows of the embedding table get a gradient in one step, and does that mean only those rows change?core
        Only rows of tokens present in the batch get a non-zero gradient, and repeated tokens get the sum of their gradients (formula 11.4); it is a scatter-add. But with a dense Adam/AdamW optimizer, other rows can still move: their momentum from earlier steps is not zero, and decoupled weight decay shrinks every row every step. Sparse optimizers (for example sparse=True with SparseAdam) update only the touched rows. With tied weights, every row also gets a gradient through the output softmax.
        Explain skip-gram with negative sampling. Why negative sampling instead of the full softmax?core
        Skip-gram predicts context words from a center word with \(p(o \mid c) \propto \exp(\mathbf{u}_o \cdot \mathbf{v}_c)\) (11.5). The normalizer sums over the whole vocabulary for each pair, which is too expensive. SGNS replaces it with \(k + 1\) binary classifications: the real pair should get \(\sigma(\mathbf{u}_o \cdot \mathbf{v}_c) \to 1\), and \(k\) sampled negatives should get \(\sigma \to 0\) (11.6). Cost per pair drops from \(|V|\) to \(k + 1\) dot products. It is no longer a normalized language model, but it gives good vectors; at the optimum the score approximates PMI minus \(\log k\) (11.9).
        Why are negatives sampled from \(\mathrm{count}^{3/4}\)?core
        Raw frequency would make most negatives “the”, “a”, “of”, which are easy and teach little; uniform sampling would pick mostly rare words, which are not realistic contrast. The power 3/4 is in between: it lowers the share of frequent words and raises rare ones (counts 100, 10, 1 give 0.901, 0.090, 0.009 raw and 0.827, 0.147, 0.026 after the power). Mikolov et al. found it worked best empirically.
        Why do vector analogies like \(\text{king} - \text{man} + \text{woman} \approx \text{queen}\) work, and when do they fail?core
        If a relation (male to female) changes the contexts of many word pairs in the same way, SGNS turns it into a roughly constant offset, because scores approximate log co-occurrence statistics (PMI), and differences of logs are ratios that can be shared across pairs. They fail when relations are not consistent, for rare words, for words with many meanings, and they look much better than they are because evaluations exclude the three input words: without that rule the nearest neighbor is often one of the inputs.
        All cosine similarities in your embedding model are between 0.6 and 0.9. What is going on and what do you do?advanced
        The space is anisotropic: vectors share a large common direction (a mean offset, often linked to frequency), so even unrelated pairs have high cosine. With a shared part of relative size \(\alpha\), random pairs get about \(\alpha^2/(1 + \alpha^2)\) (formula 11.12). Fixes: subtract the mean vector (centering), remove the top principal directions (all-but-the-top), or use whitening. For retrieval, compare to a baseline of random pairs and tune thresholds on centered vectors. Contrastive fine-tuning also spreads the space out.
        How many parameters are in Llama 3 8B’s embeddings? Why is the share larger in small models?core
        \(|V| \cdot d = 128{,}256 \times 4{,}096 = 525{,}336{,}576\) per table; Llama 3 8B does not tie, so input plus output is about 1.05 B, 13.1% of 8.03 B. In bfloat16 one table is about 1.05 GB. Transformer blocks scale like \(L d^2\), embeddings like \(|V| d\), so with small \(d\) the embeddings dominate: Qwen2.5-0.5B ties and still has 27.6% of its weights in the embedding table.
        What is weight tying? What are its benefits and costs?core
        The output layer reuses the input embedding matrix: logits \(\mathbf{z} = E\mathbf{h}\) (11.14). Benefits: removes \(|V| \cdot d\) parameters (big for small models), every row gets a gradient every step through the softmax (helps rare tokens), and Press and Wolf found lower perplexity. Costs: one matrix must serve both as an input code and as an output classifier, which can conflict; many large models (Llama 2 and Llama 3 8B, for example) keep them separate. In code, both modules must share one tensor, and resizing must update both.
        How does an image become “tokens” for an LLM if there is no vocabulary?core
        The image is cut into patches (for example 16×16 pixels; 196 patches for 224×224). Each flattened patch (768 numbers for RGB) goes through a linear layer: a patch embedding, the real-valued version of formula (11.2). A vision encoder processes these, and a projector maps the outputs to the LLM width \(d\). The LLM then treats them like token embeddings in the same sequence. Video adds many frames, so many more visual embeddings.
        You add 10 new special tokens before fine-tuning. How do you set up their embeddings?advanced
        Resize the input table (and the output layer, tied or not) to the new vocabulary size. Initialize new rows near the distribution of trained rows, for example the mean row (possibly plus small noise), not fresh \(\mathcal{N}(0, 1)\) values that may be far off-scale. Make sure the new rows are trainable even if you use LoRA (many setups freeze embeddings; you may need to unfreeze or train those rows). Check that the tokenizer actually emits the new IDs and that they appear often enough in the fine-tuning data; rarely seen rows stay near their initial values.
        What is the difference between a static word embedding and a contextual embedding?basic
        A static embedding (word2vec, GloVe, or the input table of an LLM) gives one vector per token type, regardless of the sentence: “bank” has one vector. A contextual embedding is a hidden state after some transformer layers, so it depends on the surrounding tokens: “river bank” and “bank loan” give different vectors. LLMs start from static embeddings and build contextual ones layer by layer (next session).

        Exercises

        1. A model has \(|V| = 32{,}000\) and \(d = 4{,}096\). How many embedding parameters does it have with tied and with untied embeddings? How many megabytes is one table in bfloat16?
        2. A batch contains the token IDs \((3, 1, 3, 3)\) and upstream gradients \(\mathbf{g}_1, \dots, \mathbf{g}_4\). Which rows of \(E\) get a non-zero gradient, and what is the gradient of row 3?
        3. Compute the cosine between \(\mathbf{a} = (1, 2, 2)\) and \(\mathbf{b} = (2, 1, 2)\).
        4. In SGNS, a real pair has score \(\mathbf{u}_o \cdot \mathbf{v}_c = 0\). What is its loss term \(-\log \sigma(0)\), and what is the factor \(\sigma(0) - 1\) in the gradient (11.7)?
        5. In Figure 6, what mean cosine do you expect at \(\alpha = 1\) and at \(\alpha = 3\)? What happens after centering?
        6. A 2B-parameter model uses \(|V| = 256{,}000\) and \(d = 2{,}048\) with tied embeddings. How many embedding parameters is that, and roughly what share of the model?
        7. Why does the one-hot product for Llama 3 8B cost about 1.05 GFLOPs per token? Compare it with the total forward cost of about \(2N\).
        Answerscheck
        1. Tied: \(32{,}000 \times 4{,}096 = 131{,}072{,}000\). Untied: \(262{,}144{,}000\). One table in bfloat16: \(131{,}072{,}000 \times 2\) bytes \(\approx 262\) MB. 2. Only rows 1 and 3. Row 1 gets \(\mathbf{g}_2\); row 3 gets \(\mathbf{g}_1 + \mathbf{g}_3 + \mathbf{g}_4\). 3. \(\mathbf{a}\cdot\mathbf{b} = 2 + 2 + 4 = 8\); both lengths are 3; cosine \(= 8/9 = 0.889\). 4. \(\sigma(0) = 0.5\), so the loss term is \(\log 2 = 0.693\) and the factor is \(-0.5\): the center vector moves toward \(\mathbf{u}_o\) with half strength. 5. \(1/(1+1) = 0.5\) and \(9/10 = 0.9\) (the figure measures 0.513 and 0.905). After centering the mean is about \(-1/39 \approx -0.026\) for any \(\alpha\). 6. \(256{,}000 \times 2{,}048 = 524{,}288{,}000\), about 0.52 B, roughly a quarter of a 2B model. 7. A \((1 \times |V|)(|V| \times d)\) product costs \(2|V|d = 2 \times 128{,}256 \times 4{,}096 = 1{,}050{,}673{,}152\) FLOPs. The forward pass costs about \(2 \times 8.03\) B \(\approx 16.1\) GFLOPs per token, so the wasted work would be about 6.5% extra, for a result that a memory read gives for free.

        Key takeaways

        • An embedding layer is a learned table \(E \in \R^{|V| \times d}\). Looking up row \(i\) equals multiplying the one-hot vector \(\mathbf{e}_i\) by \(E\) (11.2), so the table trains with ordinary backpropagation.
        • The gradient of a row is the sum of the gradients at the positions where that token appears (11.4); tokens not in the batch get zero, so rare tokens learn slowly.
        • word2vec makes words with similar contexts get similar vectors: skip-gram (11.5), made cheap by negative sampling (11.6–11.8). At the optimum it factorizes a shifted PMI table (11.9).
        • Cosine similarity (11.10) gives nearest neighbors and analogies (11.11), but it mixes antonyms with synonyms, gives one vector to words with many meanings, copies bias, and can be inflated by a shared direction (11.12). Center before trusting raw values.
        • Embeddings cost \(|V| \cdot d\) parameters per table (11.13): 525 M per table for Llama 3 8B, over a quarter of the weights in small models. Weight tying (11.14) reuses the input table as the output layer, so every row also gets an output-side gradient (11.15).
        • Images and video enter the same space through patch embeddings and a projector, and CLIP uses the same “pull real pairs together” idea as negative sampling.

        Key formulas

        \[ \mathbf{x} = \mathbf{e}_i^\top E = E_{i,:}, \qquad \frac{\partial \mathcal{L}}{\partial E_{j,:}} = \sum_{t:\,\mathrm{id}_t = j} \mathbf{g}_t, \qquad N_{\text{emb}} = |V|\cdot d \]
        \[ \mathcal{L}_{\text{SGNS}} = -\log \sigma(\mathbf{u}_o \cdot \mathbf{v}_c) - \sum_{i=1}^{k} \log \sigma(-\mathbf{u}_{n_i} \cdot \mathbf{v}_c), \qquad \cos(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert} \]
        \[ \mathbf{z} = E\,\mathbf{h}, \qquad \frac{\partial \mathcal{L}}{\partial E_{j,:}}\bigg|_{\text{output}} = (p_j - y_j)\,\mathbf{h} \quad \text{(tied weights)} \]

        References for this session

        1. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 5: Embeddings
        2. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 3: Words and Word Vectors
        3. Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space
        4. Mikolov, Sutskever, Chen, Corrado, Dean (2013). Distributed Representations of Words and Phrases and their Compositionality
        5. Pennington, Socher, Manning (2014). GloVe: Global Vectors for Word Representation
        6. Levy & Goldberg (2014). Neural Word Embedding as Implicit Matrix Factorization
        7. Bojanowski, Grave, Joulin, Mikolov (2017). Enriching Word Vectors with Subword Information
        8. Press & Wolf (2017). Using the Output Embedding to Improve Language Models
        9. Vaswani et al. (2017). Attention Is All You Need
        10. Bolukbasi, Chang, Zou, Saligrama, Kalai (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
        11. Mu & Viswanath (2018). All-but-the-Top: Simple and Effective Postprocessing for Word Representations
        12. Ethayarajh (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
        13. Gao, He, Tan, Qin, Wang, Liu (2019). Representation Degeneration Problem in Training Natural Language Generation Models
        14. Llama Team, Meta (2024). The Llama 3 Herd of Models
        15. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale