LLM & VLM courseBlog
15Track 1: Foundations · session 15 of 15

embeddings and RoPE by hand

In this lab you open GPT-2’s real embedding table, project 63 words to 2D with PCA and find their nearest neighbors among all 50,257 tokens. Then you write RoPE in about 15 lines of PyTorch, prove with numbers that the attention score depends only on the distance between two tokens, and plot the rotation frequencies of Llama and Qwen to see which dimensions measure short and long distances.

Time: 3 to 4 hours (mostly hands-on)Level: Beginner · hands-on labBefore this: Sessions 11 and 14 (embeddings, RoPE), PyTorch basics

What you will learn

  • How to load a trained embedding table and check that a word is a single token
  • PCA with an SVD, the explained variance ratio, and why a 2D plot keeps only a small part of the picture
  • Nearest neighbors with cosine similarity over a full vocabulary, and the effect of centering (anisotropy)
  • How to implement RoPE as a rotation of pairs, in both the interleaved and the rotate_half layout
  • How to test with numbers that RoPE scores depend only on m − n, and why absolute sinusoidal positions fail the same test
  • RoPE frequencies and wavelengths for the real bases of Llama 2, Llama 3 and Qwen2.5, and which pairs handle short and long distances
  • Common RoPE bugs: wrong layout, wrong exponent, low-precision angles, wrong positions with a cache

Why this matters

The first two things a transformer does with a token are: look up its vector in the embedding table, and tell attention where the token is. In modern LLMs the second step is usually RoPE (rotary position embedding). In this lab you open a real trained embedding table, look at its geometry, and then write RoPE yourself and prove with numbers that it does what the papers say.

Session 11 (session 11) explained that the embedding table is a matrix with one row per token, and that training puts similar tokens close together. Session 14 (session 14) explained RoPE: rotate pairs of numbers by an angle that grows with the position. Here you check both claims yourself, with real weights and real code. You will:

  1. Look at trained embeddings. Load GPT-2’s table (50,257 rows of 768 numbers), project 63 words to 2D with PCA, and find nearest neighbors in the full vocabulary.
  2. Implement RoPE in about 15 lines of PyTorch, and test with numbers that the attention score depends only on the distance between two tokens, not on where they are.
  3. Plot the RoPE frequencies and see which pairs of dimensions measure short distances and which measure long ones, for the real settings of Llama 2, Llama 3 and Qwen2.5.

Why does this matter for real work? Nearest-neighbor search over embeddings is the core of semantic search and retrieval (RAG). RoPE is in Llama, Qwen, Mistral and most open LLMs, and its 2D and 3D versions are how Qwen2-VL tells image patches and video frames apart. Bugs in RoPE are common and silent: the model still runs, it just gets worse. The tests you write in this lab are the tests you would run when you port a model or extend its context window.

Setup

You need Python 3.9 or newer, PyTorch and Hugging Face transformers. The first run downloads GPT-2 small (about 550 MB). Everything else is small and runs on a laptop CPU in seconds.

pip install torch transformers

All code on this page is deterministic. The trained weights are fixed, PCA and cosine similarity have no randomness, and the random test vectors use torch.manual_seed(0). The numbers in the code comments are what the code printed when we ran it (PyTorch 2.8, transformers 4.57). The figures compute their numbers live in your browser from the same data.

Part A, step 1: open the embedding table

GPT-2 small has a token embedding table wte of shape \(50{,}257 \times 768\): one row of \(d = 768\) numbers for each token in its vocabulary. [Certain: the model’s config.json has vocab_size 50257 and n_embd 768.] The average row length (L2 norm) is 3.96. GPT-2 also has a second table, wpe, of shape \(1{,}024 \times 768\): one learned vector per position. GPT-2 does not use RoPE; we come back to wpe in Part B.

We pick 63 common English words in 8 groups: numbers, days, months, colors, animals, countries, people and verbs. We write each one with a leading space (“␣king”, not “king”), because that is how a word looks in the middle of a sentence for a GPT-2 tokenizer (session 10). The code checks that each word is exactly one token; if it were two tokens, there would be no single row to look at.

Part A, step 2: PCA to two dimensions

PCA (principal component analysis) finds the directions in which the data varies the most. We stack our 63 vectors as rows of a matrix \(X\), subtract the mean row, and take the singular value decomposition (SVD):

\[ X_c = X - \mathbf{1}\mu^{\top} = U \Sigma V^{\top}, \qquad Z = X_c\, V_{:,1:2} \tag{15.1} \]
Read the symbols
\(X\)
the \(63 \times 768\) matrix of embeddings, one word per row
\(\mu\), \(\mathbf{1}\)
the mean of the rows (a 768-vector), and a column of ones, so \(\mathbf{1}\mu^{\top}\) subtracts the mean from every row
\(U, \Sigma, V\)
the SVD: \(V\)’s columns are the principal directions, \(\Sigma\) holds the singular values \(\sigma_1 \ge \sigma_2 \ge \dots\)
\(Z\)
the \(63 \times 2\) coordinates we plot: each word projected on the first two directions

How much of the picture do we keep? Each direction \(j\) carries a share of the total variance:

\[ r_j = \frac{\sigma_j^2}{\sum_{l} \sigma_l^2} \tag{15.2} \]

For our 63 words, \(r_1 = 11.2\%\), \(r_2 = 8.5\%\) and \(r_3 = 6.9\%\). So the 2D plot keeps only 19.7% of the variance. That is normal for embeddings: meaning is spread over many directions. Keep this in mind when you read any 2D plot of embeddings.

Figure 1. 63 rows of GPT-2’s real embedding table, projected with PCA (formula 15.1). Colors are the 8 word groups. Click a word: lines join it to its 3 nearest words in this set, measured with cosine in all 768 dimensions, and the boxes list its 8 nearest tokens in the full 50,257-token vocabulary. Labels that would overlap are hidden; hover over or click a point to see its word.
How to read Figure 1

On PC1 vs PC2, time words (days and months) sit on the right, and countries at the top: these two groups explain the most variance in this set. Colors and numbers are clearer on PC3. The readout counts, for every word, whether its nearest point in the 2D picture is from the same group: 54 of 63 on PC1 vs PC2. In the full 768 dimensions, with cosine, it is 63 of 63. The picture loses information; the vectors do not. Click “␣king”: its vocabulary neighbors are “␣kings”, “␣King”, “␣queen”, “King”, “␣prince”, “␣monarch”. Switch to “Centered vocabulary” to see the same lists after subtracting the mean of all 50,257 rows.

Part A, step 3: nearest neighbors

To find the tokens most similar to token \(w\), we use cosine similarity: the dot product of the two vectors after scaling both to length 1. It ignores length and looks only at direction:

\[ \cos(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a}^{\top}\mathbf{b}}{\lVert \mathbf{a} \rVert\, \lVert \mathbf{b} \rVert}, \qquad \mathrm{NN}_k(w) = \text{top-}k_{\,j \ne w}\ \cos(E_w, E_j) \tag{15.3} \]
Read the symbols
\(E_w\)
row \(w\) of the embedding table, the vector of token \(w\)
\(\lVert \mathbf{a} \rVert\)
the length (L2 norm) of \(\mathbf{a}\)
\(\mathrm{NN}_k(w)\)
the \(k\) tokens \(j\) with the highest cosine to \(w\), leaving out \(w\) itself

In practice you normalize the whole table once, \(\hat E = E / \lVert E \rVert_{\text{row}}\), and then one matrix-vector product \(\hat E \hat E_w\) gives the cosine to all 50,257 tokens at once. Three things stand out in the results:

  • Neighbors are often the same word in another form. “␣dog” → “␣dogs” (0.796), “␣Dog” (0.734), “Dog” (0.672), “␣canine” (0.652). The tokenizer made separate tokens for case, plural and leading space; the embedding learned that they mean almost the same thing.
  • Groups are clean. The 6 nearest tokens of “␣Monday” are the other 6 weekdays, with cosines from 0.89 down to 0.80. For “␣three” they are four, two, five, six, seven, nine.
  • All cosines are high. The average cosine between 2,000 random GPT-2 tokens is 0.27, not 0. The vectors share a common direction: this is the anisotropy from session 11. Subtracting the mean of all rows (“centering”) brings that average to almost exactly 0 and lowers every score, but in our examples the order of the neighbors barely changes.
Algorithm 1: PCA plot and nearest neighbors of an embedding table
  1. input: table \(E \in \R^{|V| \times d}\), tokenizer, word list, \(k\)
  2. ids \(\leftarrow\) [ tokenize(" " + w) for w in words ]; assert each has length 1
  3. \(X \leftarrow E[\text{ids}]\);  \(X_c \leftarrow X - \text{mean}(X)\) // center, or PC1 is just the mean
  4. \(U, \Sigma, V^{\top} \leftarrow \text{svd}(X_c)\);  \(Z \leftarrow X_c V_{:,1:2}\);  report \(r_1 + r_2\) // (15.1), (15.2)
  5. plot \(Z\), colored by group
  6. \(\hat E \leftarrow E\) with every row divided by its norm
  7. for w in words do
  8. \(s \leftarrow \hat E\, \hat E_w\);  \(s_w \leftarrow -\infty\) // (15.3), skip the word itself
  9. print top-\(k\) of \(s\), decoded to text

Here is the code. It is the reference solution: try to write it yourself first.

import torch
from transformers import AutoTokenizer, GPT2Model

tok = AutoTokenizer.from_pretrained("gpt2")
model = GPT2Model.from_pretrained("gpt2")
W = model.wte.weight.detach()                 # (50257, 768) token embedding table
print(W.shape, W.norm(dim=1).mean().item())   # torch.Size([50257, 768]) ~3.96

words = ["one", "two", "three", "four", "five", "six", "seven", "eight",
         "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday",
         "January", "February", "March", "April", "May", "June", "July", "August",
         "red", "blue", "green", "yellow", "black", "white", "purple", "orange",
         "dog", "cat", "horse", "cow", "lion", "tiger", "bird", "fish",
         "France", "Germany", "Italy", "Spain", "China", "Japan", "India", "Russia",
         "king", "queen", "man", "woman", "boy", "girl", "mother", "father",
         "run", "walk", "eat", "drink", "sleep", "write", "read", "sing"]
ids = []
for w in words:
    t = tok.encode(" " + w)                   # leading space: the form used inside a sentence
    assert len(t) == 1, (w, t)                # every word must be ONE token
    ids.append(t[0])
X = W[ids]                                    # (63, 768)

# PCA by SVD of the centered matrix
Xc = X - X.mean(dim=0)
U, S, Vh = torch.linalg.svd(Xc, full_matrices=False)
var = S**2 / (S**2).sum()
print(var[:3])                                # tensor([0.1117, 0.0853, 0.0694])
Z = Xc @ Vh[:2].T                             # (63, 2) coordinates to plot

# nearest neighbors by cosine over the WHOLE vocabulary
Wn = W / W.norm(dim=1, keepdim=True)
def neighbors(word, k=6):
    i = tok.encode(" " + word)[0]
    s = Wn @ Wn[i]
    s[i] = -1.0                               # do not return the word itself
    v, j = s.topk(k)
    return [(tok.decode([int(t)]), round(float(c), 3)) for c, t in zip(v, j)]
print(neighbors("king"))
# [(' kings', 0.733), (' King', 0.696), (' queen', 0.657), ('King', 0.604), (' prince', 0.59), (' monarch', 0.581)]
print(neighbors("Monday"))
# [(' Tuesday', 0.89), (' Thursday', 0.845), (' Wednesday', 0.844), (' Friday', 0.821), (' Sunday', 0.806), (' Saturday', 0.801)]

# anisotropy check: average cosine between random tokens
g = torch.Generator().manual_seed(0)
r = torch.randint(0, W.shape[0], (2000,), generator=g)
C = Wn[r] @ Wn[r].T
off = ~torch.eye(2000, dtype=torch.bool)
print(C[off].mean().item())                   # ~0.27, far from 0

Part B, step 4: RoPE is a rotation of pairs

GPT-2 adds a learned position vector to each token vector. Its 1,024 learned position vectors are very regular: the first 3 principal directions hold 90.4% of their variance (33.4%, 32.3% and 24.6%), against 26.6% for the first 3 directions of our 63 words. The model learned smooth, wave-like patterns over position by itself. RoPE builds such waves in by design, and it puts them inside attention instead of adding them to the input.

RoPE acts on the query and key vectors of one attention head, after their projections and before the dot product (session 16, session 16, explains queries and keys; for now, \(\mathbf{q}\) and \(\mathbf{k}\) are just two vectors of length \(d\) whose dot product is the attention score). Split a vector \(\mathbf{x} \in \R^d\) into \(d/2\) pairs. Pair \(i\) is treated as a point in a plane and rotated by the angle \(m\theta_i\), where \(m\) is the token’s position:

\[ \begin{pmatrix} y_{2i} \\ y_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos m\theta_i & -\sin m\theta_i \\ \sin m\theta_i & \cos m\theta_i \end{pmatrix} \begin{pmatrix} x_{2i} \\ x_{2i+1} \end{pmatrix}, \qquad i = 0, \dots, \tfrac{d}{2} - 1 \tag{15.4} \]
\[ \theta_i = b^{-2i/d} \tag{15.5} \]
Read the symbols
\(x_{2i}, x_{2i+1}\)
the two numbers of pair \(i\) before rotation (indices start at 0)
\(y_{2i}, y_{2i+1}\)
the same pair after rotation
\(m\)
the position of the token in the sequence: 0, 1, 2, …
\(\theta_i\)
the frequency of pair \(i\): how many radians it turns per position
\(b\)
the base: 10,000 in the RoFormer paper and Llama 2, 500,000 in Llama 3, 1,000,000 in Qwen2.5
\(d\)
the head dimension (64 or 128 in the models in this lab)

We write \(R_m\) for the whole \(d \times d\) matrix: a block-diagonal matrix with one \(2 \times 2\) rotation per pair. Two facts follow at once. A rotation keeps lengths, so \(\lVert R_m \mathbf{x} \rVert = \lVert \mathbf{x} \rVert\). And rotations add: \(R_a^{\top} R_b = R_{b-a}\), because turning back by \(a\) and forward by \(b\) is the same as turning by \(b - a\).

Part B, step 5: why the score depends only on m − n

Put the query at position \(m\) and the key at position \(n\). Using \(R_m^{\top} R_n = R_{n-m}\):

\[ s(m, n) = (R_m \mathbf{q})^{\top} (R_n \mathbf{k}) = \mathbf{q}^{\top} R_m^{\top} R_n\, \mathbf{k} = \mathbf{q}^{\top} R_{n-m}\, \mathbf{k} \tag{15.6} \]

The positions appear only as the difference \(n - m\). Written out pair by pair, with \(\varphi_i = (n-m)\theta_i\):

\[ s(m, n) = \sum_{i=0}^{d/2-1} \Big[ A_i \cos \varphi_i + B_i \sin \varphi_i \Big], \quad A_i = q_{2i}k_{2i} + q_{2i+1}k_{2i+1}, \quad B_i = q_{2i+1}k_{2i} - q_{2i}k_{2i+1} \tag{15.7} \]
Read the symbols
\(s(m,n)\)
the raw attention score between a query at position \(m\) and a key at position \(n\) (before scaling and softmax)
\(R_{n-m}\)
one rotation matrix that depends only on the distance
\(A_i\)
the ordinary dot product of pair \(i\) of \(\mathbf{q}\) and \(\mathbf{k}\): the score of that pair when the distance is 0
\(B_i\)
the 2D cross product of the two pairs; it makes the score different for \(n - m\) and \(m - n\)
\(\varphi_i\)
the angle between the two rotations for pair \(i\)
Worked example: \(d = 4\), base 10,000

Two pairs, so \(\theta_0 = 10000^{0} = 1\) and \(\theta_1 = 10000^{-2/4} = 0.01\). Take \(\mathbf{q} = (1, 2, 0.5, -1)\) and \(\mathbf{k} = (2, 1, 1, 0.5)\). Without rotation, \(\mathbf{q}^{\top}\mathbf{k} = 2 + 2 + 0.5 - 0.5 = 4\).

Query at 5, key at 2. Rotating gives \(R_5\mathbf{q} = (2.2015, -0.3916, 0.5494, -0.9738)\) and \(R_2\mathbf{k} = (-1.7416, 1.4024, 0.9898, 0.5199)\). Their dot product is −4.3458.

Check with formula 15.7. Here \(n - m = -3\). Pair 0: \(A_0 = 1\cdot2 + 2\cdot1 = 4\), \(B_0 = 2\cdot2 - 1\cdot1 = 3\), \(\varphi_0 = -3\), so \(4\cos(-3) + 3\sin(-3) = -3.9600 - 0.4234 = -4.3833\). Pair 1: \(A_1 = 0.5 - 0.5 = 0\), \(B_1 = -1\cdot1 - 0.5\cdot0.5 = -1.25\), \(\varphi_1 = -0.03\), so \(-1.25\sin(-0.03) = 0.0375\). Sum: \(-4.3833 + 0.0375 = -4.3458\). ✓

Move both. Positions (13, 10) and (103, 100) give exactly −4.3458 again: the rotated vectors are completely different, the score is the same. Positions (2, 5), distance +3 instead of −3, give −3.5741: RoPE knows the direction of the distance, because of the \(B_i \sin\varphi_i\) terms. Positions (3, 3) give 4.0, the plain dot product.

5 2
Figure 2. The worked example, live. Each circle is one pair of dimensions. The faint arrows are \(\mathbf{q}\) (blue) and \(\mathbf{k}\) (orange) before rotation; the solid arrows are after rotating by \(m\theta_i\) and \(n\theta_i\). Pair 0 turns 1 radian per position; pair 1 turns 0.01.
How to read Figure 2

Press “Move both by +10” a few times. The arrows spin, fast in pair 0 and slowly in pair 1, but the angle between the blue and orange arrow in each circle stays the same, and so do the per-pair scores and the total −4.3458. Now move only one slider: the score changes. Set \(m = n\): the arrows in each circle line up as they were before rotation and the score is the plain \(\mathbf{q}^{\top}\mathbf{k} = 4\).

Part B, step 6: implement it and test it with numbers

Formula 15.6 is a proof. Code can still be wrong: wrong pairs, wrong sign, frequencies off by a factor of 2. So we turn the proof into a test. Take one random \(\mathbf{q}\) and one random \(\mathbf{k}\), place copies of them at every position \(0..T-1\), and compute the score matrix \(S_{mn} = s(m,n)\). If the scores depend only on \(m - n\), every diagonal of \(S\) is constant. We measure the worst violation:

\[ \text{spread}(S) = \max_{o}\ \Big( \max_{n - m = o} S_{mn} \;-\; \min_{n - m = o} S_{mn} \Big) \tag{15.8} \]

For a correct RoPE this is zero up to rounding. With \(d = 64\), \(T = 32\) and float32, our implementation gives a spread of \(1.1 \times 10^{-5}\), for scores between −17.0 and −2.9. The same test with absolute sinusoidal position vectors added to \(\mathbf{q}\) and \(\mathbf{k}\) (the original Transformer method, session 13) gives a spread of 18.7: the score depends on where the tokens are, not only on their distance.

Algorithm 2: RoPE and its relative-position test
  1. function rope(\(X \in \R^{T \times d}\), pos, \(b\)):
  2. \(\theta_i \leftarrow b^{-2i/d}\) for \(i = 0..d/2-1\) // (15.5), compute in float64
  3. angle\([t, i] \leftarrow \text{pos}[t] \cdot \theta_i\);  \(c, s \leftarrow \cos(\text{angle}), \sin(\text{angle})\)
  4. \(x_1 \leftarrow X[:, 0::2]\);  \(x_2 \leftarrow X[:, 1::2]\) // the two halves of each pair
  5. return interleave(\(x_1 c - x_2 s,\ x_1 s + x_2 c\)) // (15.4)
  6. test: \(\mathbf{q}, \mathbf{k} \sim \mathcal{N}(0, I)\) with a fixed seed; \(Q \leftarrow\) rope(repeat(\(\mathbf{q}, T\)), \(0..T-1\)); same for \(K\)
  7. assert all row norms of \(Q\) equal \(\lVert \mathbf{q} \rVert\) // rotations keep length
  8. \(S \leftarrow Q K^{\top}\);  assert spread\((S) \lt 10^{-4}\) // (15.8)
  9. assert \(S_{5,2}\) equals the hand value \(-4.3458\) for the \(d = 4\) example
10,000
Figure 3. The test of Algorithm 2 in your browser: one fixed random \(\mathbf{q}\) and \(\mathbf{k}\) (\(d = 16\), seeded), copied to positions 0 to 15. Cell \((m, n)\) is the score \(s(m,n)\); blue is high, orange is low. Click a cell to see all the cells on its diagonal.
How to read Figure 3

With RoPE, the colors run in stripes parallel to the main diagonal: every diagonal is one color, and the readout shows a spread (formula 15.8) of about \(10^{-15}\), which is rounding error in JavaScript’s 64-bit numbers. Switch to “Absolute sinusoidal”: the stripes break, and the spread jumps to several units. “No position” gives one flat color: the model could not tell positions apart at all. Move the base slider: with a small base, all pairs turn fast and the pattern repeats quickly; with a large base, scores change slowly with distance.

Here is the reference code. Test 5 compares it with the Hugging Face Llama implementation.

import math, torch
torch.manual_seed(0)

def rope_freqs(d, base=10000.0):
    i = torch.arange(0, d, 2, dtype=torch.float64)      # 0, 2, 4, ..., d-2
    return base ** (-i / d)                              # theta_0 .. theta_{d/2-1}

def rope(x, pos, base=10000.0):
    """x: (T, d) with d even; pos: (T,) integer positions. Interleaved pairs (2i, 2i+1)."""
    d = x.shape[-1]
    ang = pos[:, None].double() * rope_freqs(d, base)[None, :]   # (T, d/2)
    c, s = ang.cos().to(x.dtype), ang.sin().to(x.dtype)
    x1, x2 = x[..., 0::2], x[..., 1::2]                  # first and second of each pair
    y1 = x1 * c - x2 * s
    y2 = x1 * s + x2 * c
    return torch.stack([y1, y2], dim=-1).flatten(-2)     # put the pairs back in place

# Test 1: rotation keeps the length of every vector
d, T = 64, 32
q, k = torch.randn(d), torch.randn(d)
pos = torch.arange(T)
Q = rope(q.expand(T, d), pos)
K = rope(k.expand(T, d), pos)
print(q.norm().item(), Q.norm(dim=1).min().item(), Q.norm(dim=1).max().item())   # 8.3705 for all three

# Test 2: score(m, n) depends only on m - n  (constant along every diagonal)
S = Q @ K.T                                              # S[m, n] = <R_m q, R_n k>
spread = max((S.diagonal(o).max() - S.diagonal(o).min()).item() for o in range(-T + 1, T))
print("max spread along diagonals:", spread)             # ~1e-5 in float32

# Test 3: the same test FAILS for absolute sinusoidal embeddings added to x
pe = torch.zeros(T, d)
p = torch.arange(T, dtype=torch.float32)[:, None]
div = 10000 ** (torch.arange(0, d, 2, dtype=torch.float32) / d)
pe[:, 0::2], pe[:, 1::2] = torch.sin(p / div), torch.cos(p / div)
S_abs = (q + pe) @ (k + pe).T
print("absolute PE spread:", max((S_abs.diagonal(o).max() - S_abs.diagonal(o).min()).item()
                                 for o in range(-T + 1, T)))   # ~18.69: depends on m and n, not only m - n

# Test 4: the worked example from the text
q4 = torch.tensor([1.0, 2.0, 0.5, -1.0], dtype=torch.float64)
k4 = torch.tensor([2.0, 1.0, 1.0, 0.5], dtype=torch.float64)
for m, n in [(5, 2), (13, 10), (103, 100), (2, 5), (3, 3)]:
    s = rope(q4[None], torch.tensor([m]))[0] @ rope(k4[None], torch.tensor([n]))[0]
    print(m, n, round(s.item(), 4))                      # -4.3458 three times, then -3.5741, 4.0

# Test 5: Hugging Face Llama uses "rotate_half": pairs are (i, i + d/2), not (2i, 2i+1)
from transformers.models.llama.modeling_llama import apply_rotary_pos_emb
ang = pos[:, None].double() * rope_freqs(d)[None, :]
emb = torch.cat([ang, ang], dim=-1)                      # (T, d) as in HF
cos, sin = emb.cos().float()[None, None], emb.sin().float()[None, None]
qh, kh = apply_rotary_pos_emb(q.expand(T, d)[None, None], k.expand(T, d)[None, None], cos, sin)
perm = torch.cat([torch.arange(0, d, 2), torch.arange(1, d, 2)])   # interleaved -> halves
print("HF == ours after permutation:",
      torch.allclose(qh[0, 0], rope(q[perm.argsort()].expand(T, d), pos)[:, perm], atol=1e-5))   # True

# Wavelengths: how many tokens before pair i turns a full circle
for base in (10000.0, 500000.0):
    lam = 2 * math.pi / rope_freqs(128, base)
    print(base, round(lam[0].item(), 2), round(lam[-1].item()), int((lam > 8192).sum()))
# 10000.0 6.28 54410 14       (14 of 64 pairs are slower than one turn per 8192 tokens)
# 500000.0 6.28 2559196 29

Two layouts of the same idea

The RoFormer paper pairs neighbors, \((x_{2i}, x_{2i+1})\), as in formula 15.4. The GPT-NeoX and Hugging Face Llama code pairs \(x_i\) with \(x_{i + d/2}\) instead, using a helper called rotate_half. [Certain: the HF function rotates the two halves of the vector.] Both are correct RoPE: one is the other with the dimensions reordered, which Test 5 checks. But the trained weights \(W_Q, W_K\) must match the layout. Meta’s original Llama checkpoints are converted to the Hugging Face layout by permuting rows of \(W_Q\) and \(W_K\). [Likely] If you load weights from one layout into code of the other, nothing crashes; the model simply gets much worse.

Part C, step 7: frequencies and wavelengths

The frequencies \(\theta_i = b^{-2i/d}\) form a geometric sequence: each pair turns slower than the previous one by the same factor \(b^{-2/d}\). The wavelength of pair \(i\) is the number of positions it needs for one full turn:

\[ \lambda_i = \frac{2\pi}{\theta_i} = 2\pi\, b^{2i/d}, \qquad \lambda_0 = 2\pi \approx 6.28, \qquad \lambda_{d/2-1} = 2\pi\, b^{(d-2)/d} \tag{15.9} \]
Read the symbols
\(\lambda_i\)
the wavelength of pair \(i\), in tokens
\(\lambda_0\)
the fastest pair: a full turn every 6.28 tokens, for any base
\(\lambda_{d/2-1}\)
the slowest pair: 54,410 tokens for \(d = 128\), \(b = 10{,}000\); 2,559,196 tokens for \(b = 500{,}000\)

This gives each pair a job. A fast pair (small \(i\)) changes a lot between neighbors, so it can tell distance 3 from distance 4. But after about 6 tokens it has turned a full circle, so it cannot tell distance 1 from distance 7. A slow pair (large \(i\)) barely moves between neighbors, but its angle grows steadily over thousands of tokens, so it can tell “a few hundred tokens back” from “many thousands back”. Together the pairs act like the digits of a number: seconds, minutes, hours, days.

A useful summary is the score when \(\mathbf{q} = \mathbf{k} = (1, \dots, 1)\). Then \(A_i = 2\) and \(B_i = 0\) in formula 15.7, and dividing by the distance-0 score \(d\) gives the average cosine:

\[ \bar s(\Delta) = \frac{2}{d} \sum_{i=0}^{d/2-1} \cos(\Delta\, \theta_i) \tag{15.10} \]

For \(d = 128\) and \(b = 10{,}000\): \(\bar s = 0.970\) at \(\Delta = 1\), 0.669 at 10, 0.477 at 100, 0.159 at 1,000 and −0.028 at 10,000. With \(b = 500{,}000\) the fall is slower: 0.978, 0.764, 0.611, 0.492 and 0.187. As \(\Delta\) grows, more and more pairs have turned many times and their cosines average out, so similar vectors far apart get a smaller score. The RoFormer paper calls this long-term decay. [Certain: our numbers; the paper proves an upper bound that decays.] A larger base keeps more pairs slow, so the decay stretches over a longer context.

100
Figure 4. Top: the wavelength \(\lambda_i\) of every pair (formula 15.9, log scale) for the real base and head size of four models. The dashed line is the training context length; pairs above it never finish one turn inside the context. Bottom: \(\cos(\Delta\theta_i)\) for each pair at the chosen distance \(\Delta\); the readout gives their average \(\bar s(\Delta)\) (formula 15.10).
How to read Figure 4

Start with Llama 2 7B (\(b = 10{,}000\), \(d = 128\), context 4,096). The wavelengths go from 6.28 up to 54,410 tokens; 18 of the 64 pairs lie above the 4,096 line. Move \(\Delta\) from 1 to 10,000. At \(\Delta = 1\) almost all bars are near +1. At \(\Delta = 100\) the left bars (fast pairs) jump between −1 and +1 with no pattern, while the right bars (slow pairs) are still near +1: only the slow pairs still give a smooth signal about distance. Switch to Llama 3 8B (\(b = 500{,}000\)): the curve is steeper, and 29 pairs lie above its 8,192 context line.

The table gives the numbers for real models, read from their config.json files on Hugging Face. [Certain: config values; Llama 2’s config has no rope_theta, so the library default of 10,000 is used.]

ModelBase \(b\)Head dim \(d\)Training contextSlowest \(\lambda\)Pairs with \(\lambda \gt\) context
Llama 2 7B10,0001284,09654,41018 of 64
Llama 3 8B500,0001288,1922,559,19629 of 64
Qwen2.5-7B1,000,00012832,7685,063,25624 of 64
Qwen2.5-0.5B1,000,0006432,7684,080,18512 of 32
Llama 3.2 1B500,000648,192 (before scaling)2,084,76514 of 32

Pairs whose wavelength is longer than the training context never see a full turn during training. When you run the model on a longer context, these pairs reach angles they never saw. This is why context extension methods change mainly the slow pairs: Position Interpolation divides all angles by a factor, YaRN treats fast and slow pairs differently, and Llama 3.1’s config scales the frequencies with rope_type: "llama3", factor 8 and an original context of 8,192. [Certain: config values; the methods are covered in the long-context sessions.]

A first look at 2D and 3D

For an image, a patch has a row and a column, not one position. 2D RoPE splits the pairs into groups: one group rotates with the row, the other with the column. Then the score depends only on the row offset and the column offset. Qwen2-VL’s M-RoPE does the same with three groups for (time, height, width). Its config sets mrope_section to [16, 24, 24]: 16 pairs for time, 24 for height and 24 for width, which is exactly the 64 pairs of its 128-dimensional heads. [Certain: config.json of Qwen2-VL-2B-Instruct.] Exercise 6 asks you to extend your rope function this way.

Where this is used

System or taskWhat from this lab it uses
Semantic search and RAGNormalize vectors, one matrix product, top-\(k\) (formula 15.3); at scale an approximate nearest-neighbor index does the same job
Model debugging and interpretabilityPCA plots and neighbor lists of embedding tables, to find duplicate tokens, untrained rows, or odd clusters
Llama 2 and 3, Qwen2.5, Mistral and most open LLMsRoPE on queries and keys in every attention layer, with the base as a design choice (table above)
Long-context extensionRescaling the slow pairs (Position Interpolation, YaRN, Llama 3.1 scaling); the wavelength plot shows which pairs are affected
Qwen2-VL images and videoM-RoPE: separate pair groups for time, height and width, so a patch’s score depends on its offsets in all three
Video retrieval and Video-RAGClip or frame embeddings searched with cosine nearest neighbors, exactly as Algorithm 1 does with tokens
Video fingerprinting and near-duplicate detectionNearest neighbors with a cosine threshold; centering the embeddings first reduces false matches from the shared direction [Likely]
Key-moment detection and action recognition in long videosThe temporal part of M-RoPE gives relative time between frames; the wavelength analysis tells how far apart in time two frames can still be told apart [Speculation]

Common mistakes

  • Mixing the two pair layouts. Interleaved \((2i, 2i+1)\) code with weights trained for the half-split layout \((i, i+d/2)\), or the reverse. Every test on random vectors still passes; only the real model gets worse. Compare against the reference implementation of the checkpoint (Test 5).
  • Rotating the values, or rotating after the dot product. RoPE is applied to \(\mathbf{q}\) and \(\mathbf{k}\) only, after their projections and before \(\mathbf{q}^{\top}\mathbf{k}\). The values \(\mathbf{v}\) are not rotated.
  • Wrong frequency exponent. \(\theta_i = b^{-2i/d}\) with \(i = 0..d/2-1\), or equivalently arange(0, d, 2) / d. Writing arange(d/2) / d halves every exponent and silently changes all wavelengths.
  • Computing angles in low precision at large positions. In our test, with the angles \(m\theta_i\) computed in float32, scores at positions shifted by 100,000 differ from the exact values by 0.016, and by 0.099 at 1,000,000 (for scores around 17 in size); with angles in float64 the error stays at \(10^{-14}\). Our rope computes the angles in float64 and casts only the sines and cosines. Never compute angles in bfloat16, which keeps only 2 to 3 significant decimal digits.
  • Wrong positions with a KV cache or padding. During generation, the new token’s position is the cache length, not 0. With left padding, positions must start at the first real token. A wrong offset breaks nothing visibly, which is why the relative test matters.
  • Forgetting to center before PCA, or reading too much into a 2D plot that keeps 20% of the variance. Check neighbor quality in the full dimension.
  • Using the dot product instead of cosine for neighbors. Long vectors then win regardless of direction. Normalize first, or say clearly that you want the dot product (as attention does).
  • Looking up a word that is two tokens. “␣Wednesday” is one GPT-2 token, but many words are not. Check the token count, and remember that “king” and “␣king” are different rows.

Interview questions

Write RoPE for a tensor of shape (T, d). What is applied to what?basic
Compute \(\theta_i = b^{-2i/d}\) for \(i = 0..d/2-1\), angles \(m\theta_i\) for each position \(m\), then for each pair \((x_{2i}, x_{2i+1})\) output \((x_{2i}\cos - x_{2i+1}\sin,\ x_{2i}\sin + x_{2i+1}\cos)\). It is applied to the queries and keys of every head, after \(W_Q\) and \(W_K\), before the dot product. Values are not rotated, and nothing is added to the token embeddings. The cost is a few multiplies per element; there are no learned parameters.
Prove that the RoPE attention score depends only on m − n.core
\(R_m\) is block-diagonal with \(2 \times 2\) rotations by \(m\theta_i\). Rotation matrices satisfy \(R_m^{\top} = R_{-m}\) and \(R_a R_b = R_{a+b}\). So \((R_m\mathbf{q})^{\top}(R_n\mathbf{k}) = \mathbf{q}^{\top}R_m^{\top}R_n\mathbf{k} = \mathbf{q}^{\top}R_{n-m}\mathbf{k}\), which contains positions only through \(n - m\). Numerically: place fixed \(\mathbf{q}, \mathbf{k}\) at all positions and check that every diagonal of the score matrix is constant (spread about \(10^{-5}\) in float32).
Does RoPE know whether the key is before or after the query?core
Yes. The score is \(\sum_i A_i\cos\varphi_i + B_i\sin\varphi_i\) with \(\varphi_i = (n-m)\theta_i\). The cosine part is the same for \(\pm\Delta\), but the sine part changes sign, and \(B_i\) (the 2D cross product of the pairs) is usually not zero. In the worked example, distance −3 gives −4.3458 and +3 gives −3.5741. In a causal decoder keys are always at or before the query, so this matters more for encoders and for image patches.
Why do the pairs have different frequencies? What would go wrong with one frequency?core
One frequency \(\theta\) is periodic with wavelength \(2\pi/\theta\): distances \(\Delta\) and \(\Delta + 2\pi/\theta\) would look almost the same, and if \(\theta\) is small, neighbors would look identical. A geometric range from 1 down to about \(1/b\) radians per token gives fast pairs for fine, local order and slow pairs for coarse, long-range distance, like the digits of a clock. The wavelengths run from 6.28 tokens to tens of thousands or millions.
Why did Llama 3 raise the base from 10,000 to 500,000?advanced
A larger base makes the slow pairs slower: for \(d = 128\) the slowest wavelength goes from 54,410 to about 2.56 million tokens, and the average score \(\bar s(\Delta)\) decays more slowly (0.49 instead of 0.16 at \(\Delta = 1{,}000\)). This keeps distances in a long context distinguishable and inside the range of angles the model saw in training, which supports longer contexts (8,192 for Llama 3, then 128K with extra scaling in 3.1). The fast pairs barely change: \(\lambda_0 = 2\pi\) for any base. [Likely: the stated motivation; the numbers are exact.]
What is the “long-term decay” of RoPE, and how strong is it?advanced
For aligned \(\mathbf{q}\) and \(\mathbf{k}\), the score tends to shrink as the distance grows, because the cosines of more and more pairs average out. For \(\mathbf{q} = \mathbf{k} = \mathbf{1}\), \(d = 128\), \(b = 10{,}000\), the normalized score is 0.970, 0.669, 0.477, 0.159 at distances 1, 10, 100, 1,000. It is a soft bias, not a mask: learned \(\mathbf{q}\) and \(\mathbf{k}\) can concentrate their energy in slow pairs and still attend far away.
Your RoPE port gives slightly worse perplexity than the reference, and all unit tests pass. What do you check?advanced
(1) The pair layout (interleaved vs rotate_half) against how the checkpoint was trained. (2) The frequency formula and base from the config, including any rope_scaling. (3) Precision: angles and sin/cos in float32, not bfloat16, especially at large positions. (4) Position ids with KV cache, left padding and packed sequences. (5) Partial rotary: some models rotate only part of each head, e.g. Pythia has rotary_pct 0.25. Compare the rotated \(\mathbf{q}\) and \(\mathbf{k}\) tensors with the reference, layer by layer, on the same input.
How do you find the nearest neighbors of a token in a \(50{,}000 \times 768\) table efficiently?basic
Normalize every row once, then the cosines to all tokens are one matrix-vector product \(\hat E\hat E_w\) (about 38 million multiply-adds for GPT-2), followed by top-\(k\). For many queries, use a matrix-matrix product in batches. For millions of vectors, use an approximate nearest-neighbor index (for example HNSW or IVF), which trades a little recall for much less time.
A 2D PCA plot of embeddings shows two words far apart. Are they dissimilar?core
Not necessarily. PCA keeps the directions with the most variance in this set; for our 63 GPT-2 words the first two keep only 19.7% of it. Points can be close or far in the plot for reasons that are not visible. In our data, the nearest point in the 2D plot is from the same word group for 54 of 63 words, but in 768 dimensions with cosine it is 63 of 63. Always check similarity in the full space.
Why are all cosine similarities in GPT-2’s embedding table positive, and does it matter?core
The rows share a common direction (anisotropy): the average cosine between random tokens is about 0.27. Then every pair looks somewhat similar, and a fixed threshold like 0.5 means different things in different models. Subtracting the mean row brings the average to about 0 and makes scores easier to compare. Within one model, the ranking of neighbors often changes little, as in our examples. In retrieval systems, centering or whitening can help thresholds and duplicate detection.
How does RoPE extend to images and video in a VLM?advanced
Split the pairs into groups and drive each group with one coordinate. For images, one group uses the patch row and one the column, so the score depends on the 2D offset. Qwen2-VL’s M-RoPE uses three groups (time, height, width), sized 16, 24 and 24 pairs for its 128-dimensional heads. Text tokens use the same value for all three coordinates, which reduces to ordinary 1D RoPE. Each group keeps the relative property along its own axis.

Exercises

  1. Repeat Figure 1 with the embedding table of another model (for example Qwen2.5-0.5B, whose input and output tables are tied). Do the same groups appear? What is \(r_1 + r_2\)?
  2. By hand, compute \(\theta_0, \theta_1, \theta_2, \theta_3\) for \(d = 8\) and \(b = 10{,}000\), and the wavelength of each.
  3. In the \(d = 4\) worked example, use formula 15.7 to compute \(s(2, 5)\), where \(n - m = +3\). Check that you get −3.5741. Which pair contributes most, and why?
  4. Change one line of rope to use torch.arange(d // 2) / d as the exponent. Does the relative test (15.8) still pass? What changed?
  5. Write rope_half (the rotate_half layout) yourself, and check that it equals rope after the permutation in Test 5.
  6. Implement 2D RoPE for a \(4 \times 4\) grid of patches with \(d = 16\): pairs 0 to 3 use the row, pairs 4 to 7 the column. Test that the score depends only on (row offset, column offset).
  7. Change rope so that the angles are computed in float32 (pos.float()[:, None] * rope_freqs(d).float()), and compare the scores at positions shifted by 100,000 with the float64 version. Explain the difference.
Answerscheck
1. Depends on the model; numbers, days, months and countries usually form clear groups, and \(r_1 + r_2\) is small (well under 50%). 2. \(\theta_i = 10000^{-i/4}\): 1, 0.1, 0.01, 0.001; wavelengths 6.28, 62.8, 628.3, 6,283.2 tokens. 3. Now \(\varphi_0 = 3\) and \(\varphi_1 = 0.03\). Pair 0: \(4\cos 3 + 3\sin 3 = -3.9600 + 0.4234 = -3.5366\). Pair 1: \(-1.25\sin 0.03 = -0.0375\). Sum: −3.5741. Pair 0 dominates: \(A_1 = 0\), and pair 1 has turned only 0.03 radians, so its sine term is tiny. 4. Yes, the test still passes, because any set of frequencies gives a relative score; only the wavelengths change (they become the ones for base \(\sqrt{b}\)). This is why the relative test alone cannot catch this bug: compare with the reference. 5. Rotate \((x_i, x_{i+d/2})\) by \(m\theta_i\); Test 5 shows the permutation. 6. Use \(\varphi = \Delta_{\text{row}}\theta_i\) for pairs 0 to 3 and \(\Delta_{\text{col}}\theta_i\) for pairs 4 to 7; group all 256 patch pairs by (row offset, col offset) and check that the spread within each group is about \(10^{-6}\). 7. Float32 has about 7 significant digits. An angle like \(100{,}000 \times 1\) radian is stored with an error near 0.004, and that error goes straight into the sines and cosines; in our run the score error was 0.016 in float32 and \(10^{-14}\) in float64.

Key takeaways

  • An embedding table is a \(|V| \times d\) matrix; PCA (15.1) gives a 2D view that keeps little variance (19.7% for our 63 GPT-2 words), so judge similarity in the full space with cosine (15.3).
  • Trained neighbors make sense: “␣Monday” → the other weekdays, “␣king” → “␣kings”, “␣King”, “␣queen”. GPT-2’s rows share a common direction (average cosine 0.27).
  • RoPE rotates each pair of \(\mathbf{q}\) and \(\mathbf{k}\) by \(m\theta_i\) (15.4, 15.5). Because \(R_m^{\top}R_n = R_{n-m}\), the score depends only on \(n - m\) (15.6, 15.7), including its sign.
  • Test it: constant diagonals of the score matrix (15.8), spread about \(10^{-5}\) in float32; absolute sinusoidal positions fail the same test.
  • Wavelengths run from \(2\pi\) to \(2\pi b^{(d-2)/d}\) (15.9): fast pairs see local order, slow pairs see long distances; a larger base slows the decay (15.10) and suits longer contexts.

Key formulas

\[ Z = X_c V_{:,1:2}, \quad \cos(\mathbf{a},\mathbf{b}) = \frac{\mathbf{a}^{\top}\mathbf{b}}{\lVert\mathbf{a}\rVert\lVert\mathbf{b}\rVert}, \quad \theta_i = b^{-2i/d}, \quad (R_m\mathbf{q})^{\top}(R_n\mathbf{k}) = \mathbf{q}^{\top}R_{n-m}\mathbf{k}, \quad \lambda_i = 2\pi b^{2i/d} \]

References for this session

  1. Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
  2. Raschka, Build a Large Language Model (From Scratch), chapter 3: Coding attention mechanisms
  3. Radford et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2)
  4. Vaswani et al. (2017). Attention Is All You Need
  5. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  6. Llama Team, Meta (2024). The Llama 3 Herd of Models
  7. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  8. Black et al. (2022). GPT-NeoX-20B: An Open-Source Autoregressive Language Model
  9. Chen et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation
  10. Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models
  11. Ethayarajh (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings