embeddings and RoPE by hand
In this lab you open GPT-2’s real embedding table, project 63 words to 2D with PCA and find their nearest neighbors among all 50,257 tokens. Then you write RoPE in about 15 lines of PyTorch, prove with numbers that the attention score depends only on the distance between two tokens, and plot the rotation frequencies of Llama and Qwen to see which dimensions measure short and long distances.
What you will learn
- How to load a trained embedding table and check that a word is a single token
- PCA with an SVD, the explained variance ratio, and why a 2D plot keeps only a small part of the picture
- Nearest neighbors with cosine similarity over a full vocabulary, and the effect of centering (anisotropy)
- How to implement RoPE as a rotation of pairs, in both the interleaved and the rotate_half layout
- How to test with numbers that RoPE scores depend only on m − n, and why absolute sinusoidal positions fail the same test
- RoPE frequencies and wavelengths for the real bases of Llama 2, Llama 3 and Qwen2.5, and which pairs handle short and long distances
- Common RoPE bugs: wrong layout, wrong exponent, low-precision angles, wrong positions with a cache
Why this matters
The first two things a transformer does with a token are: look up its vector in the embedding table, and tell attention where the token is. In modern LLMs the second step is usually RoPE (rotary position embedding). In this lab you open a real trained embedding table, look at its geometry, and then write RoPE yourself and prove with numbers that it does what the papers say.
Session 11 (session 11) explained that the embedding table is a matrix with one row per token, and that training puts similar tokens close together. Session 14 (session 14) explained RoPE: rotate pairs of numbers by an angle that grows with the position. Here you check both claims yourself, with real weights and real code. You will:
- Look at trained embeddings. Load GPT-2’s table (50,257 rows of 768 numbers), project 63 words to 2D with PCA, and find nearest neighbors in the full vocabulary.
- Implement RoPE in about 15 lines of PyTorch, and test with numbers that the attention score depends only on the distance between two tokens, not on where they are.
- Plot the RoPE frequencies and see which pairs of dimensions measure short distances and which measure long ones, for the real settings of Llama 2, Llama 3 and Qwen2.5.
Why does this matter for real work? Nearest-neighbor search over embeddings is the core of semantic search and retrieval (RAG). RoPE is in Llama, Qwen, Mistral and most open LLMs, and its 2D and 3D versions are how Qwen2-VL tells image patches and video frames apart. Bugs in RoPE are common and silent: the model still runs, it just gets worse. The tests you write in this lab are the tests you would run when you port a model or extend its context window.
Setup
You need Python 3.9 or newer, PyTorch and Hugging Face transformers. The first run downloads GPT-2 small (about 550 MB). Everything else is small and runs on a laptop CPU in seconds.
pip install torch transformers
All code on this page is deterministic. The trained weights are fixed, PCA and cosine similarity have no randomness, and the random test vectors use torch.manual_seed(0). The numbers in the code comments are what the code printed when we ran it (PyTorch 2.8, transformers 4.57). The figures compute their numbers live in your browser from the same data.
Part A, step 1: open the embedding table
GPT-2 small has a token embedding table wte of shape \(50{,}257 \times 768\): one row of \(d = 768\) numbers for each token in its vocabulary. [Certain: the model’s config.json has vocab_size 50257 and n_embd 768.] The average row length (L2 norm) is 3.96. GPT-2 also has a second table, wpe, of shape \(1{,}024 \times 768\): one learned vector per position. GPT-2 does not use RoPE; we come back to wpe in Part B.
We pick 63 common English words in 8 groups: numbers, days, months, colors, animals, countries, people and verbs. We write each one with a leading space (“␣king”, not “king”), because that is how a word looks in the middle of a sentence for a GPT-2 tokenizer (session 10). The code checks that each word is exactly one token; if it were two tokens, there would be no single row to look at.
Part A, step 2: PCA to two dimensions
PCA (principal component analysis) finds the directions in which the data varies the most. We stack our 63 vectors as rows of a matrix \(X\), subtract the mean row, and take the singular value decomposition (SVD):
- Read the symbols
- \(X\)
- the \(63 \times 768\) matrix of embeddings, one word per row
- \(\mu\), \(\mathbf{1}\)
- the mean of the rows (a 768-vector), and a column of ones, so \(\mathbf{1}\mu^{\top}\) subtracts the mean from every row
- \(U, \Sigma, V\)
- the SVD: \(V\)’s columns are the principal directions, \(\Sigma\) holds the singular values \(\sigma_1 \ge \sigma_2 \ge \dots\)
- \(Z\)
- the \(63 \times 2\) coordinates we plot: each word projected on the first two directions
How much of the picture do we keep? Each direction \(j\) carries a share of the total variance:
For our 63 words, \(r_1 = 11.2\%\), \(r_2 = 8.5\%\) and \(r_3 = 6.9\%\). So the 2D plot keeps only 19.7% of the variance. That is normal for embeddings: meaning is spread over many directions. Keep this in mind when you read any 2D plot of embeddings.
How to read Figure 1
On PC1 vs PC2, time words (days and months) sit on the right, and countries at the top: these two groups explain the most variance in this set. Colors and numbers are clearer on PC3. The readout counts, for every word, whether its nearest point in the 2D picture is from the same group: 54 of 63 on PC1 vs PC2. In the full 768 dimensions, with cosine, it is 63 of 63. The picture loses information; the vectors do not. Click “␣king”: its vocabulary neighbors are “␣kings”, “␣King”, “␣queen”, “King”, “␣prince”, “␣monarch”. Switch to “Centered vocabulary” to see the same lists after subtracting the mean of all 50,257 rows.
Part A, step 3: nearest neighbors
To find the tokens most similar to token \(w\), we use cosine similarity: the dot product of the two vectors after scaling both to length 1. It ignores length and looks only at direction:
- Read the symbols
- \(E_w\)
- row \(w\) of the embedding table, the vector of token \(w\)
- \(\lVert \mathbf{a} \rVert\)
- the length (L2 norm) of \(\mathbf{a}\)
- \(\mathrm{NN}_k(w)\)
- the \(k\) tokens \(j\) with the highest cosine to \(w\), leaving out \(w\) itself
In practice you normalize the whole table once, \(\hat E = E / \lVert E \rVert_{\text{row}}\), and then one matrix-vector product \(\hat E \hat E_w\) gives the cosine to all 50,257 tokens at once. Three things stand out in the results:
- Neighbors are often the same word in another form. “␣dog” → “␣dogs” (0.796), “␣Dog” (0.734), “Dog” (0.672), “␣canine” (0.652). The tokenizer made separate tokens for case, plural and leading space; the embedding learned that they mean almost the same thing.
- Groups are clean. The 6 nearest tokens of “␣Monday” are the other 6 weekdays, with cosines from 0.89 down to 0.80. For “␣three” they are four, two, five, six, seven, nine.
- All cosines are high. The average cosine between 2,000 random GPT-2 tokens is 0.27, not 0. The vectors share a common direction: this is the anisotropy from session 11. Subtracting the mean of all rows (“centering”) brings that average to almost exactly 0 and lowers every score, but in our examples the order of the neighbors barely changes.
- input: table \(E \in \R^{|V| \times d}\), tokenizer, word list, \(k\)
- ids \(\leftarrow\) [ tokenize(" " + w) for w in words ]; assert each has length 1
- \(X \leftarrow E[\text{ids}]\); \(X_c \leftarrow X - \text{mean}(X)\) // center, or PC1 is just the mean
- \(U, \Sigma, V^{\top} \leftarrow \text{svd}(X_c)\); \(Z \leftarrow X_c V_{:,1:2}\); report \(r_1 + r_2\) // (15.1), (15.2)
- plot \(Z\), colored by group
- \(\hat E \leftarrow E\) with every row divided by its norm
- for w in words do
- \(s \leftarrow \hat E\, \hat E_w\); \(s_w \leftarrow -\infty\) // (15.3), skip the word itself
- print top-\(k\) of \(s\), decoded to text
Here is the code. It is the reference solution: try to write it yourself first.
import torch from transformers import AutoTokenizer, GPT2Model tok = AutoTokenizer.from_pretrained("gpt2") model = GPT2Model.from_pretrained("gpt2") W = model.wte.weight.detach() # (50257, 768) token embedding table print(W.shape, W.norm(dim=1).mean().item()) # torch.Size([50257, 768]) ~3.96 words = ["one", "two", "three", "four", "five", "six", "seven", "eight", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday", "January", "February", "March", "April", "May", "June", "July", "August", "red", "blue", "green", "yellow", "black", "white", "purple", "orange", "dog", "cat", "horse", "cow", "lion", "tiger", "bird", "fish", "France", "Germany", "Italy", "Spain", "China", "Japan", "India", "Russia", "king", "queen", "man", "woman", "boy", "girl", "mother", "father", "run", "walk", "eat", "drink", "sleep", "write", "read", "sing"] ids = [] for w in words: t = tok.encode(" " + w) # leading space: the form used inside a sentence assert len(t) == 1, (w, t) # every word must be ONE token ids.append(t[0]) X = W[ids] # (63, 768) # PCA by SVD of the centered matrix Xc = X - X.mean(dim=0) U, S, Vh = torch.linalg.svd(Xc, full_matrices=False) var = S**2 / (S**2).sum() print(var[:3]) # tensor([0.1117, 0.0853, 0.0694]) Z = Xc @ Vh[:2].T # (63, 2) coordinates to plot # nearest neighbors by cosine over the WHOLE vocabulary Wn = W / W.norm(dim=1, keepdim=True) def neighbors(word, k=6): i = tok.encode(" " + word)[0] s = Wn @ Wn[i] s[i] = -1.0 # do not return the word itself v, j = s.topk(k) return [(tok.decode([int(t)]), round(float(c), 3)) for c, t in zip(v, j)] print(neighbors("king")) # [(' kings', 0.733), (' King', 0.696), (' queen', 0.657), ('King', 0.604), (' prince', 0.59), (' monarch', 0.581)] print(neighbors("Monday")) # [(' Tuesday', 0.89), (' Thursday', 0.845), (' Wednesday', 0.844), (' Friday', 0.821), (' Sunday', 0.806), (' Saturday', 0.801)] # anisotropy check: average cosine between random tokens g = torch.Generator().manual_seed(0) r = torch.randint(0, W.shape[0], (2000,), generator=g) C = Wn[r] @ Wn[r].T off = ~torch.eye(2000, dtype=torch.bool) print(C[off].mean().item()) # ~0.27, far from 0
Part B, step 4: RoPE is a rotation of pairs
GPT-2 adds a learned position vector to each token vector. Its 1,024 learned position vectors are very regular: the first 3 principal directions hold 90.4% of their variance (33.4%, 32.3% and 24.6%), against 26.6% for the first 3 directions of our 63 words. The model learned smooth, wave-like patterns over position by itself. RoPE builds such waves in by design, and it puts them inside attention instead of adding them to the input.
RoPE acts on the query and key vectors of one attention head, after their projections and before the dot product (session 16, session 16, explains queries and keys; for now, \(\mathbf{q}\) and \(\mathbf{k}\) are just two vectors of length \(d\) whose dot product is the attention score). Split a vector \(\mathbf{x} \in \R^d\) into \(d/2\) pairs. Pair \(i\) is treated as a point in a plane and rotated by the angle \(m\theta_i\), where \(m\) is the token’s position:
- Read the symbols
- \(x_{2i}, x_{2i+1}\)
- the two numbers of pair \(i\) before rotation (indices start at 0)
- \(y_{2i}, y_{2i+1}\)
- the same pair after rotation
- \(m\)
- the position of the token in the sequence: 0, 1, 2, …
- \(\theta_i\)
- the frequency of pair \(i\): how many radians it turns per position
- \(b\)
- the base: 10,000 in the RoFormer paper and Llama 2, 500,000 in Llama 3, 1,000,000 in Qwen2.5
- \(d\)
- the head dimension (64 or 128 in the models in this lab)
We write \(R_m\) for the whole \(d \times d\) matrix: a block-diagonal matrix with one \(2 \times 2\) rotation per pair. Two facts follow at once. A rotation keeps lengths, so \(\lVert R_m \mathbf{x} \rVert = \lVert \mathbf{x} \rVert\). And rotations add: \(R_a^{\top} R_b = R_{b-a}\), because turning back by \(a\) and forward by \(b\) is the same as turning by \(b - a\).
Part B, step 5: why the score depends only on m − n
Put the query at position \(m\) and the key at position \(n\). Using \(R_m^{\top} R_n = R_{n-m}\):
The positions appear only as the difference \(n - m\). Written out pair by pair, with \(\varphi_i = (n-m)\theta_i\):
- Read the symbols
- \(s(m,n)\)
- the raw attention score between a query at position \(m\) and a key at position \(n\) (before scaling and softmax)
- \(R_{n-m}\)
- one rotation matrix that depends only on the distance
- \(A_i\)
- the ordinary dot product of pair \(i\) of \(\mathbf{q}\) and \(\mathbf{k}\): the score of that pair when the distance is 0
- \(B_i\)
- the 2D cross product of the two pairs; it makes the score different for \(n - m\) and \(m - n\)
- \(\varphi_i\)
- the angle between the two rotations for pair \(i\)
Two pairs, so \(\theta_0 = 10000^{0} = 1\) and \(\theta_1 = 10000^{-2/4} = 0.01\). Take \(\mathbf{q} = (1, 2, 0.5, -1)\) and \(\mathbf{k} = (2, 1, 1, 0.5)\). Without rotation, \(\mathbf{q}^{\top}\mathbf{k} = 2 + 2 + 0.5 - 0.5 = 4\).
Query at 5, key at 2. Rotating gives \(R_5\mathbf{q} = (2.2015, -0.3916, 0.5494, -0.9738)\) and \(R_2\mathbf{k} = (-1.7416, 1.4024, 0.9898, 0.5199)\). Their dot product is −4.3458.
Check with formula 15.7. Here \(n - m = -3\). Pair 0: \(A_0 = 1\cdot2 + 2\cdot1 = 4\), \(B_0 = 2\cdot2 - 1\cdot1 = 3\), \(\varphi_0 = -3\), so \(4\cos(-3) + 3\sin(-3) = -3.9600 - 0.4234 = -4.3833\). Pair 1: \(A_1 = 0.5 - 0.5 = 0\), \(B_1 = -1\cdot1 - 0.5\cdot0.5 = -1.25\), \(\varphi_1 = -0.03\), so \(-1.25\sin(-0.03) = 0.0375\). Sum: \(-4.3833 + 0.0375 = -4.3458\). ✓
Move both. Positions (13, 10) and (103, 100) give exactly −4.3458 again: the rotated vectors are completely different, the score is the same. Positions (2, 5), distance +3 instead of −3, give −3.5741: RoPE knows the direction of the distance, because of the \(B_i \sin\varphi_i\) terms. Positions (3, 3) give 4.0, the plain dot product.
How to read Figure 2
Press “Move both by +10” a few times. The arrows spin, fast in pair 0 and slowly in pair 1, but the angle between the blue and orange arrow in each circle stays the same, and so do the per-pair scores and the total −4.3458. Now move only one slider: the score changes. Set \(m = n\): the arrows in each circle line up as they were before rotation and the score is the plain \(\mathbf{q}^{\top}\mathbf{k} = 4\).
Part B, step 6: implement it and test it with numbers
Formula 15.6 is a proof. Code can still be wrong: wrong pairs, wrong sign, frequencies off by a factor of 2. So we turn the proof into a test. Take one random \(\mathbf{q}\) and one random \(\mathbf{k}\), place copies of them at every position \(0..T-1\), and compute the score matrix \(S_{mn} = s(m,n)\). If the scores depend only on \(m - n\), every diagonal of \(S\) is constant. We measure the worst violation:
For a correct RoPE this is zero up to rounding. With \(d = 64\), \(T = 32\) and float32, our implementation gives a spread of \(1.1 \times 10^{-5}\), for scores between −17.0 and −2.9. The same test with absolute sinusoidal position vectors added to \(\mathbf{q}\) and \(\mathbf{k}\) (the original Transformer method, session 13) gives a spread of 18.7: the score depends on where the tokens are, not only on their distance.
- function rope(\(X \in \R^{T \times d}\), pos, \(b\)):
- \(\theta_i \leftarrow b^{-2i/d}\) for \(i = 0..d/2-1\) // (15.5), compute in float64
- angle\([t, i] \leftarrow \text{pos}[t] \cdot \theta_i\); \(c, s \leftarrow \cos(\text{angle}), \sin(\text{angle})\)
- \(x_1 \leftarrow X[:, 0::2]\); \(x_2 \leftarrow X[:, 1::2]\) // the two halves of each pair
- return interleave(\(x_1 c - x_2 s,\ x_1 s + x_2 c\)) // (15.4)
- test: \(\mathbf{q}, \mathbf{k} \sim \mathcal{N}(0, I)\) with a fixed seed; \(Q \leftarrow\) rope(repeat(\(\mathbf{q}, T\)), \(0..T-1\)); same for \(K\)
- assert all row norms of \(Q\) equal \(\lVert \mathbf{q} \rVert\) // rotations keep length
- \(S \leftarrow Q K^{\top}\); assert spread\((S) \lt 10^{-4}\) // (15.8)
- assert \(S_{5,2}\) equals the hand value \(-4.3458\) for the \(d = 4\) example
How to read Figure 3
With RoPE, the colors run in stripes parallel to the main diagonal: every diagonal is one color, and the readout shows a spread (formula 15.8) of about \(10^{-15}\), which is rounding error in JavaScript’s 64-bit numbers. Switch to “Absolute sinusoidal”: the stripes break, and the spread jumps to several units. “No position” gives one flat color: the model could not tell positions apart at all. Move the base slider: with a small base, all pairs turn fast and the pattern repeats quickly; with a large base, scores change slowly with distance.
Here is the reference code. Test 5 compares it with the Hugging Face Llama implementation.
import math, torch torch.manual_seed(0) def rope_freqs(d, base=10000.0): i = torch.arange(0, d, 2, dtype=torch.float64) # 0, 2, 4, ..., d-2 return base ** (-i / d) # theta_0 .. theta_{d/2-1} def rope(x, pos, base=10000.0): """x: (T, d) with d even; pos: (T,) integer positions. Interleaved pairs (2i, 2i+1).""" d = x.shape[-1] ang = pos[:, None].double() * rope_freqs(d, base)[None, :] # (T, d/2) c, s = ang.cos().to(x.dtype), ang.sin().to(x.dtype) x1, x2 = x[..., 0::2], x[..., 1::2] # first and second of each pair y1 = x1 * c - x2 * s y2 = x1 * s + x2 * c return torch.stack([y1, y2], dim=-1).flatten(-2) # put the pairs back in place # Test 1: rotation keeps the length of every vector d, T = 64, 32 q, k = torch.randn(d), torch.randn(d) pos = torch.arange(T) Q = rope(q.expand(T, d), pos) K = rope(k.expand(T, d), pos) print(q.norm().item(), Q.norm(dim=1).min().item(), Q.norm(dim=1).max().item()) # 8.3705 for all three # Test 2: score(m, n) depends only on m - n (constant along every diagonal) S = Q @ K.T # S[m, n] = <R_m q, R_n k> spread = max((S.diagonal(o).max() - S.diagonal(o).min()).item() for o in range(-T + 1, T)) print("max spread along diagonals:", spread) # ~1e-5 in float32 # Test 3: the same test FAILS for absolute sinusoidal embeddings added to x pe = torch.zeros(T, d) p = torch.arange(T, dtype=torch.float32)[:, None] div = 10000 ** (torch.arange(0, d, 2, dtype=torch.float32) / d) pe[:, 0::2], pe[:, 1::2] = torch.sin(p / div), torch.cos(p / div) S_abs = (q + pe) @ (k + pe).T print("absolute PE spread:", max((S_abs.diagonal(o).max() - S_abs.diagonal(o).min()).item() for o in range(-T + 1, T))) # ~18.69: depends on m and n, not only m - n # Test 4: the worked example from the text q4 = torch.tensor([1.0, 2.0, 0.5, -1.0], dtype=torch.float64) k4 = torch.tensor([2.0, 1.0, 1.0, 0.5], dtype=torch.float64) for m, n in [(5, 2), (13, 10), (103, 100), (2, 5), (3, 3)]: s = rope(q4[None], torch.tensor([m]))[0] @ rope(k4[None], torch.tensor([n]))[0] print(m, n, round(s.item(), 4)) # -4.3458 three times, then -3.5741, 4.0 # Test 5: Hugging Face Llama uses "rotate_half": pairs are (i, i + d/2), not (2i, 2i+1) from transformers.models.llama.modeling_llama import apply_rotary_pos_emb ang = pos[:, None].double() * rope_freqs(d)[None, :] emb = torch.cat([ang, ang], dim=-1) # (T, d) as in HF cos, sin = emb.cos().float()[None, None], emb.sin().float()[None, None] qh, kh = apply_rotary_pos_emb(q.expand(T, d)[None, None], k.expand(T, d)[None, None], cos, sin) perm = torch.cat([torch.arange(0, d, 2), torch.arange(1, d, 2)]) # interleaved -> halves print("HF == ours after permutation:", torch.allclose(qh[0, 0], rope(q[perm.argsort()].expand(T, d), pos)[:, perm], atol=1e-5)) # True # Wavelengths: how many tokens before pair i turns a full circle for base in (10000.0, 500000.0): lam = 2 * math.pi / rope_freqs(128, base) print(base, round(lam[0].item(), 2), round(lam[-1].item()), int((lam > 8192).sum())) # 10000.0 6.28 54410 14 (14 of 64 pairs are slower than one turn per 8192 tokens) # 500000.0 6.28 2559196 29
Two layouts of the same idea
The RoFormer paper pairs neighbors, \((x_{2i}, x_{2i+1})\), as in formula 15.4. The GPT-NeoX and Hugging Face Llama code pairs \(x_i\) with \(x_{i + d/2}\) instead, using a helper called rotate_half. [Certain: the HF function rotates the two halves of the vector.] Both are correct RoPE: one is the other with the dimensions reordered, which Test 5 checks. But the trained weights \(W_Q, W_K\) must match the layout. Meta’s original Llama checkpoints are converted to the Hugging Face layout by permuting rows of \(W_Q\) and \(W_K\). [Likely] If you load weights from one layout into code of the other, nothing crashes; the model simply gets much worse.
Part C, step 7: frequencies and wavelengths
The frequencies \(\theta_i = b^{-2i/d}\) form a geometric sequence: each pair turns slower than the previous one by the same factor \(b^{-2/d}\). The wavelength of pair \(i\) is the number of positions it needs for one full turn:
- Read the symbols
- \(\lambda_i\)
- the wavelength of pair \(i\), in tokens
- \(\lambda_0\)
- the fastest pair: a full turn every 6.28 tokens, for any base
- \(\lambda_{d/2-1}\)
- the slowest pair: 54,410 tokens for \(d = 128\), \(b = 10{,}000\); 2,559,196 tokens for \(b = 500{,}000\)
This gives each pair a job. A fast pair (small \(i\)) changes a lot between neighbors, so it can tell distance 3 from distance 4. But after about 6 tokens it has turned a full circle, so it cannot tell distance 1 from distance 7. A slow pair (large \(i\)) barely moves between neighbors, but its angle grows steadily over thousands of tokens, so it can tell “a few hundred tokens back” from “many thousands back”. Together the pairs act like the digits of a number: seconds, minutes, hours, days.
A useful summary is the score when \(\mathbf{q} = \mathbf{k} = (1, \dots, 1)\). Then \(A_i = 2\) and \(B_i = 0\) in formula 15.7, and dividing by the distance-0 score \(d\) gives the average cosine:
For \(d = 128\) and \(b = 10{,}000\): \(\bar s = 0.970\) at \(\Delta = 1\), 0.669 at 10, 0.477 at 100, 0.159 at 1,000 and −0.028 at 10,000. With \(b = 500{,}000\) the fall is slower: 0.978, 0.764, 0.611, 0.492 and 0.187. As \(\Delta\) grows, more and more pairs have turned many times and their cosines average out, so similar vectors far apart get a smaller score. The RoFormer paper calls this long-term decay. [Certain: our numbers; the paper proves an upper bound that decays.] A larger base keeps more pairs slow, so the decay stretches over a longer context.
How to read Figure 4
Start with Llama 2 7B (\(b = 10{,}000\), \(d = 128\), context 4,096). The wavelengths go from 6.28 up to 54,410 tokens; 18 of the 64 pairs lie above the 4,096 line. Move \(\Delta\) from 1 to 10,000. At \(\Delta = 1\) almost all bars are near +1. At \(\Delta = 100\) the left bars (fast pairs) jump between −1 and +1 with no pattern, while the right bars (slow pairs) are still near +1: only the slow pairs still give a smooth signal about distance. Switch to Llama 3 8B (\(b = 500{,}000\)): the curve is steeper, and 29 pairs lie above its 8,192 context line.
The table gives the numbers for real models, read from their config.json files on Hugging Face. [Certain: config values; Llama 2’s config has no rope_theta, so the library default of 10,000 is used.]
| Model | Base \(b\) | Head dim \(d\) | Training context | Slowest \(\lambda\) | Pairs with \(\lambda \gt\) context |
|---|---|---|---|---|---|
| Llama 2 7B | 10,000 | 128 | 4,096 | 54,410 | 18 of 64 |
| Llama 3 8B | 500,000 | 128 | 8,192 | 2,559,196 | 29 of 64 |
| Qwen2.5-7B | 1,000,000 | 128 | 32,768 | 5,063,256 | 24 of 64 |
| Qwen2.5-0.5B | 1,000,000 | 64 | 32,768 | 4,080,185 | 12 of 32 |
| Llama 3.2 1B | 500,000 | 64 | 8,192 (before scaling) | 2,084,765 | 14 of 32 |
Pairs whose wavelength is longer than the training context never see a full turn during training. When you run the model on a longer context, these pairs reach angles they never saw. This is why context extension methods change mainly the slow pairs: Position Interpolation divides all angles by a factor, YaRN treats fast and slow pairs differently, and Llama 3.1’s config scales the frequencies with rope_type: "llama3", factor 8 and an original context of 8,192. [Certain: config values; the methods are covered in the long-context sessions.]
A first look at 2D and 3D
For an image, a patch has a row and a column, not one position. 2D RoPE splits the pairs into groups: one group rotates with the row, the other with the column. Then the score depends only on the row offset and the column offset. Qwen2-VL’s M-RoPE does the same with three groups for (time, height, width). Its config sets mrope_section to [16, 24, 24]: 16 pairs for time, 24 for height and 24 for width, which is exactly the 64 pairs of its 128-dimensional heads. [Certain: config.json of Qwen2-VL-2B-Instruct.] Exercise 6 asks you to extend your rope function this way.
Where this is used
| System or task | What from this lab it uses |
|---|---|
| Semantic search and RAG | Normalize vectors, one matrix product, top-\(k\) (formula 15.3); at scale an approximate nearest-neighbor index does the same job |
| Model debugging and interpretability | PCA plots and neighbor lists of embedding tables, to find duplicate tokens, untrained rows, or odd clusters |
| Llama 2 and 3, Qwen2.5, Mistral and most open LLMs | RoPE on queries and keys in every attention layer, with the base as a design choice (table above) |
| Long-context extension | Rescaling the slow pairs (Position Interpolation, YaRN, Llama 3.1 scaling); the wavelength plot shows which pairs are affected |
| Qwen2-VL images and video | M-RoPE: separate pair groups for time, height and width, so a patch’s score depends on its offsets in all three |
| Video retrieval and Video-RAG | Clip or frame embeddings searched with cosine nearest neighbors, exactly as Algorithm 1 does with tokens |
| Video fingerprinting and near-duplicate detection | Nearest neighbors with a cosine threshold; centering the embeddings first reduces false matches from the shared direction [Likely] |
| Key-moment detection and action recognition in long videos | The temporal part of M-RoPE gives relative time between frames; the wavelength analysis tells how far apart in time two frames can still be told apart [Speculation] |
Common mistakes
- Mixing the two pair layouts. Interleaved \((2i, 2i+1)\) code with weights trained for the half-split layout \((i, i+d/2)\), or the reverse. Every test on random vectors still passes; only the real model gets worse. Compare against the reference implementation of the checkpoint (Test 5).
- Rotating the values, or rotating after the dot product. RoPE is applied to \(\mathbf{q}\) and \(\mathbf{k}\) only, after their projections and before \(\mathbf{q}^{\top}\mathbf{k}\). The values \(\mathbf{v}\) are not rotated.
- Wrong frequency exponent. \(\theta_i = b^{-2i/d}\) with \(i = 0..d/2-1\), or equivalently
arange(0, d, 2) / d. Writingarange(d/2) / dhalves every exponent and silently changes all wavelengths. - Computing angles in low precision at large positions. In our test, with the angles \(m\theta_i\) computed in float32, scores at positions shifted by 100,000 differ from the exact values by 0.016, and by 0.099 at 1,000,000 (for scores around 17 in size); with angles in float64 the error stays at \(10^{-14}\). Our
ropecomputes the angles in float64 and casts only the sines and cosines. Never compute angles in bfloat16, which keeps only 2 to 3 significant decimal digits. - Wrong positions with a KV cache or padding. During generation, the new token’s position is the cache length, not 0. With left padding, positions must start at the first real token. A wrong offset breaks nothing visibly, which is why the relative test matters.
- Forgetting to center before PCA, or reading too much into a 2D plot that keeps 20% of the variance. Check neighbor quality in the full dimension.
- Using the dot product instead of cosine for neighbors. Long vectors then win regardless of direction. Normalize first, or say clearly that you want the dot product (as attention does).
- Looking up a word that is two tokens. “␣Wednesday” is one GPT-2 token, but many words are not. Check the token count, and remember that “king” and “␣king” are different rows.
Interview questions
Write RoPE for a tensor of shape (T, d). What is applied to what?basic
Prove that the RoPE attention score depends only on m − n.core
Does RoPE know whether the key is before or after the query?core
Why do the pairs have different frequencies? What would go wrong with one frequency?core
Why did Llama 3 raise the base from 10,000 to 500,000?advanced
What is the “long-term decay” of RoPE, and how strong is it?advanced
Your RoPE port gives slightly worse perplexity than the reference, and all unit tests pass. What do you check?advanced
rope_scaling. (3) Precision: angles and sin/cos in float32, not bfloat16, especially at large positions. (4) Position ids with KV cache, left padding and packed sequences. (5) Partial rotary: some models rotate only part of each head, e.g. Pythia has rotary_pct 0.25. Compare the rotated \(\mathbf{q}\) and \(\mathbf{k}\) tensors with the reference, layer by layer, on the same input.How do you find the nearest neighbors of a token in a \(50{,}000 \times 768\) table efficiently?basic
A 2D PCA plot of embeddings shows two words far apart. Are they dissimilar?core
Why are all cosine similarities in GPT-2’s embedding table positive, and does it matter?core
How does RoPE extend to images and video in a VLM?advanced
Exercises
- Repeat Figure 1 with the embedding table of another model (for example Qwen2.5-0.5B, whose input and output tables are tied). Do the same groups appear? What is \(r_1 + r_2\)?
- By hand, compute \(\theta_0, \theta_1, \theta_2, \theta_3\) for \(d = 8\) and \(b = 10{,}000\), and the wavelength of each.
- In the \(d = 4\) worked example, use formula 15.7 to compute \(s(2, 5)\), where \(n - m = +3\). Check that you get −3.5741. Which pair contributes most, and why?
- Change one line of
ropeto usetorch.arange(d // 2) / das the exponent. Does the relative test (15.8) still pass? What changed? - Write
rope_half(the rotate_half layout) yourself, and check that it equalsropeafter the permutation in Test 5. - Implement 2D RoPE for a \(4 \times 4\) grid of patches with \(d = 16\): pairs 0 to 3 use the row, pairs 4 to 7 the column. Test that the score depends only on (row offset, column offset).
- Change
ropeso that the angles are computed in float32 (pos.float()[:, None] * rope_freqs(d).float()), and compare the scores at positions shifted by 100,000 with the float64 version. Explain the difference.
Answerscheck
Key takeaways
- An embedding table is a \(|V| \times d\) matrix; PCA (15.1) gives a 2D view that keeps little variance (19.7% for our 63 GPT-2 words), so judge similarity in the full space with cosine (15.3).
- Trained neighbors make sense: “␣Monday” → the other weekdays, “␣king” → “␣kings”, “␣King”, “␣queen”. GPT-2’s rows share a common direction (average cosine 0.27).
- RoPE rotates each pair of \(\mathbf{q}\) and \(\mathbf{k}\) by \(m\theta_i\) (15.4, 15.5). Because \(R_m^{\top}R_n = R_{n-m}\), the score depends only on \(n - m\) (15.6, 15.7), including its sign.
- Test it: constant diagonals of the score matrix (15.8), spread about \(10^{-5}\) in float32; absolute sinusoidal positions fail the same test.
- Wavelengths run from \(2\pi\) to \(2\pi b^{(d-2)/d}\) (15.9): fast pairs see local order, slow pairs see long distances; a larger base slows the decay (15.10) and suits longer contexts.
Key formulas
References for this session
- Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
- Raschka, Build a Large Language Model (From Scratch), chapter 3: Coding attention mechanisms
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2)
- Vaswani et al. (2017). Attention Is All You Need
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Black et al. (2022). GPT-NeoX-20B: An Open-Source Autoregressive Language Model
- Chen et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation
- Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models
- Ethayarajh (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings