Position, part 2: RoPE in depth, and a first look at 2D and 3D RoPE
Rotary position embedding (RoPE) does not add a position vector to the token. It rotates the query and the key by an angle that grows with the position, so the attention score between two tokens depends only on how far apart they are. This session builds RoPE from one 2D rotation to a full head, explains the frequencies and the base, and then shows how the same trick gives positions to image patches (2D RoPE) and to video tokens (M-RoPE in Qwen2-VL).
What you will learn
- Why rotating the query and the key, instead of adding a position vector, makes the score depend only on the distance m − n
- The 2D rotation, the complex-number view, and the block-diagonal rotation for a full head
- Frequencies θ_i = b^(−2i/d): which dimension pairs see short distances and which see long ones, and what the base b changes
- How RoPE is implemented in practice (cos/sin cache, rotate_half) and the interleaved vs half-split layout bug
- Why RoPE does not extrapolate by itself past the training length, and where context extension starts
- 2D RoPE for image patches: half the dimensions see the row, half see the column
- M-RoPE for video in Qwen2-VL: (time, height, width) position IDs, and how text, images and video share one sequence
Why this matters
Almost every open LLM you will meet in practice, Llama, Qwen, Mistral, Gemma, DeepSeek, uses the same method to tell attention where each token is: rotary position embedding, or RoPE. It was introduced by Su et al. in the RoFormer paper (2021). [Certain] Most VLMs use it too, and the Qwen-VL family stretches it to images and video.
In session 13 we saw the problem. Self-attention compares every query with every key, and that comparison does not care about order: without help, “dog bites man” and “man bites dog” look the same. We saw two kinds of fixes. Absolute methods add a position vector to each token (sinusoidal or learned). Relative methods change the attention score with a term that depends on the distance between the two tokens (T5 buckets, ALiBi). RoPE gets the best of both:
- It is applied like an absolute method. Each query and each key is changed using only its own position \(m\). Nothing about pairs is stored, so it works with a KV cache and fast attention kernels without any change to them.
- It behaves like a relative method. After the change, the score between a query at position \(m\) and a key at position \(n\) depends only on the distance \(m - n\), not on \(m\) and \(n\) themselves.
- It costs almost nothing. No learned parameters, a few multiplications per number, and it never touches the values.
The trick is a rotation. This session builds it from one pair of numbers up to a full attention head, explains the frequencies and the “base” (why Llama 3 uses 500,000 instead of 10,000), and then takes a first look at 2D RoPE for image patches and M-RoPE for video, which we meet again in the vision and video tracks.
Interviewers ask about RoPE often, because the answer shows whether you understand attention at the level of dot products: “Prove that RoPE gives relative positions”, “Why rotate the query and the key but not the value?”, “What does the base do?”, “Why does a model break past its training length?”, “How would you give positions to image patches?”. By the end of this session you can answer each one with a formula and a small example.
The core idea: rotate, do not add
What goes wrong with adding
The original transformer adds a position vector \(\mathbf{p}_m\) to the token embedding \(\mathbf{x}_m\). The query and key are then \(W_q(\mathbf{x}_m + \mathbf{p}_m)\) and \(W_k(\mathbf{x}_n + \mathbf{p}_n)\). If you expand their dot product, you get four terms: content with content, content with position, position with content, and position with position. Only some parts of the last term depend on \(m - n\) in a clean way, and the cross terms mix “what” and “where” in a way the model has to untangle. We would like a change that gives a score of the form “content of the query, content of the key, and their distance”, and nothing else.
One pair of numbers
Start with the smallest case: a query and a key with only two numbers each, \(\mathbf{q}, \mathbf{k} \in \R^2\). Think of each as an arrow in the plane. A rotation by angle \(\varphi\) turns an arrow around the origin without changing its length. In matrix form:
RoPE picks a fixed angle step \(\theta\) and rotates the query of the token at position \(m\) by \(m\theta\), and the key of the token at position \(n\) by \(n\theta\):
- Read the symbols
- \(\mathbf{x}_m\)
- the hidden vector of the token at position \(m\) (no position added to it)
- \(W_q, W_k\)
- the usual learned query and key projections
- \(m, n\)
- positions of the query token and the key token: 0, 1, 2, …
- \(\theta\)
- a fixed angle per position step (in radians). It is not learned.
- \(R(m\theta)\)
- rotation by \(m\) times \(\theta\): the further along the sequence, the more the arrow is turned
Now compute the attention score. Write \(\mathbf{q} = W_q\mathbf{x}_m\) and \(\mathbf{k} = W_k\mathbf{x}_n\) for the vectors before rotation. Two facts about rotations do all the work: the transpose of a rotation is the rotation backwards, \(R(\alpha)^\top = R(-\alpha)\), and two rotations add their angles, \(R(\alpha)R(\beta) = R(\alpha + \beta)\). So:
This is the whole idea. The positions \(m\) and \(n\) only appear as the difference \(n - m\). Move both tokens forward by 100 positions and the score does not change. The model still sees the content (through \(\mathbf{q}\) and \(\mathbf{k}\)) and the distance (through the angle), and nothing else. [Certain]
The same thing with complex numbers
A pair of real numbers \((a, b)\) can be written as one complex number \(a + ib\). Rotating by \(\varphi\) is then just multiplying by \(e^{i\varphi} = \cos\varphi + i\sin\varphi\). And the dot product of two 2D vectors is the real part of one times the conjugate of the other. Write \(\tilde q = q_1 + i q_2\) and \(\tilde k = k_1 + i k_2\). Then:
- Read the symbols
- \(\tilde q, \tilde k\)
- the query pair and the key pair, each written as one complex number
- \(\overline{z}\)
- the complex conjugate of \(z\) (flip the sign of the imaginary part)
- \(e^{i\varphi}\)
- a complex number of length 1 at angle \(\varphi\): multiplying by it rotates by \(\varphi\)
- \(\operatorname{Re}[\cdot]\)
- the real part
The RoFormer paper derives RoPE in this complex form, and Meta’s original Llama code implements it with complex multiplication. [Certain] It is the same as (14.3): the score is the length of \(\tilde q\) times the length of \(\tilde k\) times the cosine of (the angle between them plus \((m - n)\theta\)).
Take \(\mathbf{q} = (1, 0)\), \(\mathbf{k} = (0, 1)\) and \(\theta = 0.5\). Before rotation \(\mathbf{q}\cdot\mathbf{k} = 0\). With \(R(\varphi)\mathbf{k} = (-\sin\varphi, \cos\varphi)\), formula (14.3) gives \(\mathbf{q}_m^\top\mathbf{k}_n = -\sin\big((n - m)\theta\big) = \sin\big((m - n)\theta\big)\).
- \(m = 2, n = 1\): \(\sin(0.5) = 0.479\).
- \(m = 5, n = 4\): again distance 1, again \(0.479\).
- \(m = 1, n = 2\): distance \(-1\), so \(\sin(-0.5) = -0.479\). RoPE knows the sign of the distance, not only its size. A query can tell a key one step behind from a key one step ahead.
- \(\mathbf{q}_m \leftarrow R(m\theta)\,\mathbf{q}\) // (14.2)
- \(\mathbf{k}_n \leftarrow R(n\theta)\,\mathbf{k}\) // (14.2)
- \(s \leftarrow \mathbf{q}_m \cdot \mathbf{k}_n\)
- check: \(s = \mathbf{q} \cdot R\big((n - m)\theta\big)\,\mathbf{k}\) // (14.3)
- repeat with \(m \leftarrow m + 1\), \(n \leftarrow n + 1\)
How to read Figure 1
Press Play. In each round the query turns first (line 1). For a moment the two tokens are 4 apart instead of 3, and the score drops from 0.781 to a different value (0.550 in the first round). Then the key turns by the same angle (line 2), the gap between the arrows is back, and the score is 0.781 again (line 3). Line 4 checks the same number with formula (14.3), which never uses \(m\) or \(n\) alone. The arrows swing all the way around the circle, but at the end of every round the score is the same. Use the sliders to pick the start: with \(m = n\) the score is the unrotated \(\mathbf{q}\cdot\mathbf{k} = 0.69\) at every round. Swap them (start \(m = 2\), \(n = 5\)) and the score is −0.281, because the distance changed sign. A larger \(\theta\) makes the arrows turn faster per position.
From one pair to a full head
Many pairs, many speeds
A real attention head has \(d\) numbers per query and key (for Llama 3 8B, \(d = 128\)). [Certain] RoPE cuts the vector into \(d/2\) pairs and rotates each pair in its own plane, with its own angle step \(\theta_i\). The steps follow the same geometric sequence as the sinusoidal positions of the original transformer:
- Read the symbols
- \(i\)
- the index of a pair of dimensions (pair 0 is the first two numbers, and so on)
- \(b\)
- the base, a fixed number: 10,000 in RoFormer and Llama 2, 500,000 in Llama 3, 1,000,000 in Qwen2 and Qwen2-VL [Certain, from the model configs]
- \(\theta_i\)
- the angle (in radians) that pair \(i\) turns per position. Pair 0 has \(\theta_0 = 1\); the last pair turns very slowly.
The full rotation for position \(m\) is a block-diagonal matrix: \(d/2\) small \(2 \times 2\) rotations on the diagonal and zeros everywhere else. Because each block is a rotation, the whole matrix is a rotation too: it keeps lengths (\(\lVert R_m \mathbf{q}\rVert = \lVert\mathbf{q}\rVert\)), and \(R_m^\top R_n = R_{n-m}\) still holds block by block.
So the score is a sum of \(d/2\) terms, one per pair. Each term is a cosine wave in the distance \(m - n\), with its own speed \(\theta_i\) and with an amplitude and phase set by the content (\(\tilde q_i \overline{\tilde k_i}\)). The model learns \(W_q\) and \(W_k\), so it learns which pairs to use and how. A head that wants to look at the previous token can put its weight on fast pairs; a head that wants to match a word 2,000 tokens back can use slow pairs, which hardly change over short distances.
Where RoPE sits in a layer
- RoPE is applied in every attention layer, after the query and key projections and before the dot product. Nothing is added to the token embedding at the bottom of the network. [Certain]
- Only queries and keys are rotated. Values are not: the position should change who attends to whom, not the content that is copied. [Certain]
- Each head uses the same \(\theta_i\). With grouped-query attention (session 26), the shared keys are rotated once.
- Some models rotate only part of each head: GPT-NeoX rotates 25% of the dimensions and leaves the rest without position. [Likely]
The fast way to compute it
Nobody builds the \(d \times d\) matrix \(R_m\). Since each block only mixes two numbers, the rotation is two element-wise products. Real code uses the half-split layout: pair \(i\) is made of dimension \(i\) and dimension \(i + d/2\) (not dimensions \(2i\) and \(2i+1\)). With that layout:
- Read the symbols
- \(\odot\)
- element-wise product
- \(\boldsymbol{\theta}\)
- the \(d\) angle steps \((\theta_0, \dots, \theta_{d/2-1}, \theta_0, \dots, \theta_{d/2-1})\): each step written twice, once for each half
- \(\mathbf{x}_1, \mathbf{x}_2\)
- the first and the second half of \(\mathbf{x}\), each of length \(d/2\)
The tables \(\cos(m\boldsymbol\theta)\) and \(\sin(m\boldsymbol\theta)\) for all positions are computed once and cached. The half-split layout is the one in GPT-NeoX and in Hugging Face’s Llama code; the paper and Meta’s original code use the interleaved layout (pairs \((2i, 2i+1)\)). [Certain] The two are the same RoPE up to a fixed reordering of the dimensions, and because \(W_q\) and \(W_k\) are learned, a model trained in one layout works in the other only if you also reorder the rows of \(W_q\) and \(W_k\). Hugging Face’s conversion script for Llama does exactly this permutation. [Likely] Getting the layout wrong does not crash: it gives a model that runs and produces nonsense.
Let \(d = 4\) (two pairs) and, to keep numbers simple, \(b = 100\). Then \(\theta_0 = 100^0 = 1\) and \(\theta_1 = 100^{-2/4} = 0.1\). Take \(\mathbf{q} = (1, 0, 1, 0)\) and \(\mathbf{k} = (0, 1, 1, 0)\) in interleaved pairs. As complex numbers: \(\tilde q_0 = 1, \tilde k_0 = i\) and \(\tilde q_1 = 1, \tilde k_1 = 1\). Before rotation, \(\mathbf{q}\cdot\mathbf{k} = 1\).
Query at \(m = 3\), key at \(n = 1\), so \(m - n = 2\). Pair 0: \(\operatorname{Re}[1 \cdot (-i)\, e^{2i}] = \sin 2 = 0.909\). Pair 1: \(\operatorname{Re}[e^{0.2i}] = \cos 0.2 = 0.980\). The score (14.6) is \(0.909 + 0.980 = 1.889\).
Move to \(m = 10\), \(n = 8\): still distance 2, still \(1.889\) (the code below checks this). Pair 0 turns a full radian per step, so it changes a lot from one distance to the next. Pair 1 turns ten times slower; at distance 2 it has hardly moved from its unrotated value 1.
Figure 2 runs the same computation on a small head with \(d = 8\), so 4 pairs, each turning at its own speed \(\theta_i = b^{-2i/8}\). The query and key are fixed: in interleaved pairs, \(\mathbf{q} = (0.9, 0.3 \mid 0.5, 0.8 \mid 0.8, -0.4 \mid 0.3, 0.9)\) and \(\mathbf{k} = (0.6, 0.7 \mid 0.9, 0.2 \mid -0.3, 0.8 \mid 0.7, 0.6)\), with unrotated dot product 1.55. At every position step the code rotates each pair, takes its small dot product \(t_i\), and adds the four terms, exactly like formula (14.6).
- \(\theta_i \leftarrow b^{-2i/d}\) for \(i = 0, \dots, 3\) // (14.5)
- for step \(p = 0, \dots, 9\): set \(m, n\)
- for pair \(i = 0, \dots, 3\):
- \(\mathbf{q}'_i \leftarrow R(m\theta_i)\,\mathbf{q}_i\); \(\mathbf{k}'_i \leftarrow R(n\theta_i)\,\mathbf{k}_i\)
- \(t_i \leftarrow \mathbf{q}'_i \cdot \mathbf{k}'_i\)
- \(s \leftarrow t_0 + t_1 + t_2 + t_3\); plot \((p, s)\) // (14.6)
How to read Figure 2
Press Play. For each position step, the four pairs turn one after another (lines 4 and 5), then the four terms are added (line 6) and a dot is drawn. In “move both” mode with \(b = 100\) and \(\Delta = 2\), every pair turns by a different amount at each step, the four terms change, yet the sum is 0.497 every time: the dots form a flat line. At \(m = 5\), \(n = 3\) the terms are 0.097, 0.126, −0.446 and 0.720. Now pick “move query only”: the key stays at position 2 and the distance grows by one per step, so the score moves: 1.55, 1.401, 0.497, −0.505, … The curve passes 0.497 at step 2, where the distance is 2 again. Switch to \(b = 10{,}000\): pairs 2 and 3 hardly move over 10 positions, and with \(\Delta = 2\) the flat line sits at 0.771. With \(\Delta = 0\) the score is always the unrotated 1.55.
Frequencies, wavelengths and the base
Long and short wavelengths
Pair \(i\) turns by \(\theta_i\) per position, so it comes back to where it started after \(2\pi/\theta_i\) positions. That number is its wavelength:
For \(d = 128\) and \(b = 10{,}000\), the wavelengths go from \(2\pi \approx 6.28\) tokens for pair 0 up to about 54,410 tokens for pair 63. With \(b = 500{,}000\) (Llama 3) the longest wavelength is about 2.56 million tokens. Each step to the next pair multiplies the wavelength by the same factor \(b^{2/d}\) (1.155 for \(b = 10{,}000\)), so on a log scale the wavelengths form a straight line.
- Fast pairs (small \(i\), short wavelength) turn many full circles inside a sentence. They can tell distance 3 from distance 4 very sharply, but after a few tokens their angle “wraps around”, so on their own they cannot tell distance 4 from distance 10.
- Slow pairs (large \(i\), long wavelength) change little between neighbors but keep turning steadily over thousands of tokens. They tell “near” from “far”.
- Together they work like the hands of a clock: seconds, minutes, hours. Many readings combined give a unique and smooth code for each distance.
Here is a fact that matters later. If the training length is \(L\), the pairs with \(\lambda_i \gt L\) never complete even one full turn during training. The model only ever sees part of their circle. For \(d = 128\), \(b = 10{,}000\) and \(L = 4{,}096\) (Llama 2), that is 18 of the 64 pairs. Past \(L\), those pairs reach angles the model has never seen. This is a main reason why a RoPE model does not work well beyond its training length without extra tricks (session 28). [Likely]
How to read Figure 3
Each bar is one pair. Its height is the number of tokens it needs for one full turn. All bases start at the same place, \(2\pi \approx 6.28\) tokens for pair 0, because \(\theta_0 = 1\) for every base. A larger base makes the line steeper, so the slow pairs get much slower. Move the slider for \(L\): with \(b = 10{,}000\) and \(L = 4{,}096\), 18 pairs are orange. With \(b = 500{,}000\) and \(L = 8{,}192\) (Llama 3 pretraining before its long-context stage), 29 pairs are orange. At \(L = 131{,}072 = 2^{17}\), no pair is orange for \(b = 10{,}000\), but 15 still are for \(b = 500{,}000\).
Why the base matters
The base \(b\) sets how slow the slowest pair is. If the longest wavelength is shorter than the context, the slowest pairs wrap around inside the context and two far-apart distances can look alike. A larger base keeps more pairs “monotone” over the whole context, which helps the model tell a token 10,000 back from one 50,000 back. The Llama 3 paper says it raised the base to 500,000 because this helps longer contexts. [Certain] Qwen2-VL uses 1,000,000. [Certain]
The price: with a larger base, fewer pairs are fast, so there is a little less resolution for short distances (compare \(b = 10{,}000\) and \(b = 1{,}000{,}000\) in Figure 3: pair 1 has wavelength 7.26 vs 7.80, and the middle pairs differ much more). In practice, models with a large base do well on short text too, so this cost seems small. [Likely]
Changing the base or the position scale after training is the starting point of context extension: position interpolation divides every position by a factor, “NTK-aware” scaling raises the base, and YaRN treats fast and slow pairs differently. [Certain] Llama 3.1’s config has a rope_scaling entry of type llama3 with factor 8 and an original length of 8,192, which is one such scheme. [Certain] We study these methods in session 28.
Does the score decay with distance?
The RoFormer paper shows a “long-term decay” property: an upper bound on the score that tends to get smaller as \(|m - n|\) grows. [Certain] A simple way to see it: take \(\tilde q_i = \tilde k_i = 1\) for every pair (query and key with the same content). Then every term of (14.6) is a cosine, and the average is
At \(\Delta = 0\) every cosine is 1, so \(s = 1\). As \(\Delta\) grows, the fast pairs point in many different directions and cancel, and only the slow pairs still add up. For \(d = 128\) and \(b = 10{,}000\): \(s(10) = 0.669\), \(s(100) = 0.477\), \(s(1000) = 0.159\). Two warnings. First, the decay is not smooth: the curve wiggles and is not monotone. Second, it only describes this special “same content” case. For a random query and key the expected score is zero at every distance, and a trained head can learn \(W_q, W_k\) that put weight on slow pairs and attend far away with no penalty. So think of it as a mild bias toward nearby tokens, not a rule. [Certain]
How to read Figure 4
At distance 0 the score is 1 for every base. It drops fast in the first few tokens, as the fast pairs fall out of step, then drops slowly. With \(b = 10{,}000\) the curve is near zero by \(\Delta = 4{,}000\) (0.008). With \(b = 500{,}000\) it is still 0.245 there, because more pairs are still slow enough to agree. The wiggles show the curve is not monotone: for example at \(\Delta = 100\), \(b = 1{,}000{,}000\) gives 0.586, slightly below \(b = 500{,}000\) (0.611). Remember that this is one special pair of vectors, not what a trained head does.
A first look at 2D RoPE for images
A vision transformer cuts an image into a grid of patches (session 11 showed the patch embedding). Each patch has two coordinates: a row \(h\) and a column \(w\). The simplest idea is to read the patches in raster order (row by row) and use 1D RoPE on the index \(m = h \cdot W + w\), where \(W\) is the number of columns. That works badly: the patch just below is \(W\) steps away in raster order, while the last patch of one row and the first of the next are neighbors, even though they sit on opposite sides of the image.
2D RoPE fixes this. Split the dimension pairs of each head into two halves. The first half is rotated by the row, the second half by the column:
- Read the symbols
- \(R^{(d/2)}_{h}\)
- the block-diagonal RoPE rotation of (14.6) for a vector of \(d/2\) numbers, at position \(h\)
- \((h, w)\), \((h', w')\)
- row and column of the query patch and of the key patch
- \(f\)
- some function: the point is that the score depends only on the row offset and the column offset
The same proof as (14.3) works on each half, so the score depends only on the 2D offset \((h - h', w - w')\). Moving an object across the image does not change how its patches attend to each other. Qwen2-VL removed the absolute position embeddings of its ViT and used 2D RoPE instead, which lets one encoder handle images of any size. [Certain] Because RoPE needs no learned table, a new image size needs no interpolation of position embeddings, which is a known pain point with learned ViT positions. [Likely]
How to read Figure 5
Press Play. For each key patch the code first finds the offset (line 3), then averages the cosines of the row pairs and of the column pairs (lines 4 and 5, with the numbers in the box), then paints the cell (line 6). In 2D mode the bright area is a round blob around the query: the patch to the right and the patch below get the same score (0.957 for the default query), because both are one step away. For the patch to the right, the row half gives 1 (same row) and the column half gives 0.914, so the score is \((1 + 0.914)/2 = 0.957\). Click another patch: the blob moves with it and keeps its shape. That is translation invariance from (14.10). Switch to 1D raster mode: the code now uses one distance \(\Delta = 8\Delta h + \Delta w\) for all 16 pairs. The bright area becomes a horizontal stripe, the patch below the query drops to 0.401 (it is 8 steps away in raster order) while the patch to the right stays high (0.932). Click a patch at the end of a row and see that the first patch of the next row looks “near”.
A first look at M-RoPE for video
Three coordinates in one sequence
A VLM mixes text tokens and visual tokens in one sequence. A video token has three coordinates: time \(t\) (which frame), row \(h\) and column \(w\). Qwen2-VL’s multimodal RoPE (M-RoPE) gives every token a position triple \((t, h, w)\) and splits the frequency pairs of each head into three groups: one group is rotated by \(t\), one by \(h\), one by \(w\). [Certain] In the released Qwen2-VL-7B config the split is mrope_section = [16, 24, 24]: of the 64 pairs of a 128-dimensional head, 16 see time, 24 see the row and 24 see the column. [Certain, from config.json] In the Hugging Face code the time group gets the fastest frequencies and the width group the slowest. [Likely]
The rules for the triples, from the Qwen2-VL paper: [Certain]
- Text: all three components are equal, \((m, m, m)\). Then every group rotates by the same \(m\), and M-RoPE is exactly ordinary 1D RoPE. A text-only prompt behaves like a normal LLM.
- Image: the time ID is constant; the row and column IDs follow the patch position in the grid.
- Video: like an image for each frame, and the time ID goes up by one for each frame step.
- The next part of the sequence starts at the largest ID used so far plus one.
Written as a formula, for a visual block that starts at offset \(s\):
- Read the symbols
- \(m\)
- the usual running position of a text token
- \(s\)
- the start offset of a visual block: one more than the largest ID before it
- \(f, r, c\)
- frame step, row and column of the visual token inside its block, each counted from 0 (after Qwen2-VL merges \(2 \times 2\) patches and 2 frames into one token)
One nice result: an image of \(H \times W\) tokens uses only \(\max(H, W)\) position steps, not \(H \cdot W\). A \(448 \times 448\) image in Qwen2-VL becomes \(32 \times 32\) patches of 14 pixels, merged \(2 \times 2\) into \(16 \times 16 = 256\) tokens. [Certain] With 1D positions it would push the next text 256 steps further; with M-RoPE only 16. Long videos use much less of the position range, so the model meets fewer of the angles it never saw in training.
The sequence is: Describe, <|vision_start|>, an image of \(2 \times 3\) tokens, <|vision_end|>, and, <|vision_start|>, a video of 3 frame steps of \(2 \times 2\) tokens, <|vision_end|>, ?.
- Text at positions 0 and 1: \((0,0,0)\), \((1,1,1)\).
- The image starts at \(s = 2\): rows 2 to 3 and columns 2 to 4, time fixed at 2. First token \((2,2,2)\), last token \((2,3,4)\). The largest ID is 4.
- The next text starts at 5: \((5,5,5)\), \((6,6,6)\), \((7,7,7)\).
- The video starts at \(s = 8\): time 8 to 10, rows 8 to 9, columns 8 to 9. First token \((8,8,8)\), last token \((10,9,9)\). The largest ID is 10.
- The final text gets \((11,11,11)\) and \((12,12,12)\).
The sequence has 25 tokens but its largest position is only 12. The code below prints the same IDs. Figure 6 steps through it.
How to read Figure 6
Each box is one token, with its three IDs written as t, h, w. Step through the five parts. Text tokens have three equal numbers. Inside the image, t stays fixed while h follows the row and w the column. Inside the video, t also counts the frame steps. After each visual block, the next text starts at one more than the largest ID so far (5 after the image, 11 after the video), not at the token count. The readout compares the largest position with the number of tokens.
This is a preview. The details differ between models: Qwen2.5-VL ties the time ID to real seconds instead of frame count, so videos sampled at different frame rates get consistent times [Likely], and later work changes how the frequencies are split between the three axes [Likely]. We return to these in session 182 and session 205.
Pseudo-code
- input: hidden states \(X \in \R^{T \times d_{\text{model}}}\), positions \(m_1, \dots, m_T\), head size \(d\), base \(b\)
- once: \(\theta_i \leftarrow b^{-2i/d}\) for \(i = 0, \dots, d/2 - 1\) // formula (14.5), in float32
- once: for every position \(m\): \(C_m \leftarrow \cos(m\boldsymbol\theta)\), \(S_m \leftarrow \sin(m\boldsymbol\theta)\), each \(\boldsymbol\theta\) written twice // the cos/sin cache
- \(Q \leftarrow X W_q\), \(K \leftarrow X W_k\), \(V \leftarrow X W_v\), split into heads
- for each token \(t\) and each head do
- \(\mathbf{q} \leftarrow \mathbf{q} \odot C_{m_t} + \operatorname{rotate\_half}(\mathbf{q}) \odot S_{m_t}\) // formula (14.7)
- \(\mathbf{k} \leftarrow \mathbf{k} \odot C_{m_t} + \operatorname{rotate\_half}(\mathbf{k}) \odot S_{m_t}\) // V is not rotated
- store rotated \(K\) and \(V\) in the KV cache
- return \(\softmax(QK^\top/\sqrt{d} + \text{mask})\,V\) // score depends only on \(m - n\), formula (14.6)
- input: list of parts: text of length \(n\), or a visual block of \(F \times H \times W\) tokens
- \(s \leftarrow 0\); ids \(\leftarrow [\,]\)
- for each part do
- if text then for \(j = 0\) to \(n - 1\): append \((s + j, s + j, s + j)\); \(s \leftarrow s + n\)
- else for \(f, r, c\) in frame, row, column order: append \((s + f, s + r, s + c)\) // formula (14.11)
- \(s \leftarrow 1 + \max(\text{all IDs so far})\) // formula (14.12): \(s\) grows by \(\max(F, H, W)\)
- return ids // pair group “t” rotates by the 1st ID, “h” by the 2nd, “w” by the 3rd
The same ideas in PyTorch. Each block checks one claim from this session; the printed values are from a real run.
import torch, math torch.manual_seed(0) # 1) frequencies and the cos/sin cache, formulas (14.5) and (14.7) def rope_cache(T, d, base=10000.0): inv_freq = base ** (-torch.arange(0, d, 2, dtype=torch.float32) / d) # theta_i, (d/2,) ang = torch.arange(T, dtype=torch.float32)[:, None] * inv_freq[None] # m * theta_i, (T, d/2) ang = torch.cat([ang, ang], dim=-1) # (T, d) for the half-split layout return ang.cos(), ang.sin() def rotate_half(x): x1, x2 = x.chunk(2, dim=-1) return torch.cat([-x2, x1], dim=-1) def apply_rope(x, cos, sin): # x: (..., T, d) return x * cos + rotate_half(x) * sin # formula (14.7) # 2) the score depends only on m - n, formula (14.6) d, T = 64, 512 cos, sin = rope_cache(T, d) q, k = torch.randn(d), torch.randn(d) def score(m, n): qm = apply_rope(q, cos[m], sin[m]); kn = apply_rope(k, cos[n], sin[n]) return (qm @ kn).item() print(round(score(10, 3), 3), round(score(310, 303), 3)) # -12.777 -12.777: distance 7 in both # 3) norms do not change: RoPE is a rotation print(torch.allclose(apply_rope(q, cos[100], sin[100]).norm(), q.norm())) # True # 4) the worked example: d = 4, base 100, q = (1,0,1,0), k = (0,1,1,0) in interleaved pairs def rope_interleaved(x, m, base): d = x.shape[-1]; out = x.clone() for i in range(d // 2): a = m * base ** (-2 * i / d); c, s = math.cos(a), math.sin(a) out[2*i] = c * x[2*i] - s * x[2*i+1] out[2*i+1] = s * x[2*i] + c * x[2*i+1] return out qw, kw = torch.tensor([1., 0., 1., 0.]), torch.tensor([0., 1., 1., 0.]) for m, n in [(3, 1), (10, 8)]: print(m, n, round((rope_interleaved(qw, m, 100) @ rope_interleaved(kw, n, 100)).item(), 3)) # 1.889 both # 5) interleaved and half-split are the same RoPE up to a fixed permutation of dimensions perm = torch.cat([torch.arange(0, d, 2), torch.arange(1, d, 2)]) # (x0,x2,...,x1,x3,...) x = torch.randn(d) a = rope_interleaved(x, 37, 10000.0)[perm] b = apply_rope(x[perm], cos[37], sin[37]) print(torch.allclose(a, b, atol=1e-5)) # True # 6) 2D RoPE for an image patch at (row h, column w), formula (14.10) def rope_2d(x, h, w, base=10000.0): half = x.shape[-1] // 2 ch, sh = rope_cache(h + 1, half, base); cw, sw = rope_cache(w + 1, half, base) return torch.cat([apply_rope(x[:half], ch[h], sh[h]), apply_rope(x[half:], cw[w], sw[w])]) s1 = rope_2d(q, 2, 5) @ rope_2d(k, 3, 7) s2 = rope_2d(q, 10, 1) @ rope_2d(k, 11, 3) # same offset (1, 2) print(torch.allclose(s1, s2, atol=1e-4)) # True # 7) M-RoPE position IDs (t, h, w), formulas (14.11) and (14.12), Algorithm 2 def mrope_ids(segments): ids, start = [], 0 for kind, size in segments: if kind == "text": for j in range(size): ids.append((start + j,) * 3) start += size else: # (frames, height, width) after patch merging F, H, W = size for f in range(F): for r in range(H): for c in range(W): ids.append((start + f, start + r, start + c)) start = max(max(p) for p in ids) + 1 return ids ids = mrope_ids([("text", 2), ("image", (1, 2, 3)), ("text", 3), ("image", (3, 2, 2)), ("text", 2)]) print(len(ids), ids[2], ids[7], ids[8], ids[11], ids[-1]) # 25 (2, 2, 2) (2, 3, 4) (5, 5, 5) (8, 8, 8) (12, 12, 12) # 8) bfloat16 cannot hold large positions exactly: compute angles in float32 print(torch.tensor([10001.0, 10033.0]).to(torch.bfloat16).tolist()) # [9984.0, 10048.0]
Where this is used
| Application | Which RoPE | What it gives |
|---|---|---|
| Decoder LLMs (Llama 2 and 3, Qwen2, Mistral and many others) | 1D RoPE in every layer, base 10,000 to 1,000,000 | Relative positions with no learned table; works with the KV cache and fused attention kernels |
| Long-context models (Llama 3.1 at 128K tokens, for example) | 1D RoPE with a larger base and a scaling rule (rope_scaling) | Extends the usable context after pretraining; see session 28 |
| Vision encoders for any image size (Qwen2-VL ViT) | 2D RoPE on (row, column) | No position table to resize; attention depends on the 2D offset between patches |
| Image and document VLMs (Qwen2-VL, Qwen2.5-VL) | M-RoPE with \((t, h, w)\) IDs | Text keeps normal 1D RoPE; an image uses only \(\max(H, W)\) position steps |
| Video question answering, video classification and action recognition | M-RoPE, time ID per frame step | The model can tell frame order and temporal distance between visual tokens |
| Key-moment detection and temporal grounding (“when does the goal happen?”) | M-RoPE with time IDs tied to seconds (Qwen2.5-VL) [Likely] | Time IDs map to real time, so the model can answer with timestamps |
| Long-video understanding and Video-RAG | M-RoPE; compact position use | Many frames fit in a smaller position range, which reduces the out-of-range angle problem |
Common mistakes
- Mixing the interleaved and half-split layouts. Loading weights trained with pairs \((2i, 2i+1)\) into code that rotates \((i, i + d/2)\), or the reverse, gives a model that runs and outputs garbage. Check which layout the checkpoint expects (or permute \(W_q, W_k\)).
- Rotating the values, or rotating the input embedding. RoPE goes on queries and keys, in every layer, after the projection. Rotating \(V\) mixes position into the copied content; rotating only once at the bottom loses the relative property after the first layer.
- Wrong positions with a KV cache or with padding. During generation the new token’s position is the number of tokens already in the cache, not 0. With left padding in a batch, positions must skip the pad tokens (use
position_ids), or each sequence sees shifted angles. [Certain] - Computing angles in bfloat16 or float16. bfloat16 keeps only 8 significant bits: near 10,000 the gap between numbers is 64, so position 10,001 becomes 9,984. Compute \(m\theta_i\), cos and sin in float32, then cast.
- Using a different base at inference than in training. The base is part of the model. Changing it (or
rope_scaling) without matching fine-tuning shifts every angle the model has learned. - Believing RoPE “extrapolates” to any length. The score is relative, but slow pairs meet angles past the training length that they never saw (Figure 3). Quality usually drops past the training length without a scaling method.
- Treating long-term decay as a guarantee. Formula (14.9) is one special case. Trained heads attend far away when they need to; do not argue that RoPE models “cannot” look back.
- Using raster-order 1D positions for image patches. Neighbors above and below become far apart, and row ends look adjacent to the next row (Figure 5). Use 2D RoPE or 2D positions.
Interview questions
What does RoPE do, in two sentences?basic
Prove that the RoPE score depends only on the relative position.core
Why rotate the query and the key but not the value?core
How is RoPE different from adding sinusoidal position vectors?basic
What is the base \(b\), and why did Llama 3 raise it to 500,000?core
Which dimensions of a RoPE head handle short distances, and which handle long ones?core
Does RoPE let a model work past its training length? Why or why not?core
How does RoPE interact with the KV cache?core
Two RoPE implementations give different outputs for the same checkpoint. What is the likely cause?advanced
rotate_half. They are the same up to a permutation of dimensions, so a checkpoint works in the other layout only if the rows of \(W_q\) and \(W_k\) are permuted. Other causes to check: a different base or rope_scaling, partial rotary (only a fraction of dims rotated), positions starting at a different offset, or angles computed in low precision.How would you give positions to image patches? Why not flatten and use 1D RoPE?core
Explain M-RoPE in Qwen2-VL. What happens to text-only input?advanced
RoFormer claims “long-term decay”. Is the attention of a RoPE model forced to be local?advanced
Exercises
- For \(d = 128\) and \(b = 10{,}000\), compute \(\theta_0\), \(\theta_{63}\) and the wavelength \(\lambda_{63}\).
- In 2D with \(\mathbf{q} = (1, 0)\), \(\mathbf{k} = (0, 1)\) and \(\theta = 0.5\), compute the RoPE score for \((m, n) = (2, 1)\), \((5, 4)\) and \((1, 2)\).
- Show that RoPE does not change the length of a query vector. Why does this matter for the softmax temperature?
- Write the M-RoPE IDs for: 3 text tokens, then an image of \(2 \times 2\) tokens, then 1 text token.
- Qwen2-VL turns a \(448 \times 448\) image into how many tokens? How many position steps does it use with M-RoPE, and how many with plain 1D positions?
- Using Figure 3, how many of the 64 pairs never complete a full turn for \(b = 500{,}000\) and \(L = 8{,}192\)? What about \(b = 10{,}000\) and \(L = 32{,}768\)?
- A position ID of 10,001 is stored in bfloat16. What value does the model actually use, and what is the angle error for pair 0?
Answerscheck
Key takeaways
- RoPE rotates each pair of query and key dimensions by \(m\theta_i\) (14.2, 14.6). Because rotations are orthogonal and angles add, the score depends only on the content and on \(m - n\) (14.3).
- It is applied per token (cache friendly, no parameters), in every layer, to queries and keys only.
- Frequencies \(\theta_i = b^{-2i/d}\) (14.5) give wavelengths from \(2\pi\) up to \(2\pi b^{(d-2)/d}\) (14.8): fast pairs resolve short distances, slow pairs encode long ones. A larger base (500,000 in Llama 3, 1,000,000 in Qwen2-VL) makes the slow pairs slower.
- Pairs whose wavelength is longer than the training length never complete a turn in training; past that length they meet new angles, which is why RoPE models need scaling methods to go longer.
- Implementation: a float32 cos/sin cache and
x·cos + rotate_half(x)·sin(14.7). Interleaved and half-split layouts differ by a permutation; mixing them silently breaks a model. - 2D RoPE gives image patches a score that depends on the row and column offset (14.10). M-RoPE gives every token \((t, h, w)\) IDs (14.11, 14.12); text keeps plain 1D RoPE, and visual blocks use few position steps.
Key formulas
References for this session
- Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 6: Transformers
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 6: Transformers
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Vaswani et al. (2017). Attention Is All You Need
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Chen et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation
- Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models
- Black, Biderman, Hallahan, et al. (2022). GPT-NeoX-20B: An Open-Source Autoregressive Language Model