LLM & VLM courseBlog
14Track 1: Foundations · session 14 of 15

Position, part 2: RoPE in depth, and a first look at 2D and 3D RoPE

Rotary position embedding (RoPE) does not add a position vector to the token. It rotates the query and the key by an angle that grows with the position, so the attention score between two tokens depends only on how far apart they are. This session builds RoPE from one 2D rotation to a full head, explains the frequencies and the base, and then shows how the same trick gives positions to image patches (2D RoPE) and to video tokens (M-RoPE in Qwen2-VL).

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 2, 11 and 13 (vectors and dot products, embeddings, absolute and relative positions)

What you will learn

  • Why rotating the query and the key, instead of adding a position vector, makes the score depend only on the distance m − n
  • The 2D rotation, the complex-number view, and the block-diagonal rotation for a full head
  • Frequencies θ_i = b^(−2i/d): which dimension pairs see short distances and which see long ones, and what the base b changes
  • How RoPE is implemented in practice (cos/sin cache, rotate_half) and the interleaved vs half-split layout bug
  • Why RoPE does not extrapolate by itself past the training length, and where context extension starts
  • 2D RoPE for image patches: half the dimensions see the row, half see the column
  • M-RoPE for video in Qwen2-VL: (time, height, width) position IDs, and how text, images and video share one sequence

Why this matters

Almost every open LLM you will meet in practice, Llama, Qwen, Mistral, Gemma, DeepSeek, uses the same method to tell attention where each token is: rotary position embedding, or RoPE. It was introduced by Su et al. in the RoFormer paper (2021). [Certain] Most VLMs use it too, and the Qwen-VL family stretches it to images and video.

In session 13 we saw the problem. Self-attention compares every query with every key, and that comparison does not care about order: without help, “dog bites man” and “man bites dog” look the same. We saw two kinds of fixes. Absolute methods add a position vector to each token (sinusoidal or learned). Relative methods change the attention score with a term that depends on the distance between the two tokens (T5 buckets, ALiBi). RoPE gets the best of both:

  • It is applied like an absolute method. Each query and each key is changed using only its own position \(m\). Nothing about pairs is stored, so it works with a KV cache and fast attention kernels without any change to them.
  • It behaves like a relative method. After the change, the score between a query at position \(m\) and a key at position \(n\) depends only on the distance \(m - n\), not on \(m\) and \(n\) themselves.
  • It costs almost nothing. No learned parameters, a few multiplications per number, and it never touches the values.

The trick is a rotation. This session builds it from one pair of numbers up to a full attention head, explains the frequencies and the “base” (why Llama 3 uses 500,000 instead of 10,000), and then takes a first look at 2D RoPE for image patches and M-RoPE for video, which we meet again in the vision and video tracks.

Interviewers ask about RoPE often, because the answer shows whether you understand attention at the level of dot products: “Prove that RoPE gives relative positions”, “Why rotate the query and the key but not the value?”, “What does the base do?”, “Why does a model break past its training length?”, “How would you give positions to image patches?”. By the end of this session you can answer each one with a formula and a small example.

The core idea: rotate, do not add

What goes wrong with adding

The original transformer adds a position vector \(\mathbf{p}_m\) to the token embedding \(\mathbf{x}_m\). The query and key are then \(W_q(\mathbf{x}_m + \mathbf{p}_m)\) and \(W_k(\mathbf{x}_n + \mathbf{p}_n)\). If you expand their dot product, you get four terms: content with content, content with position, position with content, and position with position. Only some parts of the last term depend on \(m - n\) in a clean way, and the cross terms mix “what” and “where” in a way the model has to untangle. We would like a change that gives a score of the form “content of the query, content of the key, and their distance”, and nothing else.

One pair of numbers

Start with the smallest case: a query and a key with only two numbers each, \(\mathbf{q}, \mathbf{k} \in \R^2\). Think of each as an arrow in the plane. A rotation by angle \(\varphi\) turns an arrow around the origin without changing its length. In matrix form:

\[ R(\varphi) = \begin{pmatrix} \cos\varphi & -\sin\varphi \\ \sin\varphi & \cos\varphi \end{pmatrix} \tag{14.1} \]

RoPE picks a fixed angle step \(\theta\) and rotates the query of the token at position \(m\) by \(m\theta\), and the key of the token at position \(n\) by \(n\theta\):

\[ \mathbf{q}_m = R(m\theta)\, W_q \mathbf{x}_m, \qquad \mathbf{k}_n = R(n\theta)\, W_k \mathbf{x}_n \tag{14.2} \]
Read the symbols
\(\mathbf{x}_m\)
the hidden vector of the token at position \(m\) (no position added to it)
\(W_q, W_k\)
the usual learned query and key projections
\(m, n\)
positions of the query token and the key token: 0, 1, 2, …
\(\theta\)
a fixed angle per position step (in radians). It is not learned.
\(R(m\theta)\)
rotation by \(m\) times \(\theta\): the further along the sequence, the more the arrow is turned

Now compute the attention score. Write \(\mathbf{q} = W_q\mathbf{x}_m\) and \(\mathbf{k} = W_k\mathbf{x}_n\) for the vectors before rotation. Two facts about rotations do all the work: the transpose of a rotation is the rotation backwards, \(R(\alpha)^\top = R(-\alpha)\), and two rotations add their angles, \(R(\alpha)R(\beta) = R(\alpha + \beta)\). So:

\[ \mathbf{q}_m^\top \mathbf{k}_n = \mathbf{q}^\top R(m\theta)^\top R(n\theta)\, \mathbf{k} = \mathbf{q}^\top R\big((n - m)\theta\big)\, \mathbf{k} \tag{14.3} \]

This is the whole idea. The positions \(m\) and \(n\) only appear as the difference \(n - m\). Move both tokens forward by 100 positions and the score does not change. The model still sees the content (through \(\mathbf{q}\) and \(\mathbf{k}\)) and the distance (through the angle), and nothing else. [Certain]

The same thing with complex numbers

A pair of real numbers \((a, b)\) can be written as one complex number \(a + ib\). Rotating by \(\varphi\) is then just multiplying by \(e^{i\varphi} = \cos\varphi + i\sin\varphi\). And the dot product of two 2D vectors is the real part of one times the conjugate of the other. Write \(\tilde q = q_1 + i q_2\) and \(\tilde k = k_1 + i k_2\). Then:

\[ \mathbf{q}_m^\top \mathbf{k}_n = \operatorname{Re}\!\left[\tilde q\, e^{i m\theta}\; \overline{\tilde k\, e^{i n\theta}}\right] = \operatorname{Re}\!\left[\tilde q\, \overline{\tilde k}\; e^{i (m - n)\theta}\right] \tag{14.4} \]
Read the symbols
\(\tilde q, \tilde k\)
the query pair and the key pair, each written as one complex number
\(\overline{z}\)
the complex conjugate of \(z\) (flip the sign of the imaginary part)
\(e^{i\varphi}\)
a complex number of length 1 at angle \(\varphi\): multiplying by it rotates by \(\varphi\)
\(\operatorname{Re}[\cdot]\)
the real part

The RoFormer paper derives RoPE in this complex form, and Meta’s original Llama code implements it with complex multiplication. [Certain] It is the same as (14.3): the score is the length of \(\tilde q\) times the length of \(\tilde k\) times the cosine of (the angle between them plus \((m - n)\theta\)).

Worked example: the shift does not matter

Take \(\mathbf{q} = (1, 0)\), \(\mathbf{k} = (0, 1)\) and \(\theta = 0.5\). Before rotation \(\mathbf{q}\cdot\mathbf{k} = 0\). With \(R(\varphi)\mathbf{k} = (-\sin\varphi, \cos\varphi)\), formula (14.3) gives \(\mathbf{q}_m^\top\mathbf{k}_n = -\sin\big((n - m)\theta\big) = \sin\big((m - n)\theta\big)\).

  • \(m = 2, n = 1\): \(\sin(0.5) = 0.479\).
  • \(m = 5, n = 4\): again distance 1, again \(0.479\).
  • \(m = 1, n = 2\): distance \(-1\), so \(\sin(-0.5) = -0.479\). RoPE knows the sign of the distance, not only its size. A query can tell a key one step behind from a key one step ahead.
  1. \(\mathbf{q}_m \leftarrow R(m\theta)\,\mathbf{q}\) // (14.2)
  2. \(\mathbf{k}_n \leftarrow R(n\theta)\,\mathbf{k}\) // (14.2)
  3. \(s \leftarrow \mathbf{q}_m \cdot \mathbf{k}_n\)
  4. check: \(s = \mathbf{q} \cdot R\big((n - m)\theta\big)\,\mathbf{k}\) // (14.3)
  5. repeat with \(m \leftarrow m + 1\), \(n \leftarrow n + 1\)
Figure 1. RoPE in two dimensions, computed live. The faint arrows are the query \(\mathbf{q} = (0.9, 0.3)\) and the key \(\mathbf{k} = (0.5, 0.8)\) before rotation. The solid arrows are \(\mathbf{q}_m = R(m\theta)\mathbf{q}\) (blue) and \(\mathbf{k}_n = R(n\theta)\mathbf{k}\) (orange). Each animation round moves both tokens one position forward: first the query turns by \(\theta\), then the key. The box beside the code shows the line being run with the current numbers.
How to read Figure 1

Press Play. In each round the query turns first (line 1). For a moment the two tokens are 4 apart instead of 3, and the score drops from 0.781 to a different value (0.550 in the first round). Then the key turns by the same angle (line 2), the gap between the arrows is back, and the score is 0.781 again (line 3). Line 4 checks the same number with formula (14.3), which never uses \(m\) or \(n\) alone. The arrows swing all the way around the circle, but at the end of every round the score is the same. Use the sliders to pick the start: with \(m = n\) the score is the unrotated \(\mathbf{q}\cdot\mathbf{k} = 0.69\) at every round. Swap them (start \(m = 2\), \(n = 5\)) and the score is −0.281, because the distance changed sign. A larger \(\theta\) makes the arrows turn faster per position.

From one pair to a full head

Many pairs, many speeds

A real attention head has \(d\) numbers per query and key (for Llama 3 8B, \(d = 128\)). [Certain] RoPE cuts the vector into \(d/2\) pairs and rotates each pair in its own plane, with its own angle step \(\theta_i\). The steps follow the same geometric sequence as the sinusoidal positions of the original transformer:

\[ \theta_i = b^{-2i/d}, \qquad i = 0, 1, \dots, \tfrac{d}{2} - 1 \tag{14.5} \]
Read the symbols
\(i\)
the index of a pair of dimensions (pair 0 is the first two numbers, and so on)
\(b\)
the base, a fixed number: 10,000 in RoFormer and Llama 2, 500,000 in Llama 3, 1,000,000 in Qwen2 and Qwen2-VL [Certain, from the model configs]
\(\theta_i\)
the angle (in radians) that pair \(i\) turns per position. Pair 0 has \(\theta_0 = 1\); the last pair turns very slowly.

The full rotation for position \(m\) is a block-diagonal matrix: \(d/2\) small \(2 \times 2\) rotations on the diagonal and zeros everywhere else. Because each block is a rotation, the whole matrix is a rotation too: it keeps lengths (\(\lVert R_m \mathbf{q}\rVert = \lVert\mathbf{q}\rVert\)), and \(R_m^\top R_n = R_{n-m}\) still holds block by block.

\[ R_m = \begin{pmatrix} R(m\theta_0) & & \\ & \ddots & \\ & & R(m\theta_{d/2-1}) \end{pmatrix}, \qquad \mathbf{q}_m^\top\mathbf{k}_n = \sum_{i=0}^{d/2-1} \operatorname{Re}\!\left[\tilde q_i\, \overline{\tilde k_i}\; e^{i(m-n)\theta_i}\right] \tag{14.6} \]

So the score is a sum of \(d/2\) terms, one per pair. Each term is a cosine wave in the distance \(m - n\), with its own speed \(\theta_i\) and with an amplitude and phase set by the content (\(\tilde q_i \overline{\tilde k_i}\)). The model learns \(W_q\) and \(W_k\), so it learns which pairs to use and how. A head that wants to look at the previous token can put its weight on fast pairs; a head that wants to match a word 2,000 tokens back can use slow pairs, which hardly change over short distances.

Where RoPE sits in a layer

  • RoPE is applied in every attention layer, after the query and key projections and before the dot product. Nothing is added to the token embedding at the bottom of the network. [Certain]
  • Only queries and keys are rotated. Values are not: the position should change who attends to whom, not the content that is copied. [Certain]
  • Each head uses the same \(\theta_i\). With grouped-query attention (session 26), the shared keys are rotated once.
  • Some models rotate only part of each head: GPT-NeoX rotates 25% of the dimensions and leaves the rest without position. [Likely]

The fast way to compute it

Nobody builds the \(d \times d\) matrix \(R_m\). Since each block only mixes two numbers, the rotation is two element-wise products. Real code uses the half-split layout: pair \(i\) is made of dimension \(i\) and dimension \(i + d/2\) (not dimensions \(2i\) and \(2i+1\)). With that layout:

\[ \operatorname{RoPE}(\mathbf{x}, m) = \mathbf{x} \odot \cos(m\boldsymbol{\theta}) + \operatorname{rotate\_half}(\mathbf{x}) \odot \sin(m\boldsymbol{\theta}), \qquad \operatorname{rotate\_half}(\mathbf{x}_{1}, \mathbf{x}_{2}) = (-\mathbf{x}_{2}, \mathbf{x}_{1}) \tag{14.7} \]
Read the symbols
\(\odot\)
element-wise product
\(\boldsymbol{\theta}\)
the \(d\) angle steps \((\theta_0, \dots, \theta_{d/2-1}, \theta_0, \dots, \theta_{d/2-1})\): each step written twice, once for each half
\(\mathbf{x}_1, \mathbf{x}_2\)
the first and the second half of \(\mathbf{x}\), each of length \(d/2\)

The tables \(\cos(m\boldsymbol\theta)\) and \(\sin(m\boldsymbol\theta)\) for all positions are computed once and cached. The half-split layout is the one in GPT-NeoX and in Hugging Face’s Llama code; the paper and Meta’s original code use the interleaved layout (pairs \((2i, 2i+1)\)). [Certain] The two are the same RoPE up to a fixed reordering of the dimensions, and because \(W_q\) and \(W_k\) are learned, a model trained in one layout works in the other only if you also reorder the rows of \(W_q\) and \(W_k\). Hugging Face’s conversion script for Llama does exactly this permutation. [Likely] Getting the layout wrong does not crash: it gives a model that runs and produces nonsense.

Worked example: a 4-dimensional head

Let \(d = 4\) (two pairs) and, to keep numbers simple, \(b = 100\). Then \(\theta_0 = 100^0 = 1\) and \(\theta_1 = 100^{-2/4} = 0.1\). Take \(\mathbf{q} = (1, 0, 1, 0)\) and \(\mathbf{k} = (0, 1, 1, 0)\) in interleaved pairs. As complex numbers: \(\tilde q_0 = 1, \tilde k_0 = i\) and \(\tilde q_1 = 1, \tilde k_1 = 1\). Before rotation, \(\mathbf{q}\cdot\mathbf{k} = 1\).

Query at \(m = 3\), key at \(n = 1\), so \(m - n = 2\). Pair 0: \(\operatorname{Re}[1 \cdot (-i)\, e^{2i}] = \sin 2 = 0.909\). Pair 1: \(\operatorname{Re}[e^{0.2i}] = \cos 0.2 = 0.980\). The score (14.6) is \(0.909 + 0.980 = 1.889\).

Move to \(m = 10\), \(n = 8\): still distance 2, still \(1.889\) (the code below checks this). Pair 0 turns a full radian per step, so it changes a lot from one distance to the next. Pair 1 turns ten times slower; at distance 2 it has hardly moved from its unrotated value 1.

Figure 2 runs the same computation on a small head with \(d = 8\), so 4 pairs, each turning at its own speed \(\theta_i = b^{-2i/8}\). The query and key are fixed: in interleaved pairs, \(\mathbf{q} = (0.9, 0.3 \mid 0.5, 0.8 \mid 0.8, -0.4 \mid 0.3, 0.9)\) and \(\mathbf{k} = (0.6, 0.7 \mid 0.9, 0.2 \mid -0.3, 0.8 \mid 0.7, 0.6)\), with unrotated dot product 1.55. At every position step the code rotates each pair, takes its small dot product \(t_i\), and adds the four terms, exactly like formula (14.6).

  1. \(\theta_i \leftarrow b^{-2i/d}\) for \(i = 0, \dots, 3\) // (14.5)
  2. for step \(p = 0, \dots, 9\): set \(m, n\)
  3. for pair \(i = 0, \dots, 3\):
  4. \(\mathbf{q}'_i \leftarrow R(m\theta_i)\,\mathbf{q}_i\); \(\mathbf{k}'_i \leftarrow R(n\theta_i)\,\mathbf{k}_i\)
  5. \(t_i \leftarrow \mathbf{q}'_i \cdot \mathbf{k}'_i\)
  6. \(s \leftarrow t_0 + t_1 + t_2 + t_3\); plot \((p, s)\) // (14.6)
Figure 2. A full RoPE head with \(d = 8\), computed live. Top: the 4 pairs as 4 small planes. Pair 0 turns 1 radian per position; the last pair turns slowest. Faint arrows are the pairs before rotation; solid arrows are \(\mathbf{q}'_i\) (blue) and \(\mathbf{k}'_i\) (orange); \(t_i\) is their dot product. Bottom: the total score \(s\) at each position step. “Move both” uses \(m = p + \Delta\), \(n = p\); “move query only” keeps the key at \(n = 2\) and uses \(m = 2 + p\).
How to read Figure 2

Press Play. For each position step, the four pairs turn one after another (lines 4 and 5), then the four terms are added (line 6) and a dot is drawn. In “move both” mode with \(b = 100\) and \(\Delta = 2\), every pair turns by a different amount at each step, the four terms change, yet the sum is 0.497 every time: the dots form a flat line. At \(m = 5\), \(n = 3\) the terms are 0.097, 0.126, −0.446 and 0.720. Now pick “move query only”: the key stays at position 2 and the distance grows by one per step, so the score moves: 1.55, 1.401, 0.497, −0.505, … The curve passes 0.497 at step 2, where the distance is 2 again. Switch to \(b = 10{,}000\): pairs 2 and 3 hardly move over 10 positions, and with \(\Delta = 2\) the flat line sits at 0.771. With \(\Delta = 0\) the score is always the unrotated 1.55.

Frequencies, wavelengths and the base

Long and short wavelengths

Pair \(i\) turns by \(\theta_i\) per position, so it comes back to where it started after \(2\pi/\theta_i\) positions. That number is its wavelength:

\[ \lambda_i = \frac{2\pi}{\theta_i} = 2\pi\, b^{2i/d} \tag{14.8} \]

For \(d = 128\) and \(b = 10{,}000\), the wavelengths go from \(2\pi \approx 6.28\) tokens for pair 0 up to about 54,410 tokens for pair 63. With \(b = 500{,}000\) (Llama 3) the longest wavelength is about 2.56 million tokens. Each step to the next pair multiplies the wavelength by the same factor \(b^{2/d}\) (1.155 for \(b = 10{,}000\)), so on a log scale the wavelengths form a straight line.

  • Fast pairs (small \(i\), short wavelength) turn many full circles inside a sentence. They can tell distance 3 from distance 4 very sharply, but after a few tokens their angle “wraps around”, so on their own they cannot tell distance 4 from distance 10.
  • Slow pairs (large \(i\), long wavelength) change little between neighbors but keep turning steadily over thousands of tokens. They tell “near” from “far”.
  • Together they work like the hands of a clock: seconds, minutes, hours. Many readings combined give a unique and smooth code for each distance.

Here is a fact that matters later. If the training length is \(L\), the pairs with \(\lambda_i \gt L\) never complete even one full turn during training. The model only ever sees part of their circle. For \(d = 128\), \(b = 10{,}000\) and \(L = 4{,}096\) (Llama 2), that is 18 of the 64 pairs. Past \(L\), those pairs reach angles the model has never seen. This is a main reason why a RoPE model does not work well beyond its training length without extra tricks (session 28). [Likely]

Figure 3. The wavelength \(\lambda_i = 2\pi b^{2i/d}\) of each of the 64 dimension pairs of a head with \(d = 128\), on a log scale. The dashed line is the training length \(L\). Orange bars are pairs that never complete one full turn within \(L\) tokens.
How to read Figure 3

Each bar is one pair. Its height is the number of tokens it needs for one full turn. All bases start at the same place, \(2\pi \approx 6.28\) tokens for pair 0, because \(\theta_0 = 1\) for every base. A larger base makes the line steeper, so the slow pairs get much slower. Move the slider for \(L\): with \(b = 10{,}000\) and \(L = 4{,}096\), 18 pairs are orange. With \(b = 500{,}000\) and \(L = 8{,}192\) (Llama 3 pretraining before its long-context stage), 29 pairs are orange. At \(L = 131{,}072 = 2^{17}\), no pair is orange for \(b = 10{,}000\), but 15 still are for \(b = 500{,}000\).

Why the base matters

The base \(b\) sets how slow the slowest pair is. If the longest wavelength is shorter than the context, the slowest pairs wrap around inside the context and two far-apart distances can look alike. A larger base keeps more pairs “monotone” over the whole context, which helps the model tell a token 10,000 back from one 50,000 back. The Llama 3 paper says it raised the base to 500,000 because this helps longer contexts. [Certain] Qwen2-VL uses 1,000,000. [Certain]

The price: with a larger base, fewer pairs are fast, so there is a little less resolution for short distances (compare \(b = 10{,}000\) and \(b = 1{,}000{,}000\) in Figure 3: pair 1 has wavelength 7.26 vs 7.80, and the middle pairs differ much more). In practice, models with a large base do well on short text too, so this cost seems small. [Likely]

Changing the base or the position scale after training is the starting point of context extension: position interpolation divides every position by a factor, “NTK-aware” scaling raises the base, and YaRN treats fast and slow pairs differently. [Certain] Llama 3.1’s config has a rope_scaling entry of type llama3 with factor 8 and an original length of 8,192, which is one such scheme. [Certain] We study these methods in session 28.

Does the score decay with distance?

The RoFormer paper shows a “long-term decay” property: an upper bound on the score that tends to get smaller as \(|m - n|\) grows. [Certain] A simple way to see it: take \(\tilde q_i = \tilde k_i = 1\) for every pair (query and key with the same content). Then every term of (14.6) is a cosine, and the average is

\[ s(\Delta) = \frac{2}{d} \sum_{i=0}^{d/2-1} \cos(\Delta\, \theta_i), \qquad \Delta = m - n \tag{14.9} \]

At \(\Delta = 0\) every cosine is 1, so \(s = 1\). As \(\Delta\) grows, the fast pairs point in many different directions and cancel, and only the slow pairs still add up. For \(d = 128\) and \(b = 10{,}000\): \(s(10) = 0.669\), \(s(100) = 0.477\), \(s(1000) = 0.159\). Two warnings. First, the decay is not smooth: the curve wiggles and is not monotone. Second, it only describes this special “same content” case. For a random query and key the expected score is zero at every distance, and a trained head can learn \(W_q, W_k\) that put weight on slow pairs and attend far away with no penalty. So think of it as a mild bias toward nearby tokens, not a rule. [Certain]

Figure 4. The score \(s(\Delta)\) of formula (14.9) for a query and key with the same content in every pair, \(d = 128\), for distances 0 to 4,000 (the x-axis is on a square-root scale, so short distances get more room). The dot marks the distance chosen with the slider.
How to read Figure 4

At distance 0 the score is 1 for every base. It drops fast in the first few tokens, as the fast pairs fall out of step, then drops slowly. With \(b = 10{,}000\) the curve is near zero by \(\Delta = 4{,}000\) (0.008). With \(b = 500{,}000\) it is still 0.245 there, because more pairs are still slow enough to agree. The wiggles show the curve is not monotone: for example at \(\Delta = 100\), \(b = 1{,}000{,}000\) gives 0.586, slightly below \(b = 500{,}000\) (0.611). Remember that this is one special pair of vectors, not what a trained head does.

A first look at 2D RoPE for images

A vision transformer cuts an image into a grid of patches (session 11 showed the patch embedding). Each patch has two coordinates: a row \(h\) and a column \(w\). The simplest idea is to read the patches in raster order (row by row) and use 1D RoPE on the index \(m = h \cdot W + w\), where \(W\) is the number of columns. That works badly: the patch just below is \(W\) steps away in raster order, while the last patch of one row and the first of the next are neighbors, even though they sit on opposite sides of the image.

2D RoPE fixes this. Split the dimension pairs of each head into two halves. The first half is rotated by the row, the second half by the column:

\[ R_{(h,w)} = \begin{pmatrix} R^{(d/2)}_{h} & 0 \\ 0 & R^{(d/2)}_{w} \end{pmatrix}, \qquad \mathbf{q}_{(h,w)}^\top \mathbf{k}_{(h',w')} = f\big(\mathbf{q}, \mathbf{k},\, h - h',\, w - w'\big) \tag{14.10} \]
Read the symbols
\(R^{(d/2)}_{h}\)
the block-diagonal RoPE rotation of (14.6) for a vector of \(d/2\) numbers, at position \(h\)
\((h, w)\), \((h', w')\)
row and column of the query patch and of the key patch
\(f\)
some function: the point is that the score depends only on the row offset and the column offset

The same proof as (14.3) works on each half, so the score depends only on the 2D offset \((h - h', w - w')\). Moving an object across the image does not change how its patches attend to each other. Qwen2-VL removed the absolute position embeddings of its ViT and used 2D RoPE instead, which lets one encoder handle images of any size. [Certain] Because RoPE needs no learned table, a new image size needs no interpolation of position embeddings, which is a known pain point with learned ViT positions. [Likely]

high scoreabout zeronegativekey being scored
    Figure 5. Scores from one query patch (marked Q) to every patch of an \(8 \times 8\) image, for a head with \(d = 32\) and base 100, when the query and all keys have the same content (\(\tilde q_i = \tilde k_i = 1\), as in formula 14.9). Only the position changes the score. Press Play to watch the code score the keys one by one in raster order; striped cells are not scored yet. Color strength grows with the square of the score, to make differences easier to see. Click any patch to make it the query (the full map is then shown at once).
    How to read Figure 5

    Press Play. For each key patch the code first finds the offset (line 3), then averages the cosines of the row pairs and of the column pairs (lines 4 and 5, with the numbers in the box), then paints the cell (line 6). In 2D mode the bright area is a round blob around the query: the patch to the right and the patch below get the same score (0.957 for the default query), because both are one step away. For the patch to the right, the row half gives 1 (same row) and the column half gives 0.914, so the score is \((1 + 0.914)/2 = 0.957\). Click another patch: the blob moves with it and keeps its shape. That is translation invariance from (14.10). Switch to 1D raster mode: the code now uses one distance \(\Delta = 8\Delta h + \Delta w\) for all 16 pairs. The bright area becomes a horizontal stripe, the patch below the query drops to 0.401 (it is 8 steps away in raster order) while the patch to the right stays high (0.932). Click a patch at the end of a row and see that the first patch of the next row looks “near”.

    A first look at M-RoPE for video

    Three coordinates in one sequence

    A VLM mixes text tokens and visual tokens in one sequence. A video token has three coordinates: time \(t\) (which frame), row \(h\) and column \(w\). Qwen2-VL’s multimodal RoPE (M-RoPE) gives every token a position triple \((t, h, w)\) and splits the frequency pairs of each head into three groups: one group is rotated by \(t\), one by \(h\), one by \(w\). [Certain] In the released Qwen2-VL-7B config the split is mrope_section = [16, 24, 24]: of the 64 pairs of a 128-dimensional head, 16 see time, 24 see the row and 24 see the column. [Certain, from config.json] In the Hugging Face code the time group gets the fastest frequencies and the width group the slowest. [Likely]

    The rules for the triples, from the Qwen2-VL paper: [Certain]

    • Text: all three components are equal, \((m, m, m)\). Then every group rotates by the same \(m\), and M-RoPE is exactly ordinary 1D RoPE. A text-only prompt behaves like a normal LLM.
    • Image: the time ID is constant; the row and column IDs follow the patch position in the grid.
    • Video: like an image for each frame, and the time ID goes up by one for each frame step.
    • The next part of the sequence starts at the largest ID used so far plus one.

    Written as a formula, for a visual block that starts at offset \(s\):

    \[ \text{text: } (m, m, m), \qquad \text{visual token (frame } f, \text{ row } r, \text{ column } c\text{): } (s + f,\; s + r,\; s + c) \tag{14.11} \]
    \[ s_{\text{next}} = 1 + \max\big(\text{all IDs used so far}\big) \tag{14.12} \]
    Read the symbols
    \(m\)
    the usual running position of a text token
    \(s\)
    the start offset of a visual block: one more than the largest ID before it
    \(f, r, c\)
    frame step, row and column of the visual token inside its block, each counted from 0 (after Qwen2-VL merges \(2 \times 2\) patches and 2 frames into one token)

    One nice result: an image of \(H \times W\) tokens uses only \(\max(H, W)\) position steps, not \(H \cdot W\). A \(448 \times 448\) image in Qwen2-VL becomes \(32 \times 32\) patches of 14 pixels, merged \(2 \times 2\) into \(16 \times 16 = 256\) tokens. [Certain] With 1D positions it would push the next text 256 steps further; with M-RoPE only 16. Long videos use much less of the position range, so the model meets fewer of the angles it never saw in training.

    Worked example: position IDs for text, image, text, video, text

    The sequence is: Describe, <|vision_start|>, an image of \(2 \times 3\) tokens, <|vision_end|>, and, <|vision_start|>, a video of 3 frame steps of \(2 \times 2\) tokens, <|vision_end|>, ?.

    • Text at positions 0 and 1: \((0,0,0)\), \((1,1,1)\).
    • The image starts at \(s = 2\): rows 2 to 3 and columns 2 to 4, time fixed at 2. First token \((2,2,2)\), last token \((2,3,4)\). The largest ID is 4.
    • The next text starts at 5: \((5,5,5)\), \((6,6,6)\), \((7,7,7)\).
    • The video starts at \(s = 8\): time 8 to 10, rows 8 to 9, columns 8 to 9. First token \((8,8,8)\), last token \((10,9,9)\). The largest ID is 10.
    • The final text gets \((11,11,11)\) and \((12,12,12)\).

    The sequence has 25 tokens but its largest position is only 12. The code below prints the same IDs. Figure 6 steps through it.

    textimagevideo
    Figure 6. M-RoPE position IDs \((t, h, w)\) for a sequence of text, one image (\(2 \times 3\) tokens) and one video (3 frame steps of \(2 \times 2\) tokens), following the rules of formulas (14.11) and (14.12). Each step adds one part of the sequence.
    How to read Figure 6

    Each box is one token, with its three IDs written as t, h, w. Step through the five parts. Text tokens have three equal numbers. Inside the image, t stays fixed while h follows the row and w the column. Inside the video, t also counts the frame steps. After each visual block, the next text starts at one more than the largest ID so far (5 after the image, 11 after the video), not at the token count. The readout compares the largest position with the number of tokens.

    This is a preview. The details differ between models: Qwen2.5-VL ties the time ID to real seconds instead of frame count, so videos sampled at different frame rates get consistent times [Likely], and later work changes how the frequencies are split between the three axes [Likely]. We return to these in session 182 and session 205.

    Pseudo-code

    Algorithm 1: RoPE in one attention layer (half-split layout)
    1. input: hidden states \(X \in \R^{T \times d_{\text{model}}}\), positions \(m_1, \dots, m_T\), head size \(d\), base \(b\)
    2. once: \(\theta_i \leftarrow b^{-2i/d}\) for \(i = 0, \dots, d/2 - 1\) // formula (14.5), in float32
    3. once: for every position \(m\): \(C_m \leftarrow \cos(m\boldsymbol\theta)\), \(S_m \leftarrow \sin(m\boldsymbol\theta)\), each \(\boldsymbol\theta\) written twice // the cos/sin cache
    4. \(Q \leftarrow X W_q\), \(K \leftarrow X W_k\), \(V \leftarrow X W_v\), split into heads
    5. for each token \(t\) and each head do
    6. \(\mathbf{q} \leftarrow \mathbf{q} \odot C_{m_t} + \operatorname{rotate\_half}(\mathbf{q}) \odot S_{m_t}\) // formula (14.7)
    7. \(\mathbf{k} \leftarrow \mathbf{k} \odot C_{m_t} + \operatorname{rotate\_half}(\mathbf{k}) \odot S_{m_t}\) // V is not rotated
    8. store rotated \(K\) and \(V\) in the KV cache
    9. return \(\softmax(QK^\top/\sqrt{d} + \text{mask})\,V\) // score depends only on \(m - n\), formula (14.6)
    Algorithm 2: M-RoPE position IDs for a mixed sequence
    1. input: list of parts: text of length \(n\), or a visual block of \(F \times H \times W\) tokens
    2. \(s \leftarrow 0\); ids \(\leftarrow [\,]\)
    3. for each part do
    4. if text then for \(j = 0\) to \(n - 1\): append \((s + j, s + j, s + j)\); \(s \leftarrow s + n\)
    5. else for \(f, r, c\) in frame, row, column order: append \((s + f, s + r, s + c)\) // formula (14.11)
    6. \(s \leftarrow 1 + \max(\text{all IDs so far})\) // formula (14.12): \(s\) grows by \(\max(F, H, W)\)
    7. return ids // pair group “t” rotates by the 1st ID, “h” by the 2nd, “w” by the 3rd

    The same ideas in PyTorch. Each block checks one claim from this session; the printed values are from a real run.

    import torch, math
    torch.manual_seed(0)
    
    # 1) frequencies and the cos/sin cache, formulas (14.5) and (14.7)
    def rope_cache(T, d, base=10000.0):
        inv_freq = base ** (-torch.arange(0, d, 2, dtype=torch.float32) / d)   # theta_i, (d/2,)
        ang = torch.arange(T, dtype=torch.float32)[:, None] * inv_freq[None]  # m * theta_i, (T, d/2)
        ang = torch.cat([ang, ang], dim=-1)                                   # (T, d) for the half-split layout
        return ang.cos(), ang.sin()
    
    def rotate_half(x):
        x1, x2 = x.chunk(2, dim=-1)
        return torch.cat([-x2, x1], dim=-1)
    
    def apply_rope(x, cos, sin):                 # x: (..., T, d)
        return x * cos + rotate_half(x) * sin     # formula (14.7)
    
    # 2) the score depends only on m - n, formula (14.6)
    d, T = 64, 512
    cos, sin = rope_cache(T, d)
    q, k = torch.randn(d), torch.randn(d)
    def score(m, n):
        qm = apply_rope(q, cos[m], sin[m]); kn = apply_rope(k, cos[n], sin[n])
        return (qm @ kn).item()
    print(round(score(10, 3), 3), round(score(310, 303), 3))   # -12.777 -12.777: distance 7 in both
    
    # 3) norms do not change: RoPE is a rotation
    print(torch.allclose(apply_rope(q, cos[100], sin[100]).norm(), q.norm()))  # True
    
    # 4) the worked example: d = 4, base 100, q = (1,0,1,0), k = (0,1,1,0) in interleaved pairs
    def rope_interleaved(x, m, base):
        d = x.shape[-1]; out = x.clone()
        for i in range(d // 2):
            a = m * base ** (-2 * i / d); c, s = math.cos(a), math.sin(a)
            out[2*i]   = c * x[2*i] - s * x[2*i+1]
            out[2*i+1] = s * x[2*i] + c * x[2*i+1]
        return out
    qw, kw = torch.tensor([1., 0., 1., 0.]), torch.tensor([0., 1., 1., 0.])
    for m, n in [(3, 1), (10, 8)]:
        print(m, n, round((rope_interleaved(qw, m, 100) @ rope_interleaved(kw, n, 100)).item(), 3))  # 1.889 both
    
    # 5) interleaved and half-split are the same RoPE up to a fixed permutation of dimensions
    perm = torch.cat([torch.arange(0, d, 2), torch.arange(1, d, 2)])  # (x0,x2,...,x1,x3,...)
    x = torch.randn(d)
    a = rope_interleaved(x, 37, 10000.0)[perm]
    b = apply_rope(x[perm], cos[37], sin[37])
    print(torch.allclose(a, b, atol=1e-5))     # True
    
    # 6) 2D RoPE for an image patch at (row h, column w), formula (14.10)
    def rope_2d(x, h, w, base=10000.0):
        half = x.shape[-1] // 2
        ch, sh = rope_cache(h + 1, half, base); cw, sw = rope_cache(w + 1, half, base)
        return torch.cat([apply_rope(x[:half], ch[h], sh[h]), apply_rope(x[half:], cw[w], sw[w])])
    s1 = rope_2d(q, 2, 5) @ rope_2d(k, 3, 7)
    s2 = rope_2d(q, 10, 1) @ rope_2d(k, 11, 3)   # same offset (1, 2)
    print(torch.allclose(s1, s2, atol=1e-4))   # True
    
    # 7) M-RoPE position IDs (t, h, w), formulas (14.11) and (14.12), Algorithm 2
    def mrope_ids(segments):
        ids, start = [], 0
        for kind, size in segments:
            if kind == "text":
                for j in range(size): ids.append((start + j,) * 3)
                start += size
            else:                                # (frames, height, width) after patch merging
                F, H, W = size
                for f in range(F):
                    for r in range(H):
                        for c in range(W): ids.append((start + f, start + r, start + c))
                start = max(max(p) for p in ids) + 1
        return ids
    ids = mrope_ids([("text", 2), ("image", (1, 2, 3)), ("text", 3), ("image", (3, 2, 2)), ("text", 2)])
    print(len(ids), ids[2], ids[7], ids[8], ids[11], ids[-1])
    # 25 (2, 2, 2) (2, 3, 4) (5, 5, 5) (8, 8, 8) (12, 12, 12)
    
    # 8) bfloat16 cannot hold large positions exactly: compute angles in float32
    print(torch.tensor([10001.0, 10033.0]).to(torch.bfloat16).tolist())   # [9984.0, 10048.0]

    Where this is used

    ApplicationWhich RoPEWhat it gives
    Decoder LLMs (Llama 2 and 3, Qwen2, Mistral and many others)1D RoPE in every layer, base 10,000 to 1,000,000Relative positions with no learned table; works with the KV cache and fused attention kernels
    Long-context models (Llama 3.1 at 128K tokens, for example)1D RoPE with a larger base and a scaling rule (rope_scaling)Extends the usable context after pretraining; see session 28
    Vision encoders for any image size (Qwen2-VL ViT)2D RoPE on (row, column)No position table to resize; attention depends on the 2D offset between patches
    Image and document VLMs (Qwen2-VL, Qwen2.5-VL)M-RoPE with \((t, h, w)\) IDsText keeps normal 1D RoPE; an image uses only \(\max(H, W)\) position steps
    Video question answering, video classification and action recognitionM-RoPE, time ID per frame stepThe model can tell frame order and temporal distance between visual tokens
    Key-moment detection and temporal grounding (“when does the goal happen?”)M-RoPE with time IDs tied to seconds (Qwen2.5-VL) [Likely]Time IDs map to real time, so the model can answer with timestamps
    Long-video understanding and Video-RAGM-RoPE; compact position useMany frames fit in a smaller position range, which reduces the out-of-range angle problem

    Common mistakes

    • Mixing the interleaved and half-split layouts. Loading weights trained with pairs \((2i, 2i+1)\) into code that rotates \((i, i + d/2)\), or the reverse, gives a model that runs and outputs garbage. Check which layout the checkpoint expects (or permute \(W_q, W_k\)).
    • Rotating the values, or rotating the input embedding. RoPE goes on queries and keys, in every layer, after the projection. Rotating \(V\) mixes position into the copied content; rotating only once at the bottom loses the relative property after the first layer.
    • Wrong positions with a KV cache or with padding. During generation the new token’s position is the number of tokens already in the cache, not 0. With left padding in a batch, positions must skip the pad tokens (use position_ids), or each sequence sees shifted angles. [Certain]
    • Computing angles in bfloat16 or float16. bfloat16 keeps only 8 significant bits: near 10,000 the gap between numbers is 64, so position 10,001 becomes 9,984. Compute \(m\theta_i\), cos and sin in float32, then cast.
    • Using a different base at inference than in training. The base is part of the model. Changing it (or rope_scaling) without matching fine-tuning shifts every angle the model has learned.
    • Believing RoPE “extrapolates” to any length. The score is relative, but slow pairs meet angles past the training length that they never saw (Figure 3). Quality usually drops past the training length without a scaling method.
    • Treating long-term decay as a guarantee. Formula (14.9) is one special case. Trained heads attend far away when they need to; do not argue that RoPE models “cannot” look back.
    • Using raster-order 1D positions for image patches. Neighbors above and below become far apart, and row ends look adjacent to the next row (Figure 5). Use 2D RoPE or 2D positions.

    Interview questions

    What does RoPE do, in two sentences?basic
    RoPE splits each query and key into pairs of numbers and rotates pair \(i\) by the angle \(m\theta_i\), where \(m\) is the token’s position and \(\theta_i = b^{-2i/d}\). Because rotations preserve dot products up to the difference of angles, the attention score between positions \(m\) and \(n\) depends only on the content and on \(m - n\) (formula 14.6), while each token is processed using only its own position.
    Prove that the RoPE score depends only on the relative position.core
    With \(\mathbf{q}_m = R_m\mathbf{q}\) and \(\mathbf{k}_n = R_n\mathbf{k}\): \(\mathbf{q}_m^\top\mathbf{k}_n = \mathbf{q}^\top R_m^\top R_n\mathbf{k}\). Each \(R_m\) is block-diagonal with \(2 \times 2\) rotations \(R(m\theta_i)\). A rotation is orthogonal, \(R(\alpha)^\top = R(-\alpha)\), and angles add, \(R(-\alpha)R(\beta) = R(\beta - \alpha)\). So \(R_m^\top R_n = R_{n-m}\) and the score is \(\mathbf{q}^\top R_{n-m}\mathbf{k}\) (formula 14.3). In complex form, each pair gives \(\operatorname{Re}[\tilde q_i\overline{\tilde k_i}e^{i(m-n)\theta_i}]\).
    Why rotate the query and the key but not the value?core
    Position should decide which tokens attend to each other, and that is decided by \(\mathbf{q}\cdot\mathbf{k}\). The value is the content that gets copied into the output; rotating it would mix the absolute position of the source token into the content, and the relative property would not hold for the output. Also the rotation of a value would not cancel against anything, so the output would depend on absolute positions.
    How is RoPE different from adding sinusoidal position vectors?basic
    Both use the same geometric frequencies. Sinusoidal positions are added once to the input embedding, so the score contains cross terms of content and position and only partly depends on distance. RoPE multiplies (rotates) queries and keys in every layer, so the score is exactly a function of content and \(m - n\). RoPE has no parameters, never touches values, and the position signal does not fade through the residual stream because it is re-applied in each layer.
    What is the base \(b\), and why did Llama 3 raise it to 500,000?core
    The base sets the frequencies \(\theta_i = b^{-2i/d}\) and so the wavelengths \(2\pi b^{2i/d}\) (formula 14.8). The fastest pair always has wavelength \(2\pi\); the base decides how slow the slowest pair is: about 54K tokens for \(b = 10{,}000\) and \(d = 128\), about 2.56M for \(b = 500{,}000\). A larger base keeps more pairs from wrapping around inside a long context, so far distances stay distinguishable. The Llama 3 paper reports that the larger base helps long contexts. The cost is slightly less resolution for short distances.
    Which dimensions of a RoPE head handle short distances, and which handle long ones?core
    The first pairs (small \(i\)) have \(\theta_i\) near 1 and wavelengths of a few tokens: they separate nearby distances sharply but wrap around quickly. The last pairs have tiny \(\theta_i\) and wavelengths of tens of thousands of tokens or more: they change slowly and encode coarse, long-range distance. In the half-split layout, pair \(i\) is dimensions \(i\) and \(i + d/2\), so the “fast” dimensions are at the start of each half.
    Does RoPE let a model work past its training length? Why or why not?core
    Not by itself. The score is relative, but relative distances larger than the training length \(L\) were never seen. For pairs with wavelength longer than \(L\) (18 of 64 for \(b = 10{,}000\), \(d = 128\), \(L = 4{,}096\)), the angles past \(L\) are new, and attention patterns break, so perplexity usually rises sharply. Fixes change the positions or frequencies: position interpolation (divide \(m\) by a factor), NTK-aware scaling (raise the base), YaRN (treat fast and slow pairs differently), usually with some long-context fine-tuning.
    How does RoPE interact with the KV cache?core
    Each key is rotated with its own absolute position when it is computed, and the rotated key is stored in the cache. The new query is rotated with its own position (the current cache length). Because \(R_m^\top R_n = R_{n-m}\), the scores come out relative, with no need to re-rotate old keys. This is why RoPE fits the cache so well, unlike schemes that need a pair-wise bias table. One consequence: if you drop or move tokens in the cache (for example in StreamingLLM-style eviction), the stored rotations still carry the old positions, which you must handle.
    Two RoPE implementations give different outputs for the same checkpoint. What is the likely cause?advanced
    The pair layout. The paper and Meta’s original code rotate interleaved pairs \((2i, 2i+1)\); GPT-NeoX and Hugging Face Llama code rotate \((i, i + d/2)\) with rotate_half. They are the same up to a permutation of dimensions, so a checkpoint works in the other layout only if the rows of \(W_q\) and \(W_k\) are permuted. Other causes to check: a different base or rope_scaling, partial rotary (only a fraction of dims rotated), positions starting at a different offset, or angles computed in low precision.
    How would you give positions to image patches? Why not flatten and use 1D RoPE?core
    Use 2D RoPE: split each head’s pairs into two halves, rotate one by the row index and one by the column index (formula 14.10). Then the score depends on the 2D offset, so it is translation invariant and treats up and down like left and right. Flattened raster order puts the patch below \(W\) steps away and makes the end of one row look adjacent to the start of the next (Figure 5: the patch below drops from 0.957 to 0.401). 2D RoPE also needs no learned table, so it handles any image size, which is why Qwen2-VL’s ViT uses it.
    Explain M-RoPE in Qwen2-VL. What happens to text-only input?advanced
    Each token gets a triple \((t, h, w)\). The frequency pairs of a head are split into three groups (16, 24, 24 of 64 pairs in Qwen2-VL-7B), rotated by \(t\), \(h\) and \(w\). Text tokens use \((m, m, m)\), so all groups rotate by the same \(m\) and it is exactly 1D RoPE: text-only behavior matches a normal LLM. Image tokens keep \(t\) fixed and use their row and column; video tokens also increase \(t\) per frame step. Each new part starts at one more than the largest ID so far, so an \(H \times W\) image only uses \(\max(H, W)\) positions. Qwen2.5-VL later ties the time ID to real seconds.
    RoFormer claims “long-term decay”. Is the attention of a RoPE model forced to be local?advanced
    No. The property is an upper bound that tends to shrink with distance, and the clean example (formula 14.9) assumes the query and key have the same content in every pair; then fast pairs cancel and the score drops (0.669 at distance 10, 0.159 at 1,000 for \(b = 10{,}000\), \(d = 128\)). The curve is not even monotone. For random vectors the expected score is zero at every distance, and a trained head can put its weight on slow pairs to attend far away. So it is a mild locality bias, not a hard rule.

    Exercises

    1. For \(d = 128\) and \(b = 10{,}000\), compute \(\theta_0\), \(\theta_{63}\) and the wavelength \(\lambda_{63}\).
    2. In 2D with \(\mathbf{q} = (1, 0)\), \(\mathbf{k} = (0, 1)\) and \(\theta = 0.5\), compute the RoPE score for \((m, n) = (2, 1)\), \((5, 4)\) and \((1, 2)\).
    3. Show that RoPE does not change the length of a query vector. Why does this matter for the softmax temperature?
    4. Write the M-RoPE IDs for: 3 text tokens, then an image of \(2 \times 2\) tokens, then 1 text token.
    5. Qwen2-VL turns a \(448 \times 448\) image into how many tokens? How many position steps does it use with M-RoPE, and how many with plain 1D positions?
    6. Using Figure 3, how many of the 64 pairs never complete a full turn for \(b = 500{,}000\) and \(L = 8{,}192\)? What about \(b = 10{,}000\) and \(L = 32{,}768\)?
    7. A position ID of 10,001 is stored in bfloat16. What value does the model actually use, and what is the angle error for pair 0?
    Answerscheck
    1. \(\theta_0 = 10{,}000^0 = 1\). \(\theta_{63} = 10{,}000^{-126/128} \approx 1.155 \times 10^{-4}\). \(\lambda_{63} = 2\pi/\theta_{63} \approx 54{,}410\) tokens. 2. The score is \(\sin((m - n)\theta)\): \(\sin 0.5 = 0.479\) for both \((2,1)\) and \((5,4)\), and \(-0.479\) for \((1,2)\). 3. \(\lVert R_m\mathbf{q}\rVert^2 = \mathbf{q}^\top R_m^\top R_m\mathbf{q} = \mathbf{q}^\top\mathbf{q}\), since \(R_m^\top R_m = I\). So RoPE never makes scores bigger or smaller on average; the \(1/\sqrt{d}\) scaling and the softmax temperature stay valid at every position. 4. Text: \((0,0,0), (1,1,1), (2,2,2)\). Image starts at 3: \((3,3,3), (3,3,4), (3,4,3), (3,4,4)\); largest ID 4. Text: \((5,5,5)\). 5. \(448/14 = 32\), so \(32 \times 32 = 1{,}024\) patches, merged \(2 \times 2\) into \(16 \times 16 = 256\) tokens. With M-RoPE the next part starts 16 steps later; with 1D positions, 256 steps later. 6. 29 pairs for \(b = 500{,}000\), \(L = 8{,}192\); 4 pairs for \(b = 10{,}000\), \(L = 32{,}768\). 7. Near 10,000, bfloat16 numbers are 64 apart, so 10,001 becomes 9,984: an error of 17 positions, which for pair 0 (\(\theta_0 = 1\)) is 17 radians, almost three full turns. The model sees a completely wrong fast-pair angle. Compute positions and angles in float32.

    Key takeaways

    • RoPE rotates each pair of query and key dimensions by \(m\theta_i\) (14.2, 14.6). Because rotations are orthogonal and angles add, the score depends only on the content and on \(m - n\) (14.3).
    • It is applied per token (cache friendly, no parameters), in every layer, to queries and keys only.
    • Frequencies \(\theta_i = b^{-2i/d}\) (14.5) give wavelengths from \(2\pi\) up to \(2\pi b^{(d-2)/d}\) (14.8): fast pairs resolve short distances, slow pairs encode long ones. A larger base (500,000 in Llama 3, 1,000,000 in Qwen2-VL) makes the slow pairs slower.
    • Pairs whose wavelength is longer than the training length never complete a turn in training; past that length they meet new angles, which is why RoPE models need scaling methods to go longer.
    • Implementation: a float32 cos/sin cache and x·cos + rotate_half(x)·sin (14.7). Interleaved and half-split layouts differ by a permutation; mixing them silently breaks a model.
    • 2D RoPE gives image patches a score that depends on the row and column offset (14.10). M-RoPE gives every token \((t, h, w)\) IDs (14.11, 14.12); text keeps plain 1D RoPE, and visual blocks use few position steps.

    Key formulas

    \[ \mathbf{q}_m = R_m\mathbf{q},\quad \mathbf{k}_n = R_n\mathbf{k},\quad \mathbf{q}_m^\top\mathbf{k}_n = \mathbf{q}^\top R_{n-m}\,\mathbf{k} \]
    \[ \theta_i = b^{-2i/d},\qquad \lambda_i = 2\pi\, b^{2i/d},\qquad \operatorname{RoPE}(\mathbf{x}, m) = \mathbf{x}\odot\cos(m\boldsymbol\theta) + \operatorname{rotate\_half}(\mathbf{x})\odot\sin(m\boldsymbol\theta) \]
    \[ \text{M-RoPE: text } (m, m, m),\quad \text{visual } (s+f,\ s+r,\ s+c),\quad s_{\text{next}} = 1 + \max(\text{IDs}) \]

    References for this session

    1. Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
    2. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 6: Transformers
    3. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
    4. Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
    5. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 6: Transformers
    6. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
    7. Vaswani et al. (2017). Attention Is All You Need
    8. Llama Team, Meta (2024). The Llama 3 Herd of Models
    9. Bai et al. (2025). Qwen2.5-VL Technical Report
    10. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
    11. Chen et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation
    12. Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models
    13. Black, Biderman, Hallahan, et al. (2022). GPT-NeoX-20B: An Open-Source Autoregressive Language Model