LLM & VLM courseBlog
28Track 2: Attention and transformers · session 13 of 15

What “context” means, and how to make it longer

A model card may say “128K context”, but that number is a setting, not a skill. This session separates the context window, the training length and the effective context, shows why RoPE fails past its training length and how position interpolation, NTK-aware scaling and YaRN fix it, and explains how needle-in-a-haystack, “lost in the middle” and RULER measure what a model can really use. It ends with the choice between long context, memory and retrieval.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 14 and 26 (RoPE in depth, the KV cache and GQA); session 27 helps

What you will learn

  • The difference between the context window, the training length and the effective context, with real model configs
  • What limits long context: positions, KV memory and compute, data, and attention itself; the KV-cache formula with Llama 3.1 8B numbers
  • Why RoPE’s slow dimension pairs reach never-seen angles past the training length
  • Position interpolation, NTK-aware base scaling and YaRN (NTK-by-parts plus a temperature), and why Llama 3.1’s rope_scaling is the same ramp
  • How to test long context: needle-in-a-haystack, the lost-in-the-middle U-curve, and RULER’s effective length
  • When to use a long context, memory or retrieval (RAG), and how to combine them

Why this matters

Model cards say “128K context”. Users paste a 300-page contract and ask about page 150. Sometimes the answer is right. Sometimes the model ignores the middle of the document and answers from the first and last pages. This session explains why, and what engineers do about it.

The context of a model is the text it can see while it makes the next token: the system prompt, the chat history, pasted documents, retrieved passages, tool outputs, image and video tokens. Everything the model “knows” about your task right now must be inside it. So the size of the context, and how well the model really uses it, decides what you can build: long-document question answering, code assistants that read a whole repository, agents with long histories, and video models that watch an hour of frames.

Three questions come up again and again, in real systems and in interviews:

  • What does “context length” even mean? There are at least three different numbers: the window the code accepts, the length the model was trained on, and the length it can really use. They are often far apart.
  • How do you make a RoPE model work past its training length? In session 14 we saw that RoPE angles beyond the training length are new to the model. Position interpolation, NTK-aware scaling and YaRN change the angles so the model only sees familiar ones. Llama 3.1, Qwen2.5 and Code Llama all use one of these ideas. [Certain]
  • How do you test it? Needle-in-a-haystack, the “lost in the middle” study, and the RULER benchmark show that a long window is not the same as long understanding.

The last section steps back. A long context is only one way to give a model more information. The other two are memory (store and recall facts across sessions) and retrieval (search a large store and put only the best pieces into the context). Choosing between them is a system-design question you will meet in session 152 and session 206.

Three numbers called “context length”

When someone says “this model has a 32K context”, they could mean three different things. Keep them apart:

  1. Context window (also “maximum context”): the largest number of tokens the code and the serving stack will accept. In a Hugging Face model this is usually max_position_embeddings in config.json. It is a setting, not a skill.
  2. Training length \(L\): the longest sequences the model actually saw during (pre- or continued) training. Positions up to \(L\) are familiar. Positions past \(L\) were never seen.
  3. Effective context: the longest input on which the model still does a given task well. It depends on the task. Finding one fact is easy; combining ten facts spread over the input is hard.

Usually effective context \(\le\) training length \(\le\) context window, but real configs show how loose the link can be:

  • Mistral 7B v0.1: max_position_embeddings is 32,768, but the model uses sliding-window attention with a window of 4,096 tokens, so no single layer looks back more than 4,096 tokens (session 27). [Certain, from the config]
  • Llama 3.1 8B: the window is 131,072 tokens. The base model was pretrained at 8,192 and then extended in six stages to 128K, using about 800 billion tokens of long-context training. Meta moved to the next stage only when short-context scores had fully recovered and needle-in-a-haystack was solved at that length. [Certain]
  • Code Llama: trained on 16K-token sequences with a RoPE base of 1,000,000, and shows gains on inputs up to 100K tokens, so here the useful length is above the training length. [Certain]
  • Qwen2.5-7B-Instruct: the config says 32,768; the model card says to add a YaRN rope_scaling entry with factor 4 to go to about 128K. [Certain, from the model card]

What limits the context?

Four different things stop a model from using very long inputs. Each one has its own fixes, and this course treats them in different places:

  1. Positions. Past the training length, RoPE produces angles the model never saw. Fixes: change the angles (this session), then fine-tune a little.
  2. Memory and compute. The KV cache grows with every token, and prefill attention grows with the square of the length (session 80, session 124).
  3. Data. Few training documents are really long, so the model has little practice using far-away information. Fixes: up-sample long documents and build synthetic long tasks during continued training. [Likely]
  4. Attention itself. Softmax spreads a fixed budget of weight over more and more tokens. The model must learn to focus on the few that matter, and it often prefers the start and the end (Section “Lost in the middle”).

The memory cost is easy to compute. For each token, every layer stores one key and one value vector for each key-value head:

\[ \text{KV bytes} = 2 \cdot n_{\text{layers}} \cdot n_{\text{kv}} \cdot d_{\text{head}} \cdot \text{bytes per number} \cdot T \tag{28.1} \]
Read the symbols
\(2\)
one key and one value
\(n_{\text{layers}}, n_{\text{kv}}, d_{\text{head}}\)
number of layers, number of key-value heads (fewer than query heads with GQA), and the size of one head
\(T\)
the number of tokens in the context
Worked example: the KV cache of Llama 3.1 8B

From the config: 32 layers, 8 key-value heads, head size 128, stored in BF16 (2 bytes). Per token: \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes \(= 128\) KiB.

At 8,192 tokens that is 1 GiB. At the full 131,072 tokens it is 16 GiB for one sequence, on top of the 16 GB of weights. A 1-million-token context would need about 122 GiB. And prefill must compute about \(131{,}072^2 / 2 \approx 8.6 \times 10^{9}\) attention scores per head per layer. Long context is expensive even when the model handles it well.

Why RoPE breaks past the training length

Recall RoPE from session 14. A head of size \(d\) is cut into \(d/2\) pairs. Pair \(i\) of the query (and key) at position \(m\) is rotated by the angle \(m\theta_i\):

\[ \varphi_i(m) = m\, \theta_i, \qquad \theta_i = b^{-2i/d}, \qquad \lambda_i = \frac{2\pi}{\theta_i}, \qquad i = 0, \dots, \tfrac d2 - 1 \tag{28.2} \]
Read the symbols
\(\varphi_i(m)\)
the rotation angle of pair \(i\) at position \(m\), in radians
\(\theta_i\)
how many radians pair \(i\) turns per token; pair 0 turns 1 radian, the last pair turns very slowly
\(b\)
the base: 10,000 in Llama 2, 500,000 in Llama 3 [Certain]
\(\lambda_i\)
the wavelength: after \(\lambda_i\) tokens pair \(i\) has turned one full circle

Now look at what each pair saw during training with sequences up to length \(L\). The angle of pair \(i\) went from 0 up to \(L\theta_i\). Two cases:

  • Fast pairs (\(\lambda_i \le L\)). They turned at least one full circle inside the training length, so they have seen every possible angle many times. A longer input gives them no new angle. They are safe.
  • Slow pairs (\(\lambda_i \gt L\)). They only covered an arc from 0 to \(L\theta_i \lt 2\pi\). At a longer input of length \(L'\), they reach angles up to \(L'\theta_i\), which the model has never seen. The attention scores built on them are then unpredictable.

For Llama 2 (\(d = 128\), \(b = 10{,}000\), \(L = 4{,}096\)) there are 18 slow pairs out of 64. The slowest one, pair 63, has a wavelength of 54,410 tokens; in training it only reached 0.473 rad (27°). At 16,384 tokens it would reach 1.892 rad. That is the problem in one number. [Certain, computed]

We can write the goal of every method in this session as one rule. A method changes the angle steps from \(\theta_i\) to new values \(\theta'_i\). For the new length \(L' = sL\), we want:

\[ \text{for every slow pair } (\lambda_i \gt L): \quad L' \theta'_i \;\le\; L\, \theta_i \tag{28.3} \]

In words: the largest angle used at the new length must not go past the largest angle seen in training. The number \(s = L'/L\) is the scale factor. Fast pairs do not need the rule, because they already saw the whole circle.

angles seen in training (0 to \(L\theta_i\))angles used at \(L'\), inside the training rangeangles used at \(L'\) that were never seen
Figure 1. The angles that one RoPE pair goes through, for a Llama 2 style head (\(d = 128\), \(b = 10{,}000\), training length \(L = 4{,}096\)). The gray ring is what training covered. The inner ring is what the longer input \(L' = sL\) uses after the chosen method. Orange means new, never-seen angles.
How to read Figure 1

Start with “No change”, pair 47 and \(s = 4\). Training covered only 4.73 rad of the circle; at 16,384 tokens the pair goes to 18.92 rad, so most of the inner ring is orange. Move the pair slider below 46: the pair has a wavelength shorter than 4,096, the gray ring is full, and there is nothing new to see. Now press “Position interpolation”: every pair is slowed by \(s\), the orange part disappears, but the readout shows each token step is 4 times smaller. Press “NTK-aware” at pair 47: some orange stays (6.73 rad vs 4.73). YaRN removes it. The readout also counts, over all 64 pairs, how many reach new angles.

Position interpolation (PI)

The first fix is very simple. Chen et al. (2023) proposed to squeeze the new positions into the old range. Instead of position \(m\), use position \(m/s\):

\[ \varphi'_i(m) = \frac{m}{s}\, \theta_i \quad\Longleftrightarrow\quad \theta'_i = \frac{\theta_i}{s}, \qquad s = \frac{L'}{L} \tag{28.4} \]

Every pair now turns \(s\) times slower. The largest angle at \(L'\) is \(L' \theta_i / s = L\theta_i\): exactly the training range, for every pair, so rule (28.3) holds. Positions become fractional (token 5 sits at “position 1.25” when \(s = 4\)), but the model has seen all the angles around them. Interpolating between known points is much safer than extrapolating beyond them.

The paper extended LLaMA models from 7B to 65B parameters to windows of up to 32,768 tokens with fewer than 1,000 fine-tuning steps. It also showed that the upper bound on the error of interpolated attention scores is at least about 600 times smaller than for extrapolation. [Certain] Without any fine-tuning, PI already works better than plain extrapolation, but a short fine-tune is what makes it good. [Likely]

In Hugging Face this is rope_scaling = {"rope_type": "linear", "factor": s}. [Certain]

The price of PI: crowded neighbors

PI slows down the fast pairs too, and they never needed it. With \(s = 4\), pair 0 turns 0.25 rad per token instead of 1 rad. The angle difference between neighbors, which the model uses to tell “the token right before me” from “two tokens back”, becomes 4 times smaller. With \(s = 32\) it is 32 times smaller. Local order gets blurry, and fine-tuning has to teach the model to read these smaller differences. This is why PI loses some quality on short texts and why larger factors work worse. [Likely]

NTK-aware scaling: change the base instead

In June 2023, a user called bloc97 posted on Reddit a different idea, called “NTK-aware” scaling (the name comes from neural tangent kernel theory, which says networks have trouble learning high-frequency details from few inputs). [Certain] The idea: do not slow down every pair by the same factor. Keep the fast pairs (high frequency) as they are, slow the slowest pair by the full factor \(s\), and scale the pairs in between smoothly. All of this happens by changing only the base:

\[ b' = b \cdot s^{\frac{d}{d-2}}, \qquad \theta'_i = b'^{-2i/d} = \theta_i \cdot s^{-\frac{2i}{d-2}} \tag{28.5} \]
Read the symbols
\(b'\)
the new base
\(s^{-2i/(d-2)}\)
the slow-down of pair \(i\): equal to 1 for pair 0 (no change) and to \(1/s\) for the last pair \(i = d/2 - 1\)

Check the last pair: \(2i/(d-2) = (d-2)/(d-2) = 1\), so \(\theta'_{d/2-1} = \theta_{d/2-1}/s\). The exponent \(d/(d-2)\) is chosen exactly for that. In between, the slow-down follows a straight line on a log scale.

Worked example: Llama 2, from 4,096 to 16,384 tokens

\(d = 128\), \(b = 10{,}000\), \(s = 4\). The new base is \(b' = 10{,}000 \cdot 4^{128/126} = 40{,}890\).

  • Pair 0 (wavelength 6.3 tokens): PI slows it to 0.25 rad per token. NTK-aware keeps 1 rad per token.
  • Pair 10 (wavelength 26.5): PI multiplies \(\theta\) by 0.25, NTK-aware by 0.80.
  • Pair 63 (wavelength 54,410): both multiply by 0.25. The largest angle at 16,384 tokens is 0.473 rad, the same as in training.
  • Pair 47 (wavelength 5,441, a slow pair): NTK-aware multiplies by 0.356, so at 16,384 tokens the angle reaches 6.73 rad, but training only went to 4.73 rad. This pair still extrapolates.

Counting all 18 slow pairs, 17 still reach never-seen angles under NTK-aware scaling at \(s = 4\); only the last one is fully inside. NTK-aware keeps local detail, but it does not fully meet rule (28.3). [Certain, computed]

NTK-aware scaling worked surprisingly well without any fine-tuning, which made it popular in 2023. [Likely] A variant called dynamic NTK recomputes the base at inference from the current length, so short inputs are not changed at all and long inputs are scaled just enough (Hugging Face rope_type = "dynamic"). [Certain]

Raising the base during training

The same lever is used at training time. If you plan long contexts, you can simply train with a large base from the start, or raise it before long-context training. Code Llama uses \(b = 1{,}000{,}000\) and Llama 3 uses 500,000. [Certain] A larger base makes the slow pairs slower, so a longer \(L\) is still inside their first turn. This is often called ABF (adjusted base frequency). [Likely]

YaRN: treat each pair by how much it turned

YaRN (Peng et al., 2023) combines the two ideas and adds one more. It has two parts. [Certain]

Part 1: “NTK-by-parts”

First, measure how many full turns pair \(i\) makes inside the training length:

\[ r_i = \frac{L}{\lambda_i} = \frac{L\, \theta_i}{2\pi} \tag{28.6} \]

If \(r_i\) is large, the pair turned many times in training and is safe: leave it alone. If \(r_i \lt 1\), the pair never completed one turn: interpolate it fully, exactly like PI. In between, blend the two with a linear ramp:

\[ \gamma(r) = \begin{cases} 0 & \text{if } r \lt \alpha \\ 1 & \text{if } r \gt \beta \\ \dfrac{r - \alpha}{\beta - \alpha} & \text{otherwise} \end{cases} \tag{28.7} \]
\[ \theta'_i = \big(1 - \gamma(r_i)\big)\, \frac{\theta_i}{s} \;+\; \gamma(r_i)\, \theta_i \tag{28.8} \]
Read the symbols
\(r_i\)
the number of full turns pair \(i\) made in a training sequence of length \(L\)
\(\gamma\)
how much of the original speed to keep: 1 means “do not touch”, 0 means “divide by \(s\)”
\(\alpha, \beta\)
the two ends of the ramp. For the Llama family the paper uses \(\alpha = 1\) and \(\beta = 32\), found by experiment. [Certain]

Unlike NTK-aware scaling, every pair with \(r_i \lt 1\) (all slow pairs) is fully interpolated, so rule (28.3) holds exactly for them. Unlike PI, every pair with \(r_i \gt 32\) keeps its full speed, so local order stays sharp. For Llama 2 with \(L = 4{,}096\): 21 pairs are kept, 18 are fully interpolated and 25 are blended. These counts do not depend on \(s\), because \(r_i\) only uses the training length. [Certain, computed]

Part 2: a temperature on the attention logits

Interpolation has a side effect: at a longer length, the softmax spreads its weight over many more tokens, and the attention gets flatter (higher entropy). YaRN makes the logits a little sharper by multiplying the query and the key by a factor, which is the same as dividing the logits by a temperature \(t\):

\[ \softmax\!\left( \frac{\mathbf{q}_m^\top \mathbf{k}_n}{t \sqrt{d}} \right), \qquad \sqrt{\frac{1}{t}} = 0.1 \ln s + 1 \tag{28.9} \]

For \(s = 4\), each of \(\mathbf{q}\) and \(\mathbf{k}\) is multiplied by 1.139, so the logits are multiplied by 1.296. For \(s = 16\): 1.631. For \(s = 32\): 1.813. The formula was fitted on Llama models. [Certain] In code, it costs nothing: Hugging Face multiplies the cached cos and sin tables by \(0.1 \ln s + 1\), so it works with any attention kernel. [Certain]

With both parts, YaRN extended Llama 2 7B and 13B to 64K (\(s = 16\), 400 fine-tuning steps at batch size 64 on 64K-token chunks of PG19) and then to 128K (\(s = 32\), 200 more steps). The authors report 10 times fewer tokens and 2.5 times fewer training steps than earlier methods. [Certain] The released 128K model has rope_scaling = {"type": "yarn", "factor": 32, "original_max_position_embeddings": 4096}. [Certain, from the config]

Llama 3.1 uses the same ramp

Llama 3.1 ships with "rope_type": "llama3", factor 8, low_freq_factor 1, high_freq_factor 4 and an original length of 8,192, on top of base 500,000. [Certain, from the config] The rule in the reference code, written with wavelengths:

\[ \theta'_i = \begin{cases} \theta_i & \text{if } \lambda_i \lt L/4 \\ \theta_i / s & \text{if } \lambda_i \gt L \\ \text{linear blend in } L/\lambda_i & \text{otherwise} \end{cases} \tag{28.10} \]

Since \(L/\lambda_i = r_i\), this is exactly the NTK-by-parts rule (28.8) with \(\alpha = 1\) and \(\beta = 4\), and without the temperature. The code below checks that the two give identical numbers. For Llama 3.1: 29 pairs are unchanged, 29 are divided by 8, and 6 are blended. [Certain, computed from the Hugging Face implementation]

position interpolationNTK-awareYaRN (\(\alpha = 1,\ \beta = 32\))Llama 3.1 rule (\(\alpha = 1,\ \beta = 4\))
Figure 2. The factor \(\theta'_i / \theta_i\) for each of the 64 pairs of a head with \(d = 128\), on a log scale. A value of 1 means the pair is not changed; \(1/s\) means it is slowed by the full factor, as in PI. The dashed lines mark where the pairs make 32 turns, 4 turns and 1 turn during training.
How to read Figure 2

PI is a flat line at \(1/s\): every pair is slowed. NTK-aware is a straight diagonal from 1 (pair 0) to \(1/s\) (pair 63). YaRN and the Llama 3.1 rule are flat at 1 for fast pairs, flat at \(1/s\) for slow pairs, with a ramp in between; the Llama 3.1 ramp is shorter. Switch to the Llama 3.1 preset with \(s = 8\): the readout shows 29 unchanged, 6 blended and 29 scaled pairs. Where the blue NTK line sits above the orange YaRN line to the right of the “1 turn” mark, NTK lets slow pairs reach new angles.

Algorithm 1: new RoPE angle steps for a longer context
  1. input: head size \(d\), base \(b\), training length \(L\), target length \(L'\), method, ramp ends \(\alpha, \beta\)
  2. \(s \leftarrow L'/L\)
  3. for \(i = 0, \dots, d/2 - 1\): \(\theta_i \leftarrow b^{-2i/d}\), \(\;r_i \leftarrow L\theta_i / 2\pi\) // (28.2), (28.6)
  4. if method = PI: \(\theta'_i \leftarrow \theta_i / s\) // (28.4)
  5. if method = NTK-aware: \(\theta'_i \leftarrow (b\, s^{d/(d-2)})^{-2i/d}\) // (28.5)
  6. if method = YaRN or Llama 3.1: \(\gamma \leftarrow \operatorname{clip}((r_i - \alpha)/(\beta - \alpha), 0, 1)\); \(\;\theta'_i \leftarrow (1-\gamma)\theta_i/s + \gamma\theta_i\) // (28.7), (28.8)
  7. build the cos and sin tables for positions \(0, \dots, L'-1\) with \(\theta'\) // as in Session 14
  8. if method = YaRN: multiply both tables by \(0.1\ln s + 1\) // temperature (28.9)
  9. fine-tune briefly on long documents (hundreds to a few thousand steps), then test short and long tasks

Is a short fine-tune enough?

Changing the angles makes long inputs possible. It does not teach the model to use them. Compare the training budgets:

ModelMethodFrom → toLong-context training
LLaMA 7B to 65B (PI paper)Position interpolation2K → up to 32Kwithin 1,000 steps [Certain]
Llama 2 7B, 13B (YaRN paper)YaRN, \(s = 16\) then 324K → 64K → 128K400 + 200 steps, batch 64 [Certain]
Code LlamaBase raised to \(10^6\)trained at 16K, useful to about 100Kpart of code training [Certain]
Llama 3.1Llama 3 rule (28.10)8K → 128K in six stagesabout 800B tokens [Certain]
Qwen2.5-7B-InstructYaRN factor 4, added at deployment32K → about 128Knone needed per the model card [Certain]

The pattern: a few hundred steps are enough to make perplexity on long documents look good. Frontier labs still spend hundreds of billions of tokens, because good perplexity does not mean the model can find and combine information across 100K tokens. To see the difference we need better tests. [Likely]

Measuring long context: needles, the middle, and RULER

Why perplexity is not enough

Perplexity on a long document mostly measures how well the model predicts each token from the few hundred tokens before it. Natural text rarely needs a fact from 50,000 tokens back to predict the next word. So a model can have a fine long-document perplexity and still ignore almost everything far away. [Likely]

Needle in a haystack (NIAH)

In 2023 Greg Kamradt published a simple test that became the standard first check. [Certain] Take a long, unrelated text (the “haystack”, in the original a set of Paul Graham essays). Insert one sentence with a fact (the “needle”) at a chosen depth, from 0% (the start) to 100% (the end). At the end, ask a question that only the needle answers. Repeat over a grid of context lengths and depths, and draw a heat map of the scores. A model that “has” 128K context should be green everywhere.

Algorithm 2: a needle-in-a-haystack sweep
  1. input: haystack text, needle sentence, question, lengths \(\{N_1, \dots\}\), depths \(\{0\%, 10\%, \dots, 100\%\}\)
  2. for each length \(N\) and depth \(p\):
  3.   take the first \(N - n_{\text{needle}} - n_{\text{question}}\) tokens of the haystack
  4.   cut it at token \(\lfloor p \cdot (N - n_{\text{needle}} - n_{\text{question}}) \rfloor\), ideally at a sentence end, and insert the needle there
  5.   append the question; generate an answer with greedy decoding
  6.   score the answer (exact match or a judge model) and store it in cell \((N, p)\)
  7. draw the grid; report the longest \(N\) where every depth passes
Figure 3. The test grid of a needle-in-a-haystack sweep: 8 lengths times 11 depths = 88 prompts. Click a cell. The bar below shows the prompt: haystack, the needle at the chosen depth, and the question at the end. Cells are marked by whether the distance from the question back to the needle is longer than the training length. This figure shows the layout of the test, not measured scores.
How to read Figure 3

Each cell is one prompt. The default cell is 32K at depth 50%: the needle sits at token 16,384, so the question at the end must look 16,384 tokens back. For a model trained to 8K without any position change, every cell to the right of 8K needs relative distances it never saw in training (orange), and only small depths near the end stay under 8K. Switch to “Extended to 128K” and every cell is inside the trained range. Inside the range does not mean solved: real heat maps often show weak cells at middle depths, which is the next topic.

NIAH is useful, but easy. The needle is usually very different from the haystack (a sentence about San Francisco inside essays about startups), so the model can find it by simple matching, almost like a search. RULER reports that most long-context models get nearly perfect scores on vanilla NIAH and passkey retrieval, even models that fail harder tasks at the same length. [Certain] A green NIAH map is a minimum, not a proof.

Lost in the middle

Liu et al. (2023, published in TACL) asked a more realistic question. Give the model a question and \(k\) documents (10, 20 or 30), retrieved Wikipedia passages, where exactly one contains the answer. Move the answer document from the first slot to the last and measure accuracy. [Certain]

The result is a U-shaped curve. Accuracy is highest when the answer is at the start, a bit lower when it is at the end, and lowest in the middle. For GPT-3.5-Turbo with 20 documents, Figure 4 shows the published numbers. In the middle, the model did worse than with no documents at all (closed-book), even though the answer was in its context. [Certain] The same pattern appeared for models built for long contexts, and in a synthetic key-value retrieval task with 75, 140 and 300 pairs. [Certain]

accuracy with 20 documentsclosed-book (no documents): 56.1%oracle (only the answer document): 88.3%
Figure 4. Multi-document question answering, GPT-3.5-Turbo, 20 documents, from Liu et al. (2023), Table 6 and Table 1. Accuracy is 75.8% when the answer document is first, 53.8% when it is 10th, and 63.2% when it is last. [Certain]

Why does this happen? Two likely causes. First, training data: the important part of a text is often at the start (titles, instructions) or right before the next token, so models learn these habits. Second, causal attention: early tokens are seen by every later token and become strong anchors (the attention sinks of session 27), while recent tokens are favored by RoPE’s mild preference for near distances. [Likely] Practical advice that follows: put instructions and the most important documents at the start or the end of a long prompt, not in the middle. [Likely]

RULER: what is the real context size?

RULER (Hsieh et al., 2024) is a synthetic benchmark with 13 tasks in 4 groups, run at lengths 4K, 8K, 16K, 32K, 64K and 128K: [Certain]

  • Retrieval (8 tasks): NIAH variants with several keys, several values, several queries, and distracting needles.
  • Multi-hop tracing: variable tracking, “X1 = 12345, X2 = X1, X3 = X2, …, what has the value 12345?”.
  • Aggregation: find the most common or most frequent words in a long list.
  • Question answering: SQuAD and HotpotQA with many distracting paragraphs.

To turn scores into one number, RULER uses a threshold \(\tau\): the score of Llama 2 7B at 4K, which is 85.6%. The effective length is the largest tested length whose average score is still above it: [Certain]

\[ L_{\text{eff}} = \max\,\{\, N \in \{4\text{K}, 8\text{K}, \dots, 128\text{K}\} \;:\; \text{score}(N) \ge \tau \,\}, \qquad \tau = 85.6 \tag{28.11} \]

The headline result: of 17 models that all claim 32K tokens or more, only about half keep a satisfactory score at 32K. [Certain] The advertised window and the effective context are often far apart.

Figure 5. How RULER turns a score curve into an effective length, formula (28.11). The three models are made up for teaching; they all claim a 128K window. Green dots pass the threshold, orange dots fail.
How to read Figure 5

Model A passes up to 16K and fails from 32K on: effective length 16K, an eighth of its window. Model B stays above 85.6 up to 64K. Model C is strong at short lengths but falls quickly: effective length 8K. Move the threshold: a lower bar makes every model look longer. This is why you must always say which threshold, which tasks, and which lengths you used when you report an “effective context”.

Context, memory or retrieval?

A long context is one of three ways to give a model information that is not in its weights. They are not rivals; real systems mix them.

  • Context: put everything into the prompt. The model can attend to every token, and nothing has to be chosen in advance. Cost: KV memory grows linearly (28.1), prefill grows roughly with the square of the length, and quality drops in the middle (Figure 4).
  • Memory: the system writes notes (summaries, facts about the user, past decisions) to a store and loads a small, chosen part of them into later prompts. MemGPT (Packer et al., 2023) frames this like an operating system that moves data between a small “main memory” (the context) and a large “disk”. [Certain] Cost: what is not written down is lost, and summaries can drift.
  • Retrieval (RAG, Lewis et al., 2020): split a large collection into chunks, index them with an embedding model, and at query time put only the top \(k\) chunks into the context. [Certain] Cost: if the search misses the right chunk, the model never sees it; and questions that need the whole corpus (“summarize every contract”) do not fit the pattern.
Figure 6. Three ways to use the same information with Llama 3.1 8B (128 KiB of KV cache per token, a 131,072-token window). “Memory” is an example design that keeps a 2,048-token summary plus the last 4,096 tokens. Bars show tokens in the context on a log scale; the readout gives the KV cache and the attention scores per head and layer.
How to read Figure 6

At 128K tokens of information (the default), the whole thing just fits in the window: 16 GiB of KV cache for one request. Retrieval with 8 chunks uses 4,096 tokens: 512 MiB, 32 times less, and about 1,000 times fewer attention scores. Move the first slider past 128K: the full-context option no longer fits at all, while retrieval and memory stay the same size. Retrieval cost depends only on \(k\), not on the size of the collection; what grows with the collection is the index, and the risk that the search misses.

Long contextMemoryRetrieval (RAG)
Best forone long document, whole-input reasoning, code files that refer to each otherlong conversations, agents, personal facts across sessionslarge, changing collections; citations; facts updated daily
Main failurelost in the middle; cost per requestforgetting what was not saved; driftsearch misses the right chunk
Cost grows withinput length (linear memory, quadratic prefill)size of the kept summary\(k\) times chunk size; index size offline
Freshnesswhatever is pastedwhat was written downre-index and it is up to date

In code

The PyTorch code below builds the new angle steps for every method, checks the claims made above (NTK-aware leaves pair 0 alone and slows the last pair by \(s\); the Llama 3.1 rule gives 29 / 29 / 6 pairs; all three methods keep the slowest pair inside its training range), prints the YaRN temperature, and computes a RULER-style effective length. It runs on a CPU in under a second and prints the same output every time.

import math, torch

def inv_freq(d=128, base=10_000.0):
    # theta_i = base^(-2i/d), formula (28.2)
    return base ** (-torch.arange(0, d, 2, dtype=torch.float64) / d)

def scaled_inv_freq(method, d, base, L, s, alpha=1.0, beta=32.0):
    th = inv_freq(d, base)
    if method == "none":
        return th
    if method == "pi":                         # (28.4): every pair turns s times slower
        return th / s
    if method == "ntk":                        # (28.5): raise the base
        return inv_freq(d, base * s ** (d / (d - 2)))
    if method in ("yarn", "llama3"):           # (28.6)-(28.8): by parts, with a ramp
        if method == "llama3":
            alpha, beta = 1.0, 4.0             # low_freq_factor, high_freq_factor
        r = L / (2 * math.pi / th)             # turns during training
        g = ((r - alpha) / (beta - alpha)).clamp(0, 1)
        return (1 - g) * th / s + g * th
    raise ValueError(method)

d, L, s = 128, 4096, 4
th = inv_freq(d)
for m in ["pi", "ntk", "yarn"]:
    ratio = scaled_inv_freq(m, d, 10_000.0, L, s) / th
    print(f"{m:5s} fastest pair x{ratio[0].item():.3f}  slowest pair x{ratio[-1].item():.4f}")

# Llama 3.1: base 500,000, original length 8,192, factor 8
th3 = inv_freq(128, 500_000.0)
new = scaled_inv_freq("llama3", 128, 500_000.0, 8192, 8)
ratio = new / th3
print("unchanged:", int((ratio > 1 - 1e-9).sum()), " divided by 8:", int((ratio < 1 / 8 + 1e-9).sum()),
      " blended:", int(((ratio < 1 - 1e-9) & (ratio > 1 / 8 + 1e-9)).sum()))

# YaRN attention temperature, formula (28.9): logits are multiplied by (0.1 ln s + 1)^2
for s_ in [4, 16, 32]:
    print(s_, round((0.1 * math.log(s_) + 1) ** 2, 3))

# The largest angle each method reaches at the new length L' = s L, for the slowest pair
Lp = s * L
for m in ["none", "pi", "ntk", "yarn"]:
    a = Lp * scaled_inv_freq(m, d, 10_000.0, L, s)[-1].item()
    print(f"{m:5s} slowest pair: max angle {a:.3f} rad (training saw up to {L * th[-1].item():.3f})")

# RULER-style effective length, formula (28.11): largest tested length that passes
def effective_length(scores, threshold=85.6):
    ok = [n for n, sc in sorted(scores.items()) if sc >= threshold]
    return max(ok) if ok else 0
toy = {4096: 94.0, 8192: 92.1, 16384: 88.0, 32768: 79.5, 65536: 70.2, 131072: 51.0}   # made-up scores
print("effective length:", effective_length(toy))

# Output:
# pi    fastest pair x0.250  slowest pair x0.2500
# ntk   fastest pair x1.000  slowest pair x0.2500
# yarn  fastest pair x1.000  slowest pair x0.2500
# unchanged: 29  divided by 8: 29  blended: 6
# 4 1.296 / 16 1.631 / 32 1.813   (one per line)
# none  slowest pair: max angle 1.892 rad (training saw up to 0.473)
# pi, ntk, yarn: max angle 0.473 rad (training saw up to 0.473)
# effective length: 16384

To use one of these in a real model you do not write this yourself: set rope_scaling in the config (Hugging Face) or the matching flag in vLLM, and the library builds the tables. Writing it once by hand is still the best way to understand what the config numbers do.

Where this is used

ApplicationWhat “context” must holdTypical approach
Long-document QA (contracts, reports)one document of 50K to 200K tokensextended RoPE model (Llama 3.1 rule, YaRN); important instructions at the start and end
Code assistants over a repositorymany files that refer to each otherlarge base (Code Llama), retrieval of relevant files, then a long context
Chat assistants and agentslong histories, tool outputsmemory (summaries, saved facts) plus a moderate window (session 154)
Enterprise search, support botsmillions of pages that change dailyRAG with top-\(k\) chunks (session 152)
Long-video question answeringan hour of video is tens of thousands of visual tokens or more, even after pooling [Likely]frame sampling and token pooling, extended M-RoPE positions, or Video-RAG over clips (session 206)
Key-moment detection, highlight searchwhere in a long video an event happens“needle in a video haystack” tests: insert a clip and ask about it, the video version of NIAH [Likely]
Video retrieval and Video-RAGa library of videosretrieve clips by embedding, then reason over the few retrieved clips (session 210)
Training at long lengthssequences of 128K or more in trainingcontext parallelism splits one sequence over GPUs (session 49)

Common mistakes

  • Trusting max_position_embeddings. It is a setting. Check the training length and run a long-context test on your own task.
  • Using PI with a huge factor and no fine-tuning. Slowing every pair by 32 blurs local order. Use YaRN or an NTK-style method, and fine-tune.
  • Applying static scaling to every request. A fixed YaRN factor can hurt short inputs (Qwen2.5 card). Turn it on only when needed, or use dynamic scaling, and re-run your short-text evaluations.
  • Mismatched training and inference settings. If you fine-tuned with YaRN factor 16, serve with factor 16 and the same original length. A different original_max_position_embeddings moves every ramp boundary.
  • Forgetting the YaRN temperature when you re-implement it in your own kernel. Without the \(0.1\ln s + 1\) factor, attention is flatter than the model was tuned for.
  • Calling a model “128K” from a green NIAH map. NIAH is easy. Test multi-hop and aggregation (RULER) and your real task at the lengths you serve.
  • Putting the key document in the middle. In long prompts, place instructions and the most relevant chunks at the start or end.
  • Ignoring the KV budget. 128K tokens of Llama 3.1 8B is 16 GiB per sequence. Concurrency, not quality, is often what limits long context in production.

Interview questions

What is the difference between context window, training length and effective context?basic
The context window is the largest input the code accepts (max_position_embeddings). The training length is the longest sequence the model actually trained on; positions past it are unfamiliar. The effective context is the longest input on which the model still does a given task well, measured for example by RULER’s threshold. Usually \(\text{effective} \le \text{training} \le \text{window}\), and the gaps can be large: RULER found that only about half of 17 models that claim 32K or more stay above threshold at 32K.
Why does a RoPE model fail past its training length?core
Pair \(i\) rotates by \(m\theta_i\). Fast pairs (wavelength \(\lambda_i \le L\)) went around the full circle in training, so they have seen every angle. Slow pairs (\(\lambda_i \gt L\)) only saw angles from 0 to \(L\theta_i\). Past \(L\) they produce angles never seen, and the attention scores built on them are out of distribution. For Llama 2 (\(d = 128\), \(b = 10^4\), \(L = 4{,}096\)) that is 18 of 64 pairs.
Explain position interpolation. Why does it work better than extrapolation, and what does it cost?core
PI replaces position \(m\) by \(m/s\) with \(s = L'/L\), so every \(\theta_i\) is divided by \(s\) and the largest angle at \(L'\) equals the largest angle in training. The model only meets angles inside the range it knows (interpolation), and the PI paper shows the error bound is at least about 600 times smaller than for extrapolation. The cost: fast pairs are slowed too, so neighbor tokens differ by \(s\) times smaller angles and local order gets blurry. It needs a short fine-tune (under 1,000 steps in the paper) and degrades at large \(s\).
What does NTK-aware scaling change, and why is the exponent \(d/(d-2)\)?core
It changes the base: \(b' = b\, s^{d/(d-2)}\). Then \(\theta'_i = \theta_i\, s^{-2i/(d-2)}\): pair 0 is unchanged and the last pair \(i = d/2 - 1\) gets exactly \(\theta/s\), because \(2i/(d-2) = 1\) there. In between, the slow-down is geometric. It keeps local detail and works reasonably without fine-tuning, but intermediate slow pairs are not slowed enough: for Llama 2 at \(s = 4\), 17 of the 18 slow pairs still reach new angles.
Describe YaRN’s two parts.advanced
(1) NTK-by-parts: compute \(r_i = L/\lambda_i\), the turns per training sequence. Pairs with \(r_i \gt \beta\) (32 for Llama) are kept, pairs with \(r_i \lt \alpha\) (1) are divided by \(s\) like PI, and a linear ramp blends the ones in between: \(\theta'_i = (1-\gamma)\theta_i/s + \gamma\theta_i\). (2) An attention temperature: multiply \(\mathbf{q}\) and \(\mathbf{k}\) by \(0.1\ln s + 1\) (logits by its square, 1.631 at \(s = 16\)) to counter the flatter softmax at longer lengths. It is implemented by scaling the cos/sin cache, so it works with any kernel. YaRN reached 128K on Llama 2 with 400 + 200 fine-tuning steps.
How does Llama 3.1’s rope_scaling relate to YaRN?advanced
The “llama3” rule keeps pairs with wavelength below \(L/4\), divides pairs with wavelength above \(L\) by the factor 8, and blends linearly in \(L/\lambda\) in between, with \(L = 8{,}192\). Since \(L/\lambda_i = r_i\), it is NTK-by-parts with \(\alpha = 1\), \(\beta = 4\) and no temperature. With base 500,000 and \(d = 128\): 29 pairs unchanged, 6 blended, 29 scaled. Meta also did about 800B tokens of staged long-context training; the rule alone is not the whole story.
Why is needle-in-a-haystack not a sufficient long-context test?core
The needle is usually lexically very different from the haystack, so the model can locate it with simple matching, and only one fact must be retrieved. Real tasks need several facts, multi-step tracing, aggregation, and ignoring similar distractors. RULER shows that most models score nearly perfectly on vanilla NIAH yet drop sharply on its harder tasks as length grows. NIAH is a necessary sanity check, not proof of an effective context.
What is “lost in the middle”, and what would you do about it in a product?core
Liu et al. found a U-shaped accuracy curve: models use information best at the start and end of the context and worst in the middle. GPT-3.5-Turbo with 20 documents: 75.8% with the answer first, 53.8% at position 10, 63.2% last, versus 56.1% with no documents. In a product: keep prompts short with better retrieval and reranking, place the most relevant chunks and the instructions at the beginning or end, repeat the question after long inputs, and evaluate position sensitivity on your own data.
How does RULER define effective context length?basic
It averages the 13 task scores at each tested length (4K to 128K) and compares them with a threshold: the score of Llama 2 7B at 4K, 85.6%. The effective length is the largest tested length whose score is at least the threshold. It is a relative, task-dependent number; you should state the tasks and threshold when you quote it.
You have 50 million tokens of company documents. Long context, memory or RAG?core
50M tokens do not fit any window, and even 128K tokens cost 16 GiB of KV cache per request for an 8B model. Use RAG: chunk, embed, index, retrieve top-\(k\) (plus a reranker), and give the model a moderate context of retrieved chunks; a longer window then lets you include more candidates and reduces the chance of a miss. Add memory for per-user facts across sessions. Long context alone is the right tool only when one input (one contract, one codebase slice) must be read as a whole.
Why does raising the RoPE base help long contexts, and what is the trade-off?advanced
A larger base makes the slow pairs slower (longer wavelengths), so a longer training length still lies within their first turn and far distances remain distinguishable. Llama 3 uses 500,000 and Code Llama 1,000,000. The trade-off is fewer fast pairs, so a bit less resolution at mid distances, and it must be set before (or during) training; changing it after training without fine-tuning is the NTK-aware trick and only partly works.

Exercises

  1. A model has 40 layers, 8 KV heads, head size 128, BF16. How large is the KV cache for one 32K-token (32,768) sequence?
  2. For \(d = 128\) and \(b = 10{,}000\), which pairs are “slow” for \(L = 8{,}192\)? (Hint: \(\lambda_i = 2\pi \cdot 10{,}000^{i/64}\gt 8{,}192\).) Check with the Figure 1 slider logic.
  3. Compute the NTK-aware base for \(b = 10{,}000\), \(d = 128\), \(s = 8\).
  4. With YaRN at \(s = 8\), what number multiplies the attention logits?
  5. For Llama 2 (\(L = 4{,}096\)), pair 30 has \(r = 8.69\). With \(\alpha = 1\), \(\beta = 32\) and \(s = 4\), what is \(\theta'_{30}/\theta_{30}\)?
  6. A made-up model scores 93, 90, 86, 84, 87, 60 at 4K, 8K, 16K, 32K, 64K, 128K. What is its RULER effective length with \(\tau = 85.6\)? What is strange about this result?
Answerscheck
1. Per token \(2 \cdot 40 \cdot 8 \cdot 128 \cdot 2 = 163{,}840\) bytes = 160 KiB; times 32,768 = 5 GiB. 2. \(\lambda_i \gt 8{,}192\) means \(10{,}000^{i/64} \gt 1{,}303.8\), so \(i \gt 64 \log_{10} 1{,}303.8 / 4 = 49.8\): pairs 50 to 63, that is 14 pairs. 3. \(b' = 10{,}000 \cdot 8^{128/126} = 82{,}685\) (rounded). 4. \((0.1 \ln 8 + 1)^2 = 1.208^2 = 1.459\). 5. \(\gamma = (8.69 - 1)/31 = 0.248\), so the ratio is \((1 - 0.248)/4 + 0.248 = 0.436\). 6. By formula (28.11) it is 64K, the largest length that passes, even though 32K fails. A non-monotone curve like this usually means noise or a task mix that is uneven across lengths; report the whole curve, not only one number.

Key takeaways

  • “Context length” means three things: the window (a setting), the training length (what the model saw), and the effective context (what it can really use for a task). They often differ a lot.
  • Long context costs memory: KV bytes \(= 2 \, n_{\text{layers}} n_{\text{kv}} d_{\text{head}} \cdot \text{bytes} \cdot T\) (28.1), 16 GiB per 128K sequence for Llama 3.1 8B.
  • RoPE breaks past \(L\) because slow pairs (\(\lambda_i \gt L\)) reach angles never seen in training. Every fix tries to keep \(L'\theta'_i \le L\theta_i\) for them (28.3).
  • PI divides all \(\theta_i\) by \(s\) (28.4): safe but blurs local order. NTK-aware raises the base (28.5): keeps fast pairs but under-scales some slow ones. YaRN ramps by \(r_i = L/\lambda_i\) (28.6 to 28.8) and adds a temperature (28.9). Llama 3.1’s rule is the same ramp with \(\alpha = 1, \beta = 4\) (28.10).
  • Test with more than NIAH: lost in the middle shows a U-shaped curve; RULER’s effective length (28.11) shows that about half of the models claiming 32K or more do not hold up at 32K.
  • Context, memory and retrieval are complementary. Retrieve to narrow, use a long context to read, and keep memory for what must last across sessions.

Key formulas

\[ \theta_i = b^{-2i/d}, \quad \text{PI: } \theta'_i = \frac{\theta_i}{s}, \quad \text{NTK: } b' = b\, s^{\frac{d}{d-2}} \]
\[ r_i = \frac{L}{\lambda_i}, \quad \theta'_i = (1-\gamma(r_i))\frac{\theta_i}{s} + \gamma(r_i)\,\theta_i, \quad \sqrt{1/t} = 0.1\ln s + 1 \]
\[ \text{KV} = 2\, n_{\text{layers}}\, n_{\text{kv}}\, d_{\text{head}} \cdot \text{bytes} \cdot T, \qquad L_{\text{eff}} = \max\{N : \text{score}(N) \ge \tau\} \]

References for this session

  1. Chen et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation
  2. Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models
  3. Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts
  4. Hsieh et al. (2024). RULER: What’s the Real Context Size of Your Long-Context Language Models?
  5. Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
  6. Llama Team, Meta (2024). The Llama 3 Herd of Models
  7. bloc97 (Reddit r/LocalLLaMA) (2023). NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation
  8. Rozière et al. (2023). Code Llama: Open Foundation Models for Code
  9. Kamradt (2023). Needle In A Haystack: pressure testing LLMs
  10. Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  11. Packer et al. (2023). MemGPT: Towards LLMs as Operating Systems
  12. Xiao et al. (2023). Efficient Streaming Language Models with Attention Sinks
  13. Jiang et al. (2023). Mistral 7B