What “context” means, and how to make it longer
A model card may say “128K context”, but that number is a setting, not a skill. This session separates the context window, the training length and the effective context, shows why RoPE fails past its training length and how position interpolation, NTK-aware scaling and YaRN fix it, and explains how needle-in-a-haystack, “lost in the middle” and RULER measure what a model can really use. It ends with the choice between long context, memory and retrieval.
What you will learn
- The difference between the context window, the training length and the effective context, with real model configs
- What limits long context: positions, KV memory and compute, data, and attention itself; the KV-cache formula with Llama 3.1 8B numbers
- Why RoPE’s slow dimension pairs reach never-seen angles past the training length
- Position interpolation, NTK-aware base scaling and YaRN (NTK-by-parts plus a temperature), and why Llama 3.1’s rope_scaling is the same ramp
- How to test long context: needle-in-a-haystack, the lost-in-the-middle U-curve, and RULER’s effective length
- When to use a long context, memory or retrieval (RAG), and how to combine them
Why this matters
Model cards say “128K context”. Users paste a 300-page contract and ask about page 150. Sometimes the answer is right. Sometimes the model ignores the middle of the document and answers from the first and last pages. This session explains why, and what engineers do about it.
The context of a model is the text it can see while it makes the next token: the system prompt, the chat history, pasted documents, retrieved passages, tool outputs, image and video tokens. Everything the model “knows” about your task right now must be inside it. So the size of the context, and how well the model really uses it, decides what you can build: long-document question answering, code assistants that read a whole repository, agents with long histories, and video models that watch an hour of frames.
Three questions come up again and again, in real systems and in interviews:
- What does “context length” even mean? There are at least three different numbers: the window the code accepts, the length the model was trained on, and the length it can really use. They are often far apart.
- How do you make a RoPE model work past its training length? In session 14 we saw that RoPE angles beyond the training length are new to the model. Position interpolation, NTK-aware scaling and YaRN change the angles so the model only sees familiar ones. Llama 3.1, Qwen2.5 and Code Llama all use one of these ideas. [Certain]
- How do you test it? Needle-in-a-haystack, the “lost in the middle” study, and the RULER benchmark show that a long window is not the same as long understanding.
The last section steps back. A long context is only one way to give a model more information. The other two are memory (store and recall facts across sessions) and retrieval (search a large store and put only the best pieces into the context). Choosing between them is a system-design question you will meet in session 152 and session 206.
Three numbers called “context length”
When someone says “this model has a 32K context”, they could mean three different things. Keep them apart:
- Context window (also “maximum context”): the largest number of tokens the code and the serving stack will accept. In a Hugging Face model this is usually
max_position_embeddingsinconfig.json. It is a setting, not a skill. - Training length \(L\): the longest sequences the model actually saw during (pre- or continued) training. Positions up to \(L\) are familiar. Positions past \(L\) were never seen.
- Effective context: the longest input on which the model still does a given task well. It depends on the task. Finding one fact is easy; combining ten facts spread over the input is hard.
Usually effective context \(\le\) training length \(\le\) context window, but real configs show how loose the link can be:
- Mistral 7B v0.1:
max_position_embeddingsis 32,768, but the model uses sliding-window attention with a window of 4,096 tokens, so no single layer looks back more than 4,096 tokens (session 27). [Certain, from the config] - Llama 3.1 8B: the window is 131,072 tokens. The base model was pretrained at 8,192 and then extended in six stages to 128K, using about 800 billion tokens of long-context training. Meta moved to the next stage only when short-context scores had fully recovered and needle-in-a-haystack was solved at that length. [Certain]
- Code Llama: trained on 16K-token sequences with a RoPE base of 1,000,000, and shows gains on inputs up to 100K tokens, so here the useful length is above the training length. [Certain]
- Qwen2.5-7B-Instruct: the config says 32,768; the model card says to add a YaRN
rope_scalingentry with factor 4 to go to about 128K. [Certain, from the model card]
What limits the context?
Four different things stop a model from using very long inputs. Each one has its own fixes, and this course treats them in different places:
- Positions. Past the training length, RoPE produces angles the model never saw. Fixes: change the angles (this session), then fine-tune a little.
- Memory and compute. The KV cache grows with every token, and prefill attention grows with the square of the length (session 80, session 124).
- Data. Few training documents are really long, so the model has little practice using far-away information. Fixes: up-sample long documents and build synthetic long tasks during continued training. [Likely]
- Attention itself. Softmax spreads a fixed budget of weight over more and more tokens. The model must learn to focus on the few that matter, and it often prefers the start and the end (Section “Lost in the middle”).
The memory cost is easy to compute. For each token, every layer stores one key and one value vector for each key-value head:
- Read the symbols
- \(2\)
- one key and one value
- \(n_{\text{layers}}, n_{\text{kv}}, d_{\text{head}}\)
- number of layers, number of key-value heads (fewer than query heads with GQA), and the size of one head
- \(T\)
- the number of tokens in the context
From the config: 32 layers, 8 key-value heads, head size 128, stored in BF16 (2 bytes). Per token: \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes \(= 128\) KiB.
At 8,192 tokens that is 1 GiB. At the full 131,072 tokens it is 16 GiB for one sequence, on top of the 16 GB of weights. A 1-million-token context would need about 122 GiB. And prefill must compute about \(131{,}072^2 / 2 \approx 8.6 \times 10^{9}\) attention scores per head per layer. Long context is expensive even when the model handles it well.
Why RoPE breaks past the training length
Recall RoPE from session 14. A head of size \(d\) is cut into \(d/2\) pairs. Pair \(i\) of the query (and key) at position \(m\) is rotated by the angle \(m\theta_i\):
- Read the symbols
- \(\varphi_i(m)\)
- the rotation angle of pair \(i\) at position \(m\), in radians
- \(\theta_i\)
- how many radians pair \(i\) turns per token; pair 0 turns 1 radian, the last pair turns very slowly
- \(b\)
- the base: 10,000 in Llama 2, 500,000 in Llama 3 [Certain]
- \(\lambda_i\)
- the wavelength: after \(\lambda_i\) tokens pair \(i\) has turned one full circle
Now look at what each pair saw during training with sequences up to length \(L\). The angle of pair \(i\) went from 0 up to \(L\theta_i\). Two cases:
- Fast pairs (\(\lambda_i \le L\)). They turned at least one full circle inside the training length, so they have seen every possible angle many times. A longer input gives them no new angle. They are safe.
- Slow pairs (\(\lambda_i \gt L\)). They only covered an arc from 0 to \(L\theta_i \lt 2\pi\). At a longer input of length \(L'\), they reach angles up to \(L'\theta_i\), which the model has never seen. The attention scores built on them are then unpredictable.
For Llama 2 (\(d = 128\), \(b = 10{,}000\), \(L = 4{,}096\)) there are 18 slow pairs out of 64. The slowest one, pair 63, has a wavelength of 54,410 tokens; in training it only reached 0.473 rad (27°). At 16,384 tokens it would reach 1.892 rad. That is the problem in one number. [Certain, computed]
We can write the goal of every method in this session as one rule. A method changes the angle steps from \(\theta_i\) to new values \(\theta'_i\). For the new length \(L' = sL\), we want:
In words: the largest angle used at the new length must not go past the largest angle seen in training. The number \(s = L'/L\) is the scale factor. Fast pairs do not need the rule, because they already saw the whole circle.
How to read Figure 1
Start with “No change”, pair 47 and \(s = 4\). Training covered only 4.73 rad of the circle; at 16,384 tokens the pair goes to 18.92 rad, so most of the inner ring is orange. Move the pair slider below 46: the pair has a wavelength shorter than 4,096, the gray ring is full, and there is nothing new to see. Now press “Position interpolation”: every pair is slowed by \(s\), the orange part disappears, but the readout shows each token step is 4 times smaller. Press “NTK-aware” at pair 47: some orange stays (6.73 rad vs 4.73). YaRN removes it. The readout also counts, over all 64 pairs, how many reach new angles.
Position interpolation (PI)
The first fix is very simple. Chen et al. (2023) proposed to squeeze the new positions into the old range. Instead of position \(m\), use position \(m/s\):
Every pair now turns \(s\) times slower. The largest angle at \(L'\) is \(L' \theta_i / s = L\theta_i\): exactly the training range, for every pair, so rule (28.3) holds. Positions become fractional (token 5 sits at “position 1.25” when \(s = 4\)), but the model has seen all the angles around them. Interpolating between known points is much safer than extrapolating beyond them.
The paper extended LLaMA models from 7B to 65B parameters to windows of up to 32,768 tokens with fewer than 1,000 fine-tuning steps. It also showed that the upper bound on the error of interpolated attention scores is at least about 600 times smaller than for extrapolation. [Certain] Without any fine-tuning, PI already works better than plain extrapolation, but a short fine-tune is what makes it good. [Likely]
In Hugging Face this is rope_scaling = {"rope_type": "linear", "factor": s}. [Certain]
The price of PI: crowded neighbors
PI slows down the fast pairs too, and they never needed it. With \(s = 4\), pair 0 turns 0.25 rad per token instead of 1 rad. The angle difference between neighbors, which the model uses to tell “the token right before me” from “two tokens back”, becomes 4 times smaller. With \(s = 32\) it is 32 times smaller. Local order gets blurry, and fine-tuning has to teach the model to read these smaller differences. This is why PI loses some quality on short texts and why larger factors work worse. [Likely]
NTK-aware scaling: change the base instead
In June 2023, a user called bloc97 posted on Reddit a different idea, called “NTK-aware” scaling (the name comes from neural tangent kernel theory, which says networks have trouble learning high-frequency details from few inputs). [Certain] The idea: do not slow down every pair by the same factor. Keep the fast pairs (high frequency) as they are, slow the slowest pair by the full factor \(s\), and scale the pairs in between smoothly. All of this happens by changing only the base:
- Read the symbols
- \(b'\)
- the new base
- \(s^{-2i/(d-2)}\)
- the slow-down of pair \(i\): equal to 1 for pair 0 (no change) and to \(1/s\) for the last pair \(i = d/2 - 1\)
Check the last pair: \(2i/(d-2) = (d-2)/(d-2) = 1\), so \(\theta'_{d/2-1} = \theta_{d/2-1}/s\). The exponent \(d/(d-2)\) is chosen exactly for that. In between, the slow-down follows a straight line on a log scale.
\(d = 128\), \(b = 10{,}000\), \(s = 4\). The new base is \(b' = 10{,}000 \cdot 4^{128/126} = 40{,}890\).
- Pair 0 (wavelength 6.3 tokens): PI slows it to 0.25 rad per token. NTK-aware keeps 1 rad per token.
- Pair 10 (wavelength 26.5): PI multiplies \(\theta\) by 0.25, NTK-aware by 0.80.
- Pair 63 (wavelength 54,410): both multiply by 0.25. The largest angle at 16,384 tokens is 0.473 rad, the same as in training.
- Pair 47 (wavelength 5,441, a slow pair): NTK-aware multiplies by 0.356, so at 16,384 tokens the angle reaches 6.73 rad, but training only went to 4.73 rad. This pair still extrapolates.
Counting all 18 slow pairs, 17 still reach never-seen angles under NTK-aware scaling at \(s = 4\); only the last one is fully inside. NTK-aware keeps local detail, but it does not fully meet rule (28.3). [Certain, computed]
NTK-aware scaling worked surprisingly well without any fine-tuning, which made it popular in 2023. [Likely] A variant called dynamic NTK recomputes the base at inference from the current length, so short inputs are not changed at all and long inputs are scaled just enough (Hugging Face rope_type = "dynamic"). [Certain]
Raising the base during training
The same lever is used at training time. If you plan long contexts, you can simply train with a large base from the start, or raise it before long-context training. Code Llama uses \(b = 1{,}000{,}000\) and Llama 3 uses 500,000. [Certain] A larger base makes the slow pairs slower, so a longer \(L\) is still inside their first turn. This is often called ABF (adjusted base frequency). [Likely]
YaRN: treat each pair by how much it turned
YaRN (Peng et al., 2023) combines the two ideas and adds one more. It has two parts. [Certain]
Part 1: “NTK-by-parts”
First, measure how many full turns pair \(i\) makes inside the training length:
If \(r_i\) is large, the pair turned many times in training and is safe: leave it alone. If \(r_i \lt 1\), the pair never completed one turn: interpolate it fully, exactly like PI. In between, blend the two with a linear ramp:
- Read the symbols
- \(r_i\)
- the number of full turns pair \(i\) made in a training sequence of length \(L\)
- \(\gamma\)
- how much of the original speed to keep: 1 means “do not touch”, 0 means “divide by \(s\)”
- \(\alpha, \beta\)
- the two ends of the ramp. For the Llama family the paper uses \(\alpha = 1\) and \(\beta = 32\), found by experiment. [Certain]
Unlike NTK-aware scaling, every pair with \(r_i \lt 1\) (all slow pairs) is fully interpolated, so rule (28.3) holds exactly for them. Unlike PI, every pair with \(r_i \gt 32\) keeps its full speed, so local order stays sharp. For Llama 2 with \(L = 4{,}096\): 21 pairs are kept, 18 are fully interpolated and 25 are blended. These counts do not depend on \(s\), because \(r_i\) only uses the training length. [Certain, computed]
Part 2: a temperature on the attention logits
Interpolation has a side effect: at a longer length, the softmax spreads its weight over many more tokens, and the attention gets flatter (higher entropy). YaRN makes the logits a little sharper by multiplying the query and the key by a factor, which is the same as dividing the logits by a temperature \(t\):
For \(s = 4\), each of \(\mathbf{q}\) and \(\mathbf{k}\) is multiplied by 1.139, so the logits are multiplied by 1.296. For \(s = 16\): 1.631. For \(s = 32\): 1.813. The formula was fitted on Llama models. [Certain] In code, it costs nothing: Hugging Face multiplies the cached cos and sin tables by \(0.1 \ln s + 1\), so it works with any attention kernel. [Certain]
With both parts, YaRN extended Llama 2 7B and 13B to 64K (\(s = 16\), 400 fine-tuning steps at batch size 64 on 64K-token chunks of PG19) and then to 128K (\(s = 32\), 200 more steps). The authors report 10 times fewer tokens and 2.5 times fewer training steps than earlier methods. [Certain] The released 128K model has rope_scaling = {"type": "yarn", "factor": 32, "original_max_position_embeddings": 4096}. [Certain, from the config]
Llama 3.1 uses the same ramp
Llama 3.1 ships with "rope_type": "llama3", factor 8, low_freq_factor 1, high_freq_factor 4 and an original length of 8,192, on top of base 500,000. [Certain, from the config] The rule in the reference code, written with wavelengths:
Since \(L/\lambda_i = r_i\), this is exactly the NTK-by-parts rule (28.8) with \(\alpha = 1\) and \(\beta = 4\), and without the temperature. The code below checks that the two give identical numbers. For Llama 3.1: 29 pairs are unchanged, 29 are divided by 8, and 6 are blended. [Certain, computed from the Hugging Face implementation]
How to read Figure 2
PI is a flat line at \(1/s\): every pair is slowed. NTK-aware is a straight diagonal from 1 (pair 0) to \(1/s\) (pair 63). YaRN and the Llama 3.1 rule are flat at 1 for fast pairs, flat at \(1/s\) for slow pairs, with a ramp in between; the Llama 3.1 ramp is shorter. Switch to the Llama 3.1 preset with \(s = 8\): the readout shows 29 unchanged, 6 blended and 29 scaled pairs. Where the blue NTK line sits above the orange YaRN line to the right of the “1 turn” mark, NTK lets slow pairs reach new angles.
- input: head size \(d\), base \(b\), training length \(L\), target length \(L'\), method, ramp ends \(\alpha, \beta\)
- \(s \leftarrow L'/L\)
- for \(i = 0, \dots, d/2 - 1\): \(\theta_i \leftarrow b^{-2i/d}\), \(\;r_i \leftarrow L\theta_i / 2\pi\) // (28.2), (28.6)
- if method = PI: \(\theta'_i \leftarrow \theta_i / s\) // (28.4)
- if method = NTK-aware: \(\theta'_i \leftarrow (b\, s^{d/(d-2)})^{-2i/d}\) // (28.5)
- if method = YaRN or Llama 3.1: \(\gamma \leftarrow \operatorname{clip}((r_i - \alpha)/(\beta - \alpha), 0, 1)\); \(\;\theta'_i \leftarrow (1-\gamma)\theta_i/s + \gamma\theta_i\) // (28.7), (28.8)
- build the cos and sin tables for positions \(0, \dots, L'-1\) with \(\theta'\) // as in Session 14
- if method = YaRN: multiply both tables by \(0.1\ln s + 1\) // temperature (28.9)
- fine-tune briefly on long documents (hundreds to a few thousand steps), then test short and long tasks
Is a short fine-tune enough?
Changing the angles makes long inputs possible. It does not teach the model to use them. Compare the training budgets:
| Model | Method | From → to | Long-context training |
|---|---|---|---|
| LLaMA 7B to 65B (PI paper) | Position interpolation | 2K → up to 32K | within 1,000 steps [Certain] |
| Llama 2 7B, 13B (YaRN paper) | YaRN, \(s = 16\) then 32 | 4K → 64K → 128K | 400 + 200 steps, batch 64 [Certain] |
| Code Llama | Base raised to \(10^6\) | trained at 16K, useful to about 100K | part of code training [Certain] |
| Llama 3.1 | Llama 3 rule (28.10) | 8K → 128K in six stages | about 800B tokens [Certain] |
| Qwen2.5-7B-Instruct | YaRN factor 4, added at deployment | 32K → about 128K | none needed per the model card [Certain] |
The pattern: a few hundred steps are enough to make perplexity on long documents look good. Frontier labs still spend hundreds of billions of tokens, because good perplexity does not mean the model can find and combine information across 100K tokens. To see the difference we need better tests. [Likely]
Measuring long context: needles, the middle, and RULER
Why perplexity is not enough
Perplexity on a long document mostly measures how well the model predicts each token from the few hundred tokens before it. Natural text rarely needs a fact from 50,000 tokens back to predict the next word. So a model can have a fine long-document perplexity and still ignore almost everything far away. [Likely]
Needle in a haystack (NIAH)
In 2023 Greg Kamradt published a simple test that became the standard first check. [Certain] Take a long, unrelated text (the “haystack”, in the original a set of Paul Graham essays). Insert one sentence with a fact (the “needle”) at a chosen depth, from 0% (the start) to 100% (the end). At the end, ask a question that only the needle answers. Repeat over a grid of context lengths and depths, and draw a heat map of the scores. A model that “has” 128K context should be green everywhere.
- input: haystack text, needle sentence, question, lengths \(\{N_1, \dots\}\), depths \(\{0\%, 10\%, \dots, 100\%\}\)
- for each length \(N\) and depth \(p\):
- take the first \(N - n_{\text{needle}} - n_{\text{question}}\) tokens of the haystack
- cut it at token \(\lfloor p \cdot (N - n_{\text{needle}} - n_{\text{question}}) \rfloor\), ideally at a sentence end, and insert the needle there
- append the question; generate an answer with greedy decoding
- score the answer (exact match or a judge model) and store it in cell \((N, p)\)
- draw the grid; report the longest \(N\) where every depth passes
How to read Figure 3
Each cell is one prompt. The default cell is 32K at depth 50%: the needle sits at token 16,384, so the question at the end must look 16,384 tokens back. For a model trained to 8K without any position change, every cell to the right of 8K needs relative distances it never saw in training (orange), and only small depths near the end stay under 8K. Switch to “Extended to 128K” and every cell is inside the trained range. Inside the range does not mean solved: real heat maps often show weak cells at middle depths, which is the next topic.
NIAH is useful, but easy. The needle is usually very different from the haystack (a sentence about San Francisco inside essays about startups), so the model can find it by simple matching, almost like a search. RULER reports that most long-context models get nearly perfect scores on vanilla NIAH and passkey retrieval, even models that fail harder tasks at the same length. [Certain] A green NIAH map is a minimum, not a proof.
Lost in the middle
Liu et al. (2023, published in TACL) asked a more realistic question. Give the model a question and \(k\) documents (10, 20 or 30), retrieved Wikipedia passages, where exactly one contains the answer. Move the answer document from the first slot to the last and measure accuracy. [Certain]
The result is a U-shaped curve. Accuracy is highest when the answer is at the start, a bit lower when it is at the end, and lowest in the middle. For GPT-3.5-Turbo with 20 documents, Figure 4 shows the published numbers. In the middle, the model did worse than with no documents at all (closed-book), even though the answer was in its context. [Certain] The same pattern appeared for models built for long contexts, and in a synthetic key-value retrieval task with 75, 140 and 300 pairs. [Certain]
Why does this happen? Two likely causes. First, training data: the important part of a text is often at the start (titles, instructions) or right before the next token, so models learn these habits. Second, causal attention: early tokens are seen by every later token and become strong anchors (the attention sinks of session 27), while recent tokens are favored by RoPE’s mild preference for near distances. [Likely] Practical advice that follows: put instructions and the most important documents at the start or the end of a long prompt, not in the middle. [Likely]
RULER: what is the real context size?
RULER (Hsieh et al., 2024) is a synthetic benchmark with 13 tasks in 4 groups, run at lengths 4K, 8K, 16K, 32K, 64K and 128K: [Certain]
- Retrieval (8 tasks): NIAH variants with several keys, several values, several queries, and distracting needles.
- Multi-hop tracing: variable tracking, “X1 = 12345, X2 = X1, X3 = X2, …, what has the value 12345?”.
- Aggregation: find the most common or most frequent words in a long list.
- Question answering: SQuAD and HotpotQA with many distracting paragraphs.
To turn scores into one number, RULER uses a threshold \(\tau\): the score of Llama 2 7B at 4K, which is 85.6%. The effective length is the largest tested length whose average score is still above it: [Certain]
The headline result: of 17 models that all claim 32K tokens or more, only about half keep a satisfactory score at 32K. [Certain] The advertised window and the effective context are often far apart.
How to read Figure 5
Model A passes up to 16K and fails from 32K on: effective length 16K, an eighth of its window. Model B stays above 85.6 up to 64K. Model C is strong at short lengths but falls quickly: effective length 8K. Move the threshold: a lower bar makes every model look longer. This is why you must always say which threshold, which tasks, and which lengths you used when you report an “effective context”.
Context, memory or retrieval?
A long context is one of three ways to give a model information that is not in its weights. They are not rivals; real systems mix them.
- Context: put everything into the prompt. The model can attend to every token, and nothing has to be chosen in advance. Cost: KV memory grows linearly (28.1), prefill grows roughly with the square of the length, and quality drops in the middle (Figure 4).
- Memory: the system writes notes (summaries, facts about the user, past decisions) to a store and loads a small, chosen part of them into later prompts. MemGPT (Packer et al., 2023) frames this like an operating system that moves data between a small “main memory” (the context) and a large “disk”. [Certain] Cost: what is not written down is lost, and summaries can drift.
- Retrieval (RAG, Lewis et al., 2020): split a large collection into chunks, index them with an embedding model, and at query time put only the top \(k\) chunks into the context. [Certain] Cost: if the search misses the right chunk, the model never sees it; and questions that need the whole corpus (“summarize every contract”) do not fit the pattern.
How to read Figure 6
At 128K tokens of information (the default), the whole thing just fits in the window: 16 GiB of KV cache for one request. Retrieval with 8 chunks uses 4,096 tokens: 512 MiB, 32 times less, and about 1,000 times fewer attention scores. Move the first slider past 128K: the full-context option no longer fits at all, while retrieval and memory stay the same size. Retrieval cost depends only on \(k\), not on the size of the collection; what grows with the collection is the index, and the risk that the search misses.
| Long context | Memory | Retrieval (RAG) | |
|---|---|---|---|
| Best for | one long document, whole-input reasoning, code files that refer to each other | long conversations, agents, personal facts across sessions | large, changing collections; citations; facts updated daily |
| Main failure | lost in the middle; cost per request | forgetting what was not saved; drift | search misses the right chunk |
| Cost grows with | input length (linear memory, quadratic prefill) | size of the kept summary | \(k\) times chunk size; index size offline |
| Freshness | whatever is pasted | what was written down | re-index and it is up to date |
In code
The PyTorch code below builds the new angle steps for every method, checks the claims made above (NTK-aware leaves pair 0 alone and slows the last pair by \(s\); the Llama 3.1 rule gives 29 / 29 / 6 pairs; all three methods keep the slowest pair inside its training range), prints the YaRN temperature, and computes a RULER-style effective length. It runs on a CPU in under a second and prints the same output every time.
import math, torch def inv_freq(d=128, base=10_000.0): # theta_i = base^(-2i/d), formula (28.2) return base ** (-torch.arange(0, d, 2, dtype=torch.float64) / d) def scaled_inv_freq(method, d, base, L, s, alpha=1.0, beta=32.0): th = inv_freq(d, base) if method == "none": return th if method == "pi": # (28.4): every pair turns s times slower return th / s if method == "ntk": # (28.5): raise the base return inv_freq(d, base * s ** (d / (d - 2))) if method in ("yarn", "llama3"): # (28.6)-(28.8): by parts, with a ramp if method == "llama3": alpha, beta = 1.0, 4.0 # low_freq_factor, high_freq_factor r = L / (2 * math.pi / th) # turns during training g = ((r - alpha) / (beta - alpha)).clamp(0, 1) return (1 - g) * th / s + g * th raise ValueError(method) d, L, s = 128, 4096, 4 th = inv_freq(d) for m in ["pi", "ntk", "yarn"]: ratio = scaled_inv_freq(m, d, 10_000.0, L, s) / th print(f"{m:5s} fastest pair x{ratio[0].item():.3f} slowest pair x{ratio[-1].item():.4f}") # Llama 3.1: base 500,000, original length 8,192, factor 8 th3 = inv_freq(128, 500_000.0) new = scaled_inv_freq("llama3", 128, 500_000.0, 8192, 8) ratio = new / th3 print("unchanged:", int((ratio > 1 - 1e-9).sum()), " divided by 8:", int((ratio < 1 / 8 + 1e-9).sum()), " blended:", int(((ratio < 1 - 1e-9) & (ratio > 1 / 8 + 1e-9)).sum())) # YaRN attention temperature, formula (28.9): logits are multiplied by (0.1 ln s + 1)^2 for s_ in [4, 16, 32]: print(s_, round((0.1 * math.log(s_) + 1) ** 2, 3)) # The largest angle each method reaches at the new length L' = s L, for the slowest pair Lp = s * L for m in ["none", "pi", "ntk", "yarn"]: a = Lp * scaled_inv_freq(m, d, 10_000.0, L, s)[-1].item() print(f"{m:5s} slowest pair: max angle {a:.3f} rad (training saw up to {L * th[-1].item():.3f})") # RULER-style effective length, formula (28.11): largest tested length that passes def effective_length(scores, threshold=85.6): ok = [n for n, sc in sorted(scores.items()) if sc >= threshold] return max(ok) if ok else 0 toy = {4096: 94.0, 8192: 92.1, 16384: 88.0, 32768: 79.5, 65536: 70.2, 131072: 51.0} # made-up scores print("effective length:", effective_length(toy)) # Output: # pi fastest pair x0.250 slowest pair x0.2500 # ntk fastest pair x1.000 slowest pair x0.2500 # yarn fastest pair x1.000 slowest pair x0.2500 # unchanged: 29 divided by 8: 29 blended: 6 # 4 1.296 / 16 1.631 / 32 1.813 (one per line) # none slowest pair: max angle 1.892 rad (training saw up to 0.473) # pi, ntk, yarn: max angle 0.473 rad (training saw up to 0.473) # effective length: 16384
To use one of these in a real model you do not write this yourself: set rope_scaling in the config (Hugging Face) or the matching flag in vLLM, and the library builds the tables. Writing it once by hand is still the best way to understand what the config numbers do.
Where this is used
| Application | What “context” must hold | Typical approach |
|---|---|---|
| Long-document QA (contracts, reports) | one document of 50K to 200K tokens | extended RoPE model (Llama 3.1 rule, YaRN); important instructions at the start and end |
| Code assistants over a repository | many files that refer to each other | large base (Code Llama), retrieval of relevant files, then a long context |
| Chat assistants and agents | long histories, tool outputs | memory (summaries, saved facts) plus a moderate window (session 154) |
| Enterprise search, support bots | millions of pages that change daily | RAG with top-\(k\) chunks (session 152) |
| Long-video question answering | an hour of video is tens of thousands of visual tokens or more, even after pooling [Likely] | frame sampling and token pooling, extended M-RoPE positions, or Video-RAG over clips (session 206) |
| Key-moment detection, highlight search | where in a long video an event happens | “needle in a video haystack” tests: insert a clip and ask about it, the video version of NIAH [Likely] |
| Video retrieval and Video-RAG | a library of videos | retrieve clips by embedding, then reason over the few retrieved clips (session 210) |
| Training at long lengths | sequences of 128K or more in training | context parallelism splits one sequence over GPUs (session 49) |
Common mistakes
- Trusting
max_position_embeddings. It is a setting. Check the training length and run a long-context test on your own task. - Using PI with a huge factor and no fine-tuning. Slowing every pair by 32 blurs local order. Use YaRN or an NTK-style method, and fine-tune.
- Applying static scaling to every request. A fixed YaRN factor can hurt short inputs (Qwen2.5 card). Turn it on only when needed, or use dynamic scaling, and re-run your short-text evaluations.
- Mismatched training and inference settings. If you fine-tuned with YaRN factor 16, serve with factor 16 and the same original length. A different
original_max_position_embeddingsmoves every ramp boundary. - Forgetting the YaRN temperature when you re-implement it in your own kernel. Without the \(0.1\ln s + 1\) factor, attention is flatter than the model was tuned for.
- Calling a model “128K” from a green NIAH map. NIAH is easy. Test multi-hop and aggregation (RULER) and your real task at the lengths you serve.
- Putting the key document in the middle. In long prompts, place instructions and the most relevant chunks at the start or end.
- Ignoring the KV budget. 128K tokens of Llama 3.1 8B is 16 GiB per sequence. Concurrency, not quality, is often what limits long context in production.
Interview questions
What is the difference between context window, training length and effective context?basic
max_position_embeddings). The training length is the longest sequence the model actually trained on; positions past it are unfamiliar. The effective context is the longest input on which the model still does a given task well, measured for example by RULER’s threshold. Usually \(\text{effective} \le \text{training} \le \text{window}\), and the gaps can be large: RULER found that only about half of 17 models that claim 32K or more stay above threshold at 32K.Why does a RoPE model fail past its training length?core
Explain position interpolation. Why does it work better than extrapolation, and what does it cost?core
What does NTK-aware scaling change, and why is the exponent \(d/(d-2)\)?core
Describe YaRN’s two parts.advanced
How does Llama 3.1’s rope_scaling relate to YaRN?advanced
Why is needle-in-a-haystack not a sufficient long-context test?core
What is “lost in the middle”, and what would you do about it in a product?core
How does RULER define effective context length?basic
You have 50 million tokens of company documents. Long context, memory or RAG?core
Why does raising the RoPE base help long contexts, and what is the trade-off?advanced
Exercises
- A model has 40 layers, 8 KV heads, head size 128, BF16. How large is the KV cache for one 32K-token (32,768) sequence?
- For \(d = 128\) and \(b = 10{,}000\), which pairs are “slow” for \(L = 8{,}192\)? (Hint: \(\lambda_i = 2\pi \cdot 10{,}000^{i/64}\gt 8{,}192\).) Check with the Figure 1 slider logic.
- Compute the NTK-aware base for \(b = 10{,}000\), \(d = 128\), \(s = 8\).
- With YaRN at \(s = 8\), what number multiplies the attention logits?
- For Llama 2 (\(L = 4{,}096\)), pair 30 has \(r = 8.69\). With \(\alpha = 1\), \(\beta = 32\) and \(s = 4\), what is \(\theta'_{30}/\theta_{30}\)?
- A made-up model scores 93, 90, 86, 84, 87, 60 at 4K, 8K, 16K, 32K, 64K, 128K. What is its RULER effective length with \(\tau = 85.6\)? What is strange about this result?
Answerscheck
Key takeaways
- “Context length” means three things: the window (a setting), the training length (what the model saw), and the effective context (what it can really use for a task). They often differ a lot.
- Long context costs memory: KV bytes \(= 2 \, n_{\text{layers}} n_{\text{kv}} d_{\text{head}} \cdot \text{bytes} \cdot T\) (28.1), 16 GiB per 128K sequence for Llama 3.1 8B.
- RoPE breaks past \(L\) because slow pairs (\(\lambda_i \gt L\)) reach angles never seen in training. Every fix tries to keep \(L'\theta'_i \le L\theta_i\) for them (28.3).
- PI divides all \(\theta_i\) by \(s\) (28.4): safe but blurs local order. NTK-aware raises the base (28.5): keeps fast pairs but under-scales some slow ones. YaRN ramps by \(r_i = L/\lambda_i\) (28.6 to 28.8) and adds a temperature (28.9). Llama 3.1’s rule is the same ramp with \(\alpha = 1, \beta = 4\) (28.10).
- Test with more than NIAH: lost in the middle shows a U-shaped curve; RULER’s effective length (28.11) shows that about half of the models claiming 32K or more do not hold up at 32K.
- Context, memory and retrieval are complementary. Retrieve to narrow, use a long context to read, and keep memory for what must last across sessions.
Key formulas
References for this session
- Chen et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation
- Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models
- Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts
- Hsieh et al. (2024). RULER: What’s the Real Context Size of Your Long-Context Language Models?
- Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- bloc97 (Reddit r/LocalLLaMA) (2023). NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation
- Rozière et al. (2023). Code Llama: Open Foundation Models for Code
- Kamradt (2023). Needle In A Haystack: pressure testing LLMs
- Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Packer et al. (2023). MemGPT: Towards LLMs as Operating Systems
- Xiao et al. (2023). Efficient Streaming Language Models with Attention Sinks
- Jiang et al. (2023). Mistral 7B