Position, part 1: why order needs help; absolute and relative positions
Self-attention looks at a set of tokens, not a list: if you shuffle the input, the outputs are shuffled the same way and nothing else changes. So a transformer needs extra help to tell “dog bites man” from “man bites dog”. This session shows why, then builds the two main fixes: absolute positions (sinusoidal and learned) added to the input, and relative positions (Shaw, T5 buckets, ALiBi) added to the attention scores, and ends with what each one does past the training length.
What you will learn
- Why self-attention without positions is permutation-equivariant, and what that means for “dog bites man”
- How a causal mask leaks some order information, and why that is not enough
- Sinusoidal positions: frequencies, wavelengths, and why a fixed offset is a rotation
- Learned absolute positions (BERT, GPT-2, ViT) and their hard length limit
- Relative positions: Shaw et al., T5 bias buckets, and ALiBi linear biases with their slopes
- What learned, sinusoidal, T5 and ALiBi positions do past the training length
- Where position matters in VLMs and video: patches, frames and timestamps
Why this matters
“Dog bites man” is not news. “Man bites dog” is. The two sentences use the same three words; only the order is different. A transformer’s core operation, self-attention, cannot see that difference on its own. If you feed it the same tokens in a different order, it gives the same outputs, just moved around. This session explains why, and how models add the missing information.
In session 11 we turned each token into a vector by looking up a row in a table. That row depends only on which token it is, never on where it is: both copies of “the” in “the cat sat on the mat” get the same vector. In session 12 we saw that the transformer layers mix information between tokens to build contextual vectors. Here we look at the one piece of information that the mixing step does not provide: the position of each token.
There are two families of fixes, and both are still in use:
- Absolute positions. Give each position \(t = 0, 1, 2, \dots\) its own vector and add it to the token embedding before the first layer. The original transformer used fixed sine and cosine waves; BERT, GPT-2 and ViT learn a table of position vectors.
- Relative positions. Leave the inputs alone and instead change the attention score between two tokens depending on how far apart they are. T5 learns one number per distance bucket; ALiBi subtracts a penalty that grows linearly with distance. RoPE, used in Llama, Qwen and most current LLMs, is also relative; it gets its own session (session 14).
The choice matters most at one moment: when the model reads a text longer than anything it saw in training. Some methods break at once, some degrade slowly, some keep working in a limited way. This is the root of a whole research area on long context (session 28).
Interviewers ask about this often: “Is a transformer permutation-invariant?”, “Why sine and cosine?”, “What happens to GPT-2 at token 1,025?”, “How does T5 encode position?”, “Why does ALiBi extrapolate?”. By the end of this session you can answer each with a formula and a small example.
Self-attention ignores order
The attention formula, once more
Stack the \(T\) input vectors of a sequence as the rows of a matrix \(X \in \R^{T \times d}\). One head of self-attention computes queries, keys and values with three learned matrices, then a weighted average of the values (session 17 covers this in depth):
- Read the symbols
- \(X\)
- the input: row \(t\) is the vector of token \(t\) (here, its embedding)
- \(W_Q, W_K, W_V\)
- learned matrices. They are the same for every position: the layer applies one rule to every row.
- \(QK^\top\)
- a \(T \times T\) table of scores: entry \((i, j)\) says how much token \(i\) wants to read from token \(j\)
- \(\sqrt{d_k}\)
- the square root of the key size; it keeps the scores in a good range
- \(\softmax\)
- applied to each row separately, so each row of weights sums to 1
Look at what the formula uses. Row \(i\) of the output depends on the query of token \(i\) and on the set of all (key, value) pairs. Nothing in the formula says “token \(j\) is the third one”. The score between two tokens depends only on their two vectors.
Permutation equivariance
Let \(P\) be a permutation matrix: a \(T \times T\) matrix with a single 1 in each row and column. \(PX\) is \(X\) with its rows shuffled. Then \(PQ = PXW_Q\), \(PK\), \(PV\) are shuffled the same way, the score table becomes \(P(QK^\top)P^\top\) (rows and columns shuffled together), the row-wise softmax just moves its entries along, and:
This property is called permutation equivariance: shuffle the input, and the output is shuffled the same way, with no other change. The feed-forward block, layer norm and residual connections work on each row separately, so they are equivariant too. A whole transformer stack without positions is therefore equivariant. [Certain] If you then summarize the sequence with an average, the shuffle disappears completely:
(since \(\mathbf{1}^\top P = \mathbf{1}^\top\): a permutation does not change a sum). The pooled vector is permutation-invariant. So a sentence classifier built this way would give exactly the same answer for “dog bites man” and “man bites dog”. The token “dog” gets the same output vector in both sentences, because it sees the same three keys and values, only listed in a different order.
Figure 1 uses 4-number embeddings, for example dog \(= (0.9, 0.1, -0.3, 0.5)\), and fixed small matrices \(W_Q, W_K\) with \(W_V = I\). In “dog bites man” the scores of the query “dog” against the keys (dog, bites, man) are \((0.553, 0.091, 0.163)\), and softmax gives the weights \((0.434, 0.273, 0.293)\). In “man bites dog” the keys come in the order (man, bites, dog), so the scores are \((0.163, 0.091, 0.553)\) and the weights \((0.293, 0.273, 0.434)\). It is the same list in a different order. Each value is multiplied by the same weight as before, so the weighted sum is the same: \((0.482, 0.144, 0.155, 0.278)\) in both sentences.
The causal mask leaks a little order
A decoder (GPT-style) model adds a causal mask: token \(i\) may only read tokens \(j \le i\) (session 18). This breaks the symmetry. The first token sees only itself, the second sees two tokens, and so on. In “dog bites man”, “bites” sees {dog, bites}; in “man bites dog” it sees {man, bites}, so its output differs. A deep causal model can even learn to count how many tokens came before, and so recover absolute positions. Haviv et al. (2022) showed that causal language models with no position encoding (“NoPE”) still learn positional information and reach perplexity close to models with positions. [Certain] Kazemnejad et al. (2023) found that NoPE can generalize to longer inputs better than several explicit encodings on small tasks. [Certain]
But inside the visible prefix the order is still invisible to one attention layer: the last token of “dog bites man” and of “bites dog man” sees the same set {dog, bites, man} and gets the same output. Encoders (BERT, ViT, the encoder of T5) have no causal mask at all, so they need positions completely. That is why almost every model adds explicit position information.
How to read Figure 1
Press Play (or Step). For each query (blue outline), the arcs first show the scores \(s_{ij}\) to every key, then the weights after softmax (thicker arc = larger weight). The live panel prints the numbers of each line of the algorithm. With both boxes unticked, look at the weights of a word in the new order: they are the same numbers as in the reference, only listed in a different order, so the output is the same and \(\Delta = 0.000\) for every word (formula 13.2). The last step compares the averaged sentence vectors: also identical (13.3). Press Shuffle for another order: still 0. Tick “Causal mask”: each slot sees only the keys up to itself, so the first slot attends only to itself, and the distances become larger than 0. This is the order leak of the causal mask. Tick “Add sinusoidal positions”: each input is now its embedding plus a vector for its slot, and every distance becomes larger than 0. The model can now tell the orders apart.
Absolute positions: add a vector per slot
The simplest fix: give every position \(t\) its own vector \(\mathbf{p}_t \in \R^d\) and add it to the token embedding before the first layer:
- Read the symbols
- \(\mathbf{e}_{\mathrm{id}_t}\)
- the token embedding of the token at position \(t\) (a row of the table \(E\) from session 11)
- \(\mathbf{p}_t\)
- the position vector for slot \(t\). The same \(\mathbf{p}_t\) is used for every sentence.
- \(\mathbf{x}_t\)
- the input of the first transformer layer at position \(t\)
Now “dog” at slot 0 and “dog” at slot 2 are different vectors, so equivariance (13.2) no longer holds. Why add and not concatenate? Adding keeps the width \(d\) unchanged and costs nothing. In high dimension, the network can learn to keep the two kinds of information in roughly separate directions. [Likely] Concatenation also works but makes every layer wider.
Sinusoidal positions
The original transformer (Vaswani et al., 2017) used fixed waves, with no learned parameters. For position \(t\) and pair index \(i = 0, 1, \dots, d/2 - 1\):
- Read the symbols
- \(t\)
- the position: 0 for the first token, 1 for the second, and so on
- \(i\)
- which pair of dimensions: each pair holds one sine and one cosine of the same frequency
- \(\omega_i\)
- the frequency of pair \(i\): how fast its angle turns per position. Pair 0 turns 1 radian per token; the last pairs turn very slowly.
- \(\lambda_i\)
- the wavelength: how many positions one full turn takes. It grows from \(2\pi \approx 6.3\) to about \(2\pi \cdot 10000\).
- 10000
- the base, a fixed choice of the paper. It sets the longest wavelength.
Think of a clock with many hands. The fast hand (pair 0) goes round every 6.3 tokens: it tells nearby tokens apart but repeats quickly. The slowest hands barely move: they tell “early in the text” from “late in the text”. Together, the hands give every position a different reading, like seconds, minutes and hours. For \(d = 512\), as in the original model, the wavelengths go from 6.28 to 60,611 positions. [Certain]
Why sine and cosine? Because together they make a point on a circle, and moving by a fixed offset \(k\) is a rotation of that point, whatever the starting position \(t\):
This is just the sum formulas \(\sin(a + b) = \sin a \cos b + \cos a \sin b\) and \(\cos(a + b) = \cos a \cos b - \sin a \sin b\). The matrix depends only on \(k\), not on \(t\). So “the token 3 places back” is the same linear map at every position. The paper says it chose these waves for exactly this reason: it hoped the model could then easily learn to attend by relative positions. [Certain] A direct consequence: the dot product of two position vectors depends only on their distance:
It equals \(d/2\) at distance 0 and mostly gets smaller as the distance grows (with small ripples). Figure 2 draws it.
With \(d = 4\) there are two pairs: \(\omega_0 = 1\) and \(\omega_1 = 10000^{-2/4} = 0.01\). So \(\mathbf{p}_t = (\sin t, \cos t, \sin 0.01t, \cos 0.01t)\).
\(\mathbf{p}_0 = (0, 1, 0, 1)\), \(\mathbf{p}_1 = (0.8415, 0.5403, 0.0100, 1.0000)\), \(\mathbf{p}_2 = (0.9093, -0.4161, 0.0200, 0.9998)\).
Dot product: \(\mathbf{p}_1 \cdot \mathbf{p}_2 = 0.7651 - 0.2248 + 0.0002 + 0.9998 = 1.5403\), and formula (13.8) gives \(\cos 1 + \cos 0.01 = 0.5403 + 0.99995 = 1.5403\). The same. And \(\mathbf{p}_1 \cdot \mathbf{p}_3 = \mathbf{p}_5 \cdot \mathbf{p}_7 = 0.5837\): both pairs are 2 apart, so they get the same number.
How to read Figure 2
In the heatmap, the left columns change color quickly from row to row (fast hands), and the right columns hardly change (slow hands). The outlined row is \(\mathbf{p}_t\). The curve below is 1 at \(s = t\) and falls off on both sides with the same shape: move the slider and the curve only slides, it does not change shape. That is formula (13.8): similarity depends only on \(t - s\). The readout checks this by comparing \(\mathbf{p}_t \cdot \mathbf{p}_{t+8}\) with \(\mathbf{p}_0 \cdot \mathbf{p}_8\). With a small \(d\) the curve has big ripples; with more pairs it becomes smoother.
Figure 3 runs formulas (13.5) to (13.8) step by step. It first builds \(\mathbf{p}_t\) one pair at a time: compute the frequency, turn the hand of that pair to the angle \(\omega_i t\), and read off the sine and the cosine. Then it shifts every hand by \(k\) positions with the rotation (13.7) and adds up the dot product pair by pair.
How to read Figure 3
Press Play. Steps 2 to 5 (for \(d = 8\)) each compute one pair: the hand turns to its angle and two numbers of \(\mathbf{p}_t\) appear. Pair 0 turns 1 radian per position, so at \(t = 10\) it has gone round more than once; pair 3 turns only 0.001 radian per position and has barely moved. The next steps rotate each pair by \(\omega_i k\) and add \(\cos(\omega_i k)\) to the running sum \(S\). For \(t = 10\), \(k = 3\), \(d = 8\) the sum is \(-0.9900 + 0.9553 + 0.9996 + 1.0000 = 1.9649\). The last step checks that \(S\) equals the direct dot product \(\mathbf{p}_t \cdot \mathbf{p}_{t+k}\) and also \(\mathbf{p}_0 \cdot \mathbf{p}_k\). Move \(t\): all blue hands move, but the green angles and \(S\) do not change, because they depend only on \(k\) (13.8).
Learned absolute positions
The other option is to learn the position vectors: a second embedding table \(P \in \R^{L \times d}\) with one row per position, trained by backpropagation like the token table. \(L\) is the maximum length, fixed before training. BERT uses \(L = 512\) and GPT-2 uses \(L = 1024\); with \(d = 768\) these tables have 393,216 and 786,432 parameters, tiny next to the rest of the model. [Certain] The Vision Transformer learns one vector per image patch position in the same way (session 164).
Vaswani et al. tried both and found “nearly identical results”. [Certain] Learned tables are simpler and can fit whatever the data needs. Their big weakness is the hard limit: there is no row for position \(L\). GPT-2 simply cannot be run on token 1,025 without changing the model. We come back to this below.
What absolute positions do inside attention
Absolute vectors enter only at the bottom, but they shape every score. Put (13.4) into the score of the first layer (ignore the \(\sqrt{d_k}\) for now, and write \(W = W_Q W_K^\top\)):
The last term is where order enters. With sinusoidal vectors and \(W = I\) it would depend only on \(i - j\) (formula 13.8). But for a general learned \(W\), the term \(\mathbf{p}_i W \mathbf{p}_j^\top\) can depend on \(i\) and \(j\) separately. So absolute encodings let the model use relative distance, but do not force it to. The middle terms mix content and position, which is often useless noise. These two observations motivated the relative methods: put the distance directly into the score instead.
Relative positions: put the distance in the score
For language, “the word two places before me” usually matters more than “the word at slot 517”. A relative method makes the score between query \(i\) and key \(j\) depend on the offset \(j - i\) (or the distance \(i - j\)), and does not add anything to the inputs.
Shaw et al.: learned relative vectors
Shaw, Uszkoreit and Vaswani (2018) add a learned vector \(\mathbf{a}_{j-i}\) to each key, chosen by the offset, and clip offsets beyond a maximum \(k\):
There are only \(2k + 1\) different vectors, shared by all positions. Offsets farther than \(k\) all share the edge vector. On WMT 2014 translation this improved BLEU by 1.3 (English–German) and 0.3 (English–French) over absolute positions. [Certain] Transformer-XL and DeBERTa later used related ideas. [Certain]
T5: one learned number per distance bucket
T5 (Raffel et al., 2019) simplifies this a lot. Instead of a vector, each offset gets a single learned scalar that is added to the score:
- Read the symbols
- \(b_{h,\,n}\)
- a learned number for head \(h\) and bucket \(n\). Different heads can prefer different distances.
- \(\mathrm{bucket}(j - i)\)
- maps the offset to one of 32 buckets: exact for small distances, coarser for large ones
- no \(\sqrt{d_k}\)
- T5 folds the usual scale into the initialization of \(W_Q\), so the score has no division. [Certain]
The bias table is shared by all layers of a stack; each head has its own column. With 32 buckets and 12 heads (T5-base) that is only \(32 \times 12 = 384\) numbers per stack. [Likely] The bucket map is the clever part. Nearby distances get their own bucket, because “the previous word” and “two words back” are very different. Far distances share buckets on a log scale, because “100 back” and “110 back” rarely need different treatment. For a decoder (causal, only the past counts), with \(n = 32\) buckets and maximum distance \(D = 128\), let \(r = \max(0,\ i - j)\) be how far back the key is:
In the encoder (bidirectional) the 32 buckets are split in two: 16 for keys before the query and 16 for keys after it, each with 8 exact buckets and 8 log-spaced ones. [Certain] The formula is the one in the T5 code (and in Hugging Face’s _relative_position_bucket).
Decoder (\(n = 32\), \(D = 128\)): distances 0 to 15 get buckets 0 to 15. Distance 16 gives \(16 + \lfloor 0 \rfloor = 16\). Distance 19 gives \(16 + \lfloor \log(19/16)/\log 8 \cdot 16 \rfloor = 16 + \lfloor 1.32 \rfloor = 17\). Bucket 16 covers distances 16–18, bucket 17 covers 19–20, bucket 30 covers 99–112, and bucket 31 covers every distance from 113 up.
Encoder: keys 0 to 7 back get buckets 0 to 7; 8 to 11 back share bucket 8; 91 or more back share bucket 15. Keys after the query use buckets 16 to 31: one step ahead is bucket 17, and 91 or more ahead is bucket 31.
How to read Figure 4
Near 0 the staircase has steps of width 1: every small distance has its own learned bias. Farther away the steps get wider and wider; past 128 the line is flat, so all far tokens share one bias. Move the slider: the readout shows the bucket and all the offsets that share it. In the encoder there are two staircases, one for the past (buckets 0–15) and one for the future (16–31). In the decoder, the future side is gone (those keys are masked) and the past side gets all 32 buckets, so it is finer: 16 exact buckets instead of 8.
ALiBi: a fixed linear penalty
ALiBi (“Attention with Linear Biases”, Press, Smith and Lewis, 2021) goes one step further: nothing is learned. Each head subtracts a penalty proportional to the distance, and no position vector is added to the inputs:
- Read the symbols
- \(i - j\)
- how many tokens back the key is (0 for the token itself). ALiBi was designed for causal models, so \(j \le i\).
- \(m_h\)
- the slope of head \(h\), a fixed number, not learned. A large slope means “look only at the last few tokens”; a small one means “look far back”.
- \(H\)
- the number of heads. For \(H = 8\) the slopes are \(1/2, 1/4, \dots, 1/256\): a geometric sequence. [Certain]
Why is this a good idea? After softmax, a bias of \(-m\Delta\) multiplies the weight of a key \(\Delta\) tokens back by \(e^{-m\Delta}\). So each head has a built-in preference for recent tokens, with its own “half-life”, while the content scores can still override it. The heads together cover a range of scales, like the many wavelengths of the sinusoidal method:
- Read the symbols
- \(a_{i,j}\)
- the attention weight from query \(i\) to key \(j\) after softmax
- \(s^{c}\)
- the content part of the score, \(\mathbf{q} \cdot \mathbf{k}/\sqrt{d_k}\)
- \(\Delta_{\text{reach}}\)
- a rough reach of the head: the distance at which the penalty is \(c\). With \(c = 10\) the weight is cut by \(e^{-10} \approx 1/22{,}000\).
Take \(H = 8\), head 1, so \(m = 1/2\). A query at position 5 sees keys 0 to 5 at distances 5, 4, 3, 2, 1, 0. Suppose all content scores are equal (0). The biases are \((-2.5, -2, -1.5, -1, -0.5, 0)\) and softmax gives the weights \((0.034, 0.056, 0.092, 0.152, 0.251, 0.414)\): strongly recent. For head 8, \(m = 1/256\), the weights are almost flat: from 0.165 to 0.168.
Reach with \(c = 10\): head 1 reaches about 20 tokens, head 8 about 2,560. With equal content scores over 2,048 keys, head 1 puts 99.3% of its weight on the last 10 tokens, and head 8 puts 63.2% on the last 256.
How to read Figure 5
The bias matrix is constant along each diagonal: the penalty depends only on the distance, never on the absolute position. That is why ALiBi is a relative method. Press Play: the player walks along the outlined row, one key per step, and the live panel substitutes the numbers into \(s_{ij} = c_{ij} - m(i - j)\). With head 1 (\(m = 1/2\)) the key 7 back gets a bias of \(-3.5\), so its score falls from 0.850 to \(-2.650\), and after softmax its weight drops from 0.229 (no bias) to 0.021, while the token itself gets 0.442. Move the slider to head 8: the slope is \(1/256\), the red lines almost vanish, and the solid and dashed weights nearly coincide. With \(H = 16\) the slopes start at \(2^{-0.5} = 0.707\) and also end at \(1/256\). Tick “Equal content scores” and set query \(i = 5\) to reproduce the worked example above: weights 0.034 to 0.414 for head 1.
Past the training length
A model is trained with sequences up to some length \(L\). What happens when it reads a longer text at test time? This question, called length extrapolation, separates the methods clearly.
- Learned absolute. There is no row for position \(L\) or beyond. GPT-2 with \(L = 1024\) gives an index error at the 1,025th token. You must cut the input, use a sliding window, or extend the table (copy or interpolate rows) and fine-tune. ViT does the same for images: at a higher resolution there are more patches, so the position table is resized by 2D interpolation. [Certain]
- Sinusoidal. The formula works for any \(t\), so nothing crashes. But the model has never seen these vectors. The slow pairs, those with a wavelength longer than \(L\), have only turned part of the way round during training; past \(L\) they point in directions never seen before (Figure 6). In the ALiBi paper, the perplexity of a sinusoidal model rises quickly once the input is longer than \(L\). [Certain]
- T5 bias. Every distance beyond 128 falls in the last bucket, which was trained. So any length gets a valid bias, and the ALiBi paper found that T5’s bias does extrapolate to some degree, but it was slower to compute than the other methods they compared. [Likely]
- ALiBi. A longer distance just gives a larger penalty of the same form, which the model has already learned to handle. Perplexity stays roughly flat well past \(L\). [Certain] The price: far tokens get tiny weights, so in practice the model mostly uses a recent window. Good perplexity at long lengths does not prove that it can use information from 10,000 tokens back. [Likely]
- RoPE. Like sinusoidal, it is defined everywhere but extrapolates poorly without help. Position interpolation and YaRN rescale its angles to stretch the context (session 14, session 28).
The count in Figure 6 has a simple formula. Pair \(i\) completes at least one full turn during training if \(\lambda_i \le L\). Using (13.6), the number of pairs that do not is the number of \(i\) with \(2\pi \cdot 10000^{2i/d} \gt L\). For \(d = 512\) and \(L = 1024\), that is 114 of the 256 pairs; for \(L = 4096\), still 75. [Certain]
How to read Figure 6
The fast pairs (left) went round many times during training, so their arc is a full circle and every test angle is familiar. The slow pairs (right) covered only part of the circle. Move the test position past \(L\): their dots leave the arc and turn orange. The readout counts how many of the 256 pairs show a new angle. Raise \(L\): fewer pairs are new, but the slowest ones (wavelength 60,611) never complete a turn for any realistic \(L\). Below \(L\) the count is 0 by definition. A learned table would simply have no row there.
| Method | Where it acts | Learned? | Past training length | Used in |
|---|---|---|---|---|
| None (NoPE) | only through the causal mask | — | no hard limit; can generalize on some tasks | research models |
| Sinusoidal | added to input (13.5) | no | defined, but unseen values; degrades fast | original transformer |
| Learned absolute | added to input (13.4) | yes, \(L \times d\) | impossible without resizing | BERT, GPT-2, ViT |
| Shaw relative | vector added to keys (13.10) | yes, \((2k+1) \times d_k\) | clipped: far offsets share one vector | Music Transformer and others [Likely] |
| T5 bias | scalar added to scores (13.11) | yes, 32 × heads | far offsets share the last bucket | T5, Flan-T5 |
| ALiBi | linear penalty on scores (13.13) | no | works, mostly as a recency window | BLOOM, MPT |
| RoPE | rotates \(\mathbf{q}\) and \(\mathbf{k}\) | no | needs scaling (PI, YaRN) | Llama, Qwen, most current LLMs |
Position beyond text: patches, frames and time
Images and videos have more than one axis, and the same two families appear there.
- Images. ViT cuts an image into patches and adds a learned 1D position vector per patch index. The paper reports that 2D-aware position embeddings gave no significant gain over 1D ones in their setting. [Certain] Swin uses a learned relative bias, much like T5, inside each local window. [Certain]
- Video. Order in time decides meaning: “opening a door” played backwards is “closing a door”. Video transformers add a time position on top of the spatial one, for example separate learned spatial and temporal embeddings. [Likely] Without a time signal, a video model is a bag of frames and cannot tell the two actions apart.
- VLMs. When image tokens are placed inside a text sequence, the model needs both a sequence position (for the LLM) and a 2D position (for the image layout). Qwen2-VL’s M-RoPE gives each token separate time, height and width positions; we preview it in the next session. [Certain]
Where this is used
| Application | What position does | Link |
|---|---|---|
| Every LLM | Tells word order apart (“dog bites man”); modern models use RoPE, older ones learned or ALiBi positions | session 14 |
| Encoders for classification and search | BERT-style encoders have no causal mask, so without positions a pooled sentence vector would be a bag of words (13.3) | session 33 |
| Long-document models and RAG | The choice of encoding decides how far past the training length a model can read | session 28 |
| Image encoders in VLMs | Learned patch positions are interpolated when the image resolution changes | session 164 |
| Action recognition | A time position lets the model separate actions that differ only in order (open vs close, push vs pull) | session 202 |
| Key-moment detection and video QA | Frame positions or timestamps let the model answer “when did it happen?” and point to a time | session 210 |
| Video fingerprinting | Matching two clips uses the order of frame descriptors, not just the set; a bag-of-frames match cannot detect reordered edits | session 209 |
| Encoder–decoder translation and summarization | T5’s bucketed relative bias in both the encoder and the decoder | session 24 |
Pseudo-code
- input: token IDs \(\mathrm{id}_0, \dots, \mathrm{id}_{T-1}\), token table \(E\), mode, max length \(L\)
- if mode = learned and \(T \gt L\) then error // no row for position L
- for \(t = 0\) to \(T - 1\) do
- if mode = sinusoidal then
- for \(i = 0\) to \(d/2 - 1\) do \(\omega \leftarrow 10000^{-2i/d}\); \(p_{2i} \leftarrow \sin \omega t\); \(p_{2i+1} \leftarrow \cos \omega t\) // (13.5)
- else \(\mathbf{p} \leftarrow P_{t,:}\) // learned row
- \(\mathbf{x}_t \leftarrow E_{\mathrm{id}_t,:} + \mathbf{p}\) // (13.4)
- return \(X\) // then the usual attention (13.1)
- input: \(Q, K, V \in \R^{T \times d_k}\), head \(h\), mode
- \(S \leftarrow QK^\top\) // divide by \(\sqrt{d_k}\) for ALiBi; T5 does not
- for \(i = 0\) to \(T - 1\) do
- for \(j = 0\) to \(T - 1\) do
- if causal and \(j \gt i\) then \(S_{ij} \leftarrow -\infty\); continue
- if mode = T5 then \(S_{ij} \leftarrow S_{ij} + b_{h,\ \mathrm{bucket}(j - i)}\) // learned scalar, (13.11)–(13.12)
- else \(S_{ij} \leftarrow S_{ij} - m_h (i - j)\) // fixed slope \(m_h = 2^{-8h/H}\), (13.13)–(13.14)
- return \(\softmax(S)\,V\) // row-wise softmax; no position added to inputs
The same ideas in PyTorch. Each part checks one claim of this session; the printed values are in the comments:
import math import torch torch.manual_seed(0) # 1) self-attention without positions is permutation-equivariant, formula (13.2) T, d = 5, 8 X = torch.randn(T, d) Wq, Wk, Wv = torch.randn(d, d), torch.randn(d, d), torch.randn(d, d) def attn(X, bias=None): S = (X @ Wq) @ (X @ Wk).T / math.sqrt(d) if bias is not None: S = S + bias return torch.softmax(S, dim=-1) @ (X @ Wv) perm = torch.tensor([4, 2, 0, 3, 1]) print(torch.allclose(attn(X[perm]), attn(X)[perm], atol=1e-6)) # True: outputs just move print(torch.allclose(attn(X[perm]).mean(0), attn(X).mean(0), atol=1e-6)) # True: pooled vector identical (13.3) # 2) sinusoidal positions, formula (13.5) def sinusoidal(T, d, base=10000.0): pos = torch.arange(T).float()[:, None] # (T, 1) w = base ** (-torch.arange(0, d, 2).float() / d) # (d/2,) frequencies P = torch.zeros(T, d) P[:, 0::2] = torch.sin(pos * w) P[:, 1::2] = torch.cos(pos * w) return P P = sinusoidal(16, 4) print(P[1].round(decimals=4), P[2].round(decimals=4)) # tensor([0.8415, 0.5403, 0.0100, 1.0000]) tensor([ 0.9093, -0.4161, 0.0200, 0.9998]) print(round((P[1] @ P[3]).item(), 4), round((P[5] @ P[7]).item(), 4)) # 0.5837 0.5837 (13.8) pe = sinusoidal(T, d) print(torch.allclose(attn(X[perm] + pe), attn(X + pe)[perm], atol=1e-6)) # False: order now matters # 3) T5 relative position buckets, formula (13.12) def t5_bucket(rel, bidirectional=True, num_buckets=32, max_distance=128): # rel = key_position - query_position out = torch.zeros_like(rel) if bidirectional: num_buckets //= 2 out += (rel > 0).long() * num_buckets n = rel.abs() else: n = (-rel).clamp(min=0) max_exact = num_buckets // 2 large = max_exact + (torch.log(n.float().clamp(min=1) / max_exact) / math.log(max_distance / max_exact) * (num_buckets - max_exact)).long() large = large.clamp(max=num_buckets - 1) return out + torch.where(n < max_exact, n, large) rel = torch.tensor([0, -1, -7, -8, -12, -32, -128, -1000, 1, 8, 128]) print(t5_bucket(rel).tolist()) # [0, 1, 7, 8, 9, 12, 15, 15, 17, 24, 31] print(t5_bucket(-torch.tensor([0, 15, 16, 19, 113, 5000]), bidirectional=False).tolist()) # [0, 15, 16, 17, 31, 31] # 4) ALiBi bias, formulas (13.13) and (13.14) def alibi_slopes(H): s = 2 ** (-8 / H) return torch.tensor([s ** (h + 1) for h in range(H)]) def alibi_bias(T, H): i = torch.arange(T)[:, None]; j = torch.arange(T)[None, :] dist = (i - j).clamp(min=0).float() # how far back key j is causal = torch.full((T, T), float("-inf")).triu(1) return -alibi_slopes(H)[:, None, None] * dist + causal # (H, T, T) print(alibi_slopes(8).tolist()) # [0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625, 0.0078125, 0.00390625] B = alibi_bias(6, 8) print(torch.softmax(B[0, 5], -1).round(decimals=3)) # tensor([0.0340, 0.0560, 0.0920, 0.1520, 0.2510, 0.4140]) head 1, query at position 5 print(attn(X, B[0][:T, :T]).shape) # torch.Size([5, 8]): the bias plugs straight into the scores
Common mistakes
- Saying “transformers are permutation-invariant”. Self-attention is equivariant (13.2); only a pooled output is invariant (13.3). And a causal decoder is not fully either, because the mask leaks order.
- Forgetting positions in a custom encoder. A small transformer over frames, patches or set elements without any position signal treats the input as a bag. It may still train, and quietly fail on order-dependent examples.
- Running a learned-position model past \(L\). GPT-2 or BERT past 1,024 or 512 tokens is an index error, or, with some code, silent reuse of wrong rows. Truncate, use a sliding window, or resize and fine-tune.
- Mixing up the sinusoidal layout. Some code interleaves sin and cos (dimensions \(2i, 2i + 1\)); other code puts all sines first and all cosines second. Both are valid, but loading weights trained with one layout into code using the other breaks the model.
- Getting the sign of the offset wrong. T5 uses \(j - i\) (key minus query); ALiBi penalizes \(i - j\). Flip a sign and a decoder attends to the wrong side, or every distance lands in a wrong bucket.
- Treating ALiBi slopes as learnable or reusing 8-head slopes for 16 heads. The slopes are fixed and depend on \(H\) (13.14). Changing the head count changes the slopes.
- Reading flat perplexity past \(L\) as “uses the whole context”. With ALiBi, distant tokens get tiny weights. Test retrieval from far back (a needle-in-a-haystack test), not only perplexity. [Likely]
- Resizing image position tables naively. When ViT runs at a new resolution, the position grid must be interpolated in 2D, not as a flat 1D list; otherwise rows and columns get mixed.
Interview questions
Why does a transformer need position information at all?basic
Is a transformer permutation-invariant or permutation-equivariant? What changes with a causal mask?core
Write the sinusoidal position encoding. Why use both sine and cosine, and why many frequencies?core
Learned vs sinusoidal absolute positions: what are the trade-offs?core
Expand the first-layer attention score with absolute positions. What does it tell you?advanced
How does T5 encode position? Explain the buckets.core
What is ALiBi and how are its slopes chosen?core
Why does ALiBi extrapolate better than sinusoidal positions? What is the catch?advanced
GPT-2 has 1,024 learned positions. You need to process 4,000 tokens. What are your options?advanced
How do vision transformers and video models handle position, and what happens at a new resolution?core
Why not just concatenate a one-hot position vector to each token?basic
Exercises
- Show that \(\mathrm{softmax}\) applied row-wise satisfies \(\softmax(PSP^\top) = P\,\softmax(S)\,P^\top\) for a permutation matrix \(P\), and use it to prove (13.2).
- For \(d = 4\) sinusoidal positions, compute \(\mathbf{p}_0 \cdot \mathbf{p}_1\) using formula (13.8).
- For \(d = 512\), what is the wavelength of pair \(i = 128\)? How many positions does pair \(i = 0\) need for one full turn?
- In a T5 decoder (32 buckets, max distance 128), which bucket does distance 20 fall in? Distance 500?
- An ALiBi model has 16 heads. What are the slopes of heads 1, 2 and 16?
- With ALiBi slope \(m = 1/4\) and equal content scores, what is the ratio of the weight on a token 8 back to the weight on the token itself?
- BERT-base has \(L = 512\), \(d = 768\). How many parameters is its position table? Why does it not matter for the parameter count but matter a lot for usage?
Answerscheck
Key takeaways
- Self-attention without positions is permutation-equivariant (13.2): shuffle the tokens and the outputs only move. Pooled outputs are invariant (13.3). “Dog bites man” = “man bites dog”.
- A causal mask leaks some order, so NoPE decoders learn positions, but encoders, ViTs and video models need explicit positions.
- Absolute positions add a vector per slot (13.4). Sinusoidal vectors (13.5) use wavelengths from \(2\pi\) to \(2\pi \cdot 10^4\); a shift is a rotation (13.7), so dot products depend only on distance (13.8). Learned tables are simple but stop at \(L\).
- Relative methods put the distance in the score: Shaw adds a clipped vector to keys (13.10); T5 adds one learned scalar per head and log-spaced bucket (13.11–13.12); ALiBi subtracts a fixed slope times the distance (13.13–13.14).
- Past the training length: learned tables fail, sinusoidal values are unseen, T5 buckets saturate, ALiBi keeps working mostly as a recency window. RoPE, next session, is the method most current LLMs use.
Key formulas
References for this session
- Vaswani et al. (2017). Attention Is All You Need
- Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Press et al. (2021). Train Short, Test Long: Attention with Linear Biases
- Vaswani et al. (2017). Attention Is All You Need
- Shaw, Uszkoreit, Vaswani (2018). Self-Attention with Relative Position Representations
- Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Press et al. (2021). Train Short, Test Long: Attention with Linear Biases
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Haviv, Ram, Press, Izsak, Levy (2022). Transformer Language Models without Positional Encodings Still Learn Positional Information
- Kazemnejad, Padhi, Natesan Ramamurthy, Das, Reddy (2023). The Impact of Positional Encoding on Length Generalization in Transformers
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding