LLM & VLM courseBlog
13Track 1: Foundations · session 13 of 15

Position, part 1: why order needs help; absolute and relative positions

Self-attention looks at a set of tokens, not a list: if you shuffle the input, the outputs are shuffled the same way and nothing else changes. So a transformer needs extra help to tell “dog bites man” from “man bites dog”. This session shows why, then builds the two main fixes: absolute positions (sinusoidal and learned) added to the input, and relative positions (Shaw, T5 buckets, ALiBi) added to the attention scores, and ends with what each one does past the training length.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 2, 11 and 12 (dot products and softmax, embeddings, hidden states)

What you will learn

  • Why self-attention without positions is permutation-equivariant, and what that means for “dog bites man”
  • How a causal mask leaks some order information, and why that is not enough
  • Sinusoidal positions: frequencies, wavelengths, and why a fixed offset is a rotation
  • Learned absolute positions (BERT, GPT-2, ViT) and their hard length limit
  • Relative positions: Shaw et al., T5 bias buckets, and ALiBi linear biases with their slopes
  • What learned, sinusoidal, T5 and ALiBi positions do past the training length
  • Where position matters in VLMs and video: patches, frames and timestamps

Why this matters

“Dog bites man” is not news. “Man bites dog” is. The two sentences use the same three words; only the order is different. A transformer’s core operation, self-attention, cannot see that difference on its own. If you feed it the same tokens in a different order, it gives the same outputs, just moved around. This session explains why, and how models add the missing information.

In session 11 we turned each token into a vector by looking up a row in a table. That row depends only on which token it is, never on where it is: both copies of “the” in “the cat sat on the mat” get the same vector. In session 12 we saw that the transformer layers mix information between tokens to build contextual vectors. Here we look at the one piece of information that the mixing step does not provide: the position of each token.

There are two families of fixes, and both are still in use:

  • Absolute positions. Give each position \(t = 0, 1, 2, \dots\) its own vector and add it to the token embedding before the first layer. The original transformer used fixed sine and cosine waves; BERT, GPT-2 and ViT learn a table of position vectors.
  • Relative positions. Leave the inputs alone and instead change the attention score between two tokens depending on how far apart they are. T5 learns one number per distance bucket; ALiBi subtracts a penalty that grows linearly with distance. RoPE, used in Llama, Qwen and most current LLMs, is also relative; it gets its own session (session 14).

The choice matters most at one moment: when the model reads a text longer than anything it saw in training. Some methods break at once, some degrade slowly, some keep working in a limited way. This is the root of a whole research area on long context (session 28).

Interviewers ask about this often: “Is a transformer permutation-invariant?”, “Why sine and cosine?”, “What happens to GPT-2 at token 1,025?”, “How does T5 encode position?”, “Why does ALiBi extrapolate?”. By the end of this session you can answer each with a formula and a small example.

Self-attention ignores order

The attention formula, once more

Stack the \(T\) input vectors of a sequence as the rows of a matrix \(X \in \R^{T \times d}\). One head of self-attention computes queries, keys and values with three learned matrices, then a weighted average of the values (session 17 covers this in depth):

\[ Q = XW_Q,\quad K = XW_K,\quad V = XW_V,\qquad \mathrm{Attn}(X) = \softmax\!\Big(\frac{QK^\top}{\sqrt{d_k}}\Big)\,V \tag{13.1} \]
Read the symbols
\(X\)
the input: row \(t\) is the vector of token \(t\) (here, its embedding)
\(W_Q, W_K, W_V\)
learned matrices. They are the same for every position: the layer applies one rule to every row.
\(QK^\top\)
a \(T \times T\) table of scores: entry \((i, j)\) says how much token \(i\) wants to read from token \(j\)
\(\sqrt{d_k}\)
the square root of the key size; it keeps the scores in a good range
\(\softmax\)
applied to each row separately, so each row of weights sums to 1

Look at what the formula uses. Row \(i\) of the output depends on the query of token \(i\) and on the set of all (key, value) pairs. Nothing in the formula says “token \(j\) is the third one”. The score between two tokens depends only on their two vectors.

Permutation equivariance

Let \(P\) be a permutation matrix: a \(T \times T\) matrix with a single 1 in each row and column. \(PX\) is \(X\) with its rows shuffled. Then \(PQ = PXW_Q\), \(PK\), \(PV\) are shuffled the same way, the score table becomes \(P(QK^\top)P^\top\) (rows and columns shuffled together), the row-wise softmax just moves its entries along, and:

\[ \mathrm{Attn}(PX) = P\,\mathrm{Attn}(X) \tag{13.2} \]

This property is called permutation equivariance: shuffle the input, and the output is shuffled the same way, with no other change. The feed-forward block, layer norm and residual connections work on each row separately, so they are equivariant too. A whole transformer stack without positions is therefore equivariant. [Certain] If you then summarize the sequence with an average, the shuffle disappears completely:

\[ \frac{1}{T}\,\mathbf{1}^\top \mathrm{Attn}(PX) = \frac{1}{T}\,\mathbf{1}^\top P\,\mathrm{Attn}(X) = \frac{1}{T}\,\mathbf{1}^\top \mathrm{Attn}(X) \tag{13.3} \]

(since \(\mathbf{1}^\top P = \mathbf{1}^\top\): a permutation does not change a sum). The pooled vector is permutation-invariant. So a sentence classifier built this way would give exactly the same answer for “dog bites man” and “man bites dog”. The token “dog” gets the same output vector in both sentences, because it sees the same three keys and values, only listed in a different order.

Worked example: “dog” in both sentences

Figure 1 uses 4-number embeddings, for example dog \(= (0.9, 0.1, -0.3, 0.5)\), and fixed small matrices \(W_Q, W_K\) with \(W_V = I\). In “dog bites man” the scores of the query “dog” against the keys (dog, bites, man) are \((0.553, 0.091, 0.163)\), and softmax gives the weights \((0.434, 0.273, 0.293)\). In “man bites dog” the keys come in the order (man, bites, dog), so the scores are \((0.163, 0.091, 0.553)\) and the weights \((0.293, 0.273, 0.434)\). It is the same list in a different order. Each value is multiplied by the same weight as before, so the weighted sum is the same: \((0.482, 0.144, 0.155, 0.278)\) in both sentences.

The causal mask leaks a little order

A decoder (GPT-style) model adds a causal mask: token \(i\) may only read tokens \(j \le i\) (session 18). This breaks the symmetry. The first token sees only itself, the second sees two tokens, and so on. In “dog bites man”, “bites” sees {dog, bites}; in “man bites dog” it sees {man, bites}, so its output differs. A deep causal model can even learn to count how many tokens came before, and so recover absolute positions. Haviv et al. (2022) showed that causal language models with no position encoding (“NoPE”) still learn positional information and reach perplexity close to models with positions. [Certain] Kazemnejad et al. (2023) found that NoPE can generalize to longer inputs better than several explicit encodings on small tasks. [Certain]

But inside the visible prefix the order is still invisible to one attention layer: the last token of “dog bites man” and of “bites dog man” sees the same set {dog, bites, man} and gets the same output. Encoders (BERT, ViT, the encoder of T5) have no causal mask at all, so they need positions completely. That is why almost every model adds explicit position information.

Algorithm: one attention head, row by row
    Figure 1. One self-attention head (formula 13.1), computed live in your browser, with 4-number embeddings and fixed matrices \(W_Q, W_K\) (\(W_V = I\)). Top: the reference sentence “dog bites man” and the output vector \(\mathbf{o}\) of each word (4 bars). Bottom: the same words in a new order; dashed lines show where each word moved. The player computes the new order one query at a time: scores, softmax, output, then the distance \(\Delta\) to the output of the same word in the reference.
    How to read Figure 1

    Press Play (or Step). For each query (blue outline), the arcs first show the scores \(s_{ij}\) to every key, then the weights after softmax (thicker arc = larger weight). The live panel prints the numbers of each line of the algorithm. With both boxes unticked, look at the weights of a word in the new order: they are the same numbers as in the reference, only listed in a different order, so the output is the same and \(\Delta = 0.000\) for every word (formula 13.2). The last step compares the averaged sentence vectors: also identical (13.3). Press Shuffle for another order: still 0. Tick “Causal mask”: each slot sees only the keys up to itself, so the first slot attends only to itself, and the distances become larger than 0. This is the order leak of the causal mask. Tick “Add sinusoidal positions”: each input is now its embedding plus a vector for its slot, and every distance becomes larger than 0. The model can now tell the orders apart.

    Absolute positions: add a vector per slot

    The simplest fix: give every position \(t\) its own vector \(\mathbf{p}_t \in \R^d\) and add it to the token embedding before the first layer:

    \[ \mathbf{x}_t = \mathbf{e}_{\mathrm{id}_t} + \mathbf{p}_t \tag{13.4} \]
    Read the symbols
    \(\mathbf{e}_{\mathrm{id}_t}\)
    the token embedding of the token at position \(t\) (a row of the table \(E\) from session 11)
    \(\mathbf{p}_t\)
    the position vector for slot \(t\). The same \(\mathbf{p}_t\) is used for every sentence.
    \(\mathbf{x}_t\)
    the input of the first transformer layer at position \(t\)

    Now “dog” at slot 0 and “dog” at slot 2 are different vectors, so equivariance (13.2) no longer holds. Why add and not concatenate? Adding keeps the width \(d\) unchanged and costs nothing. In high dimension, the network can learn to keep the two kinds of information in roughly separate directions. [Likely] Concatenation also works but makes every layer wider.

    Sinusoidal positions

    The original transformer (Vaswani et al., 2017) used fixed waves, with no learned parameters. For position \(t\) and pair index \(i = 0, 1, \dots, d/2 - 1\):

    \[ p_{t,\,2i} = \sin(\omega_i\, t), \qquad p_{t,\,2i+1} = \cos(\omega_i\, t) \tag{13.5} \]
    \[ \omega_i = 10000^{-2i/d}, \qquad \lambda_i = \frac{2\pi}{\omega_i} = 2\pi \cdot 10000^{2i/d} \tag{13.6} \]
    Read the symbols
    \(t\)
    the position: 0 for the first token, 1 for the second, and so on
    \(i\)
    which pair of dimensions: each pair holds one sine and one cosine of the same frequency
    \(\omega_i\)
    the frequency of pair \(i\): how fast its angle turns per position. Pair 0 turns 1 radian per token; the last pairs turn very slowly.
    \(\lambda_i\)
    the wavelength: how many positions one full turn takes. It grows from \(2\pi \approx 6.3\) to about \(2\pi \cdot 10000\).
    10000
    the base, a fixed choice of the paper. It sets the longest wavelength.

    Think of a clock with many hands. The fast hand (pair 0) goes round every 6.3 tokens: it tells nearby tokens apart but repeats quickly. The slowest hands barely move: they tell “early in the text” from “late in the text”. Together, the hands give every position a different reading, like seconds, minutes and hours. For \(d = 512\), as in the original model, the wavelengths go from 6.28 to 60,611 positions. [Certain]

    Why sine and cosine? Because together they make a point on a circle, and moving by a fixed offset \(k\) is a rotation of that point, whatever the starting position \(t\):

    \[ \begin{pmatrix} p_{t+k,\,2i} \\ p_{t+k,\,2i+1} \end{pmatrix} = \begin{pmatrix} \cos \omega_i k & \sin \omega_i k \\ -\sin \omega_i k & \cos \omega_i k \end{pmatrix} \begin{pmatrix} p_{t,\,2i} \\ p_{t,\,2i+1} \end{pmatrix} \tag{13.7} \]

    This is just the sum formulas \(\sin(a + b) = \sin a \cos b + \cos a \sin b\) and \(\cos(a + b) = \cos a \cos b - \sin a \sin b\). The matrix depends only on \(k\), not on \(t\). So “the token 3 places back” is the same linear map at every position. The paper says it chose these waves for exactly this reason: it hoped the model could then easily learn to attend by relative positions. [Certain] A direct consequence: the dot product of two position vectors depends only on their distance:

    \[ \mathbf{p}_t \cdot \mathbf{p}_{s} = \sum_{i=0}^{d/2-1} \big(\sin \omega_i t \,\sin \omega_i s + \cos \omega_i t\, \cos \omega_i s\big) = \sum_{i=0}^{d/2-1} \cos\big(\omega_i (t - s)\big) \tag{13.8} \]

    It equals \(d/2\) at distance 0 and mostly gets smaller as the distance grows (with small ripples). Figure 2 draws it.

    Worked example: \(d = 4\) by hand

    With \(d = 4\) there are two pairs: \(\omega_0 = 1\) and \(\omega_1 = 10000^{-2/4} = 0.01\). So \(\mathbf{p}_t = (\sin t, \cos t, \sin 0.01t, \cos 0.01t)\).

    \(\mathbf{p}_0 = (0, 1, 0, 1)\), \(\mathbf{p}_1 = (0.8415, 0.5403, 0.0100, 1.0000)\), \(\mathbf{p}_2 = (0.9093, -0.4161, 0.0200, 0.9998)\).

    Dot product: \(\mathbf{p}_1 \cdot \mathbf{p}_2 = 0.7651 - 0.2248 + 0.0002 + 0.9998 = 1.5403\), and formula (13.8) gives \(\cos 1 + \cos 0.01 = 0.5403 + 0.99995 = 1.5403\). The same. And \(\mathbf{p}_1 \cdot \mathbf{p}_3 = \mathbf{p}_5 \cdot \mathbf{p}_7 = 0.5837\): both pairs are 2 apart, so they get the same number.

    Figure 2. Sinusoidal positions, formulas (13.5) and (13.6). Top: the position vectors \(\mathbf{p}_0, \dots, \mathbf{p}_{63}\) as rows (columns are dimensions; blue is +1, orange is −1). Bottom: the similarity \(\mathbf{p}_t \cdot \mathbf{p}_s / (d/2)\) of the chosen position \(t\) with every position \(s\), formula (13.8).
    How to read Figure 2

    In the heatmap, the left columns change color quickly from row to row (fast hands), and the right columns hardly change (slow hands). The outlined row is \(\mathbf{p}_t\). The curve below is 1 at \(s = t\) and falls off on both sides with the same shape: move the slider and the curve only slides, it does not change shape. That is formula (13.8): similarity depends only on \(t - s\). The readout checks this by comparing \(\mathbf{p}_t \cdot \mathbf{p}_{t+8}\) with \(\mathbf{p}_0 \cdot \mathbf{p}_8\). With a small \(d\) the curve has big ripples; with more pairs it becomes smoother.

    Figure 3 runs formulas (13.5) to (13.8) step by step. It first builds \(\mathbf{p}_t\) one pair at a time: compute the frequency, turn the hand of that pair to the angle \(\omega_i t\), and read off the sine and the cosine. Then it shifts every hand by \(k\) positions with the rotation (13.7) and adds up the dot product pair by pair.

    Algorithm: build \(\mathbf{p}_t\), then shift by \(k\)
      Figure 3. A sinusoidal position vector built dimension by dimension. Each dial is one pair \(i\); the blue hand points to the angle \(\omega_i t\), so its tip is \((\cos \omega_i t,\ \sin \omega_i t)\) and the pair stores \(p_{t,2i} = \sin \omega_i t\), \(p_{t,2i+1} = \cos \omega_i t\) (13.5). In the second half, the orange hand is position \(t + k\): the same hand turned by the green angle \(\omega_i k\) (13.7). The strips below show the numbers of \(\mathbf{p}_t\) and \(\mathbf{p}_{t+k}\) as they are filled in.
      How to read Figure 3

      Press Play. Steps 2 to 5 (for \(d = 8\)) each compute one pair: the hand turns to its angle and two numbers of \(\mathbf{p}_t\) appear. Pair 0 turns 1 radian per position, so at \(t = 10\) it has gone round more than once; pair 3 turns only 0.001 radian per position and has barely moved. The next steps rotate each pair by \(\omega_i k\) and add \(\cos(\omega_i k)\) to the running sum \(S\). For \(t = 10\), \(k = 3\), \(d = 8\) the sum is \(-0.9900 + 0.9553 + 0.9996 + 1.0000 = 1.9649\). The last step checks that \(S\) equals the direct dot product \(\mathbf{p}_t \cdot \mathbf{p}_{t+k}\) and also \(\mathbf{p}_0 \cdot \mathbf{p}_k\). Move \(t\): all blue hands move, but the green angles and \(S\) do not change, because they depend only on \(k\) (13.8).

      Learned absolute positions

      The other option is to learn the position vectors: a second embedding table \(P \in \R^{L \times d}\) with one row per position, trained by backpropagation like the token table. \(L\) is the maximum length, fixed before training. BERT uses \(L = 512\) and GPT-2 uses \(L = 1024\); with \(d = 768\) these tables have 393,216 and 786,432 parameters, tiny next to the rest of the model. [Certain] The Vision Transformer learns one vector per image patch position in the same way (session 164).

      Vaswani et al. tried both and found “nearly identical results”. [Certain] Learned tables are simpler and can fit whatever the data needs. Their big weakness is the hard limit: there is no row for position \(L\). GPT-2 simply cannot be run on token 1,025 without changing the model. We come back to this below.

      What absolute positions do inside attention

      Absolute vectors enter only at the bottom, but they shape every score. Put (13.4) into the score of the first layer (ignore the \(\sqrt{d_k}\) for now, and write \(W = W_Q W_K^\top\)):

      \[ (\mathbf{e}_i + \mathbf{p}_i)\,W\,(\mathbf{e}_j + \mathbf{p}_j)^\top = \underbrace{\mathbf{e}_i W \mathbf{e}_j^\top}_{\text{content–content}} + \underbrace{\mathbf{e}_i W \mathbf{p}_j^\top}_{\text{content–position}} + \underbrace{\mathbf{p}_i W \mathbf{e}_j^\top}_{\text{position–content}} + \underbrace{\mathbf{p}_i W \mathbf{p}_j^\top}_{\text{position–position}} \tag{13.9} \]

      The last term is where order enters. With sinusoidal vectors and \(W = I\) it would depend only on \(i - j\) (formula 13.8). But for a general learned \(W\), the term \(\mathbf{p}_i W \mathbf{p}_j^\top\) can depend on \(i\) and \(j\) separately. So absolute encodings let the model use relative distance, but do not force it to. The middle terms mix content and position, which is often useless noise. These two observations motivated the relative methods: put the distance directly into the score instead.

      Relative positions: put the distance in the score

      For language, “the word two places before me” usually matters more than “the word at slot 517”. A relative method makes the score between query \(i\) and key \(j\) depend on the offset \(j - i\) (or the distance \(i - j\)), and does not add anything to the inputs.

      Shaw et al.: learned relative vectors

      Shaw, Uszkoreit and Vaswani (2018) add a learned vector \(\mathbf{a}_{j-i}\) to each key, chosen by the offset, and clip offsets beyond a maximum \(k\):

      \[ s_{ij} = \frac{\mathbf{q}_i \cdot \big(\mathbf{k}_j + \mathbf{a}_{\mathrm{clip}(j - i,\, -k,\, k)}\big)}{\sqrt{d_k}} \tag{13.10} \]

      There are only \(2k + 1\) different vectors, shared by all positions. Offsets farther than \(k\) all share the edge vector. On WMT 2014 translation this improved BLEU by 1.3 (English–German) and 0.3 (English–French) over absolute positions. [Certain] Transformer-XL and DeBERTa later used related ideas. [Certain]

      T5: one learned number per distance bucket

      T5 (Raffel et al., 2019) simplifies this a lot. Instead of a vector, each offset gets a single learned scalar that is added to the score:

      \[ s_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j + b_{h,\ \mathrm{bucket}(j - i)} \tag{13.11} \]
      Read the symbols
      \(b_{h,\,n}\)
      a learned number for head \(h\) and bucket \(n\). Different heads can prefer different distances.
      \(\mathrm{bucket}(j - i)\)
      maps the offset to one of 32 buckets: exact for small distances, coarser for large ones
      no \(\sqrt{d_k}\)
      T5 folds the usual scale into the initialization of \(W_Q\), so the score has no division. [Certain]

      The bias table is shared by all layers of a stack; each head has its own column. With 32 buckets and 12 heads (T5-base) that is only \(32 \times 12 = 384\) numbers per stack. [Likely] The bucket map is the clever part. Nearby distances get their own bucket, because “the previous word” and “two words back” are very different. Far distances share buckets on a log scale, because “100 back” and “110 back” rarely need different treatment. For a decoder (causal, only the past counts), with \(n = 32\) buckets and maximum distance \(D = 128\), let \(r = \max(0,\ i - j)\) be how far back the key is:

      \[ \mathrm{bucket}(r) = \begin{cases} r & \text{if } r \lt n/2 \\[4pt] \min\!\Big(n - 1,\ \ \dfrac{n}{2} + \Big\lfloor \dfrac{\log\big(r/(n/2)\big)}{\log\big(D/(n/2)\big)} \cdot \dfrac{n}{2} \Big\rfloor\Big) & \text{otherwise} \end{cases} \tag{13.12} \]

      In the encoder (bidirectional) the 32 buckets are split in two: 16 for keys before the query and 16 for keys after it, each with 8 exact buckets and 8 log-spaced ones. [Certain] The formula is the one in the T5 code (and in Hugging Face’s _relative_position_bucket).

      Worked example: T5 buckets

      Decoder (\(n = 32\), \(D = 128\)): distances 0 to 15 get buckets 0 to 15. Distance 16 gives \(16 + \lfloor 0 \rfloor = 16\). Distance 19 gives \(16 + \lfloor \log(19/16)/\log 8 \cdot 16 \rfloor = 16 + \lfloor 1.32 \rfloor = 17\). Bucket 16 covers distances 16–18, bucket 17 covers 19–20, bucket 30 covers 99–112, and bucket 31 covers every distance from 113 up.

      Encoder: keys 0 to 7 back get buckets 0 to 7; 8 to 11 back share bucket 8; 91 or more back share bucket 15. Keys after the query use buckets 16 to 31: one step ahead is bucket 17, and 91 or more ahead is bucket 31.

      Figure 4. The T5 bucket map, formula (13.12), with 32 buckets and maximum distance 128. The horizontal axis is the offset \(j - i\) of the key from the query (negative: the key is in the past). The vertical axis is the bucket. Untick “Bidirectional” to see the decoder version, where all future offsets are masked and all buckets go to the past.
      How to read Figure 4

      Near 0 the staircase has steps of width 1: every small distance has its own learned bias. Farther away the steps get wider and wider; past 128 the line is flat, so all far tokens share one bias. Move the slider: the readout shows the bucket and all the offsets that share it. In the encoder there are two staircases, one for the past (buckets 0–15) and one for the future (16–31). In the decoder, the future side is gone (those keys are masked) and the past side gets all 32 buckets, so it is finer: 16 exact buckets instead of 8.

      ALiBi: a fixed linear penalty

      ALiBi (“Attention with Linear Biases”, Press, Smith and Lewis, 2021) goes one step further: nothing is learned. Each head subtracts a penalty proportional to the distance, and no position vector is added to the inputs:

      \[ s_{ij} = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}} - m_h\,(i - j), \qquad j \le i \tag{13.13} \]
      \[ m_h = 2^{-8h/H}, \qquad h = 1, 2, \dots, H \tag{13.14} \]
      Read the symbols
      \(i - j\)
      how many tokens back the key is (0 for the token itself). ALiBi was designed for causal models, so \(j \le i\).
      \(m_h\)
      the slope of head \(h\), a fixed number, not learned. A large slope means “look only at the last few tokens”; a small one means “look far back”.
      \(H\)
      the number of heads. For \(H = 8\) the slopes are \(1/2, 1/4, \dots, 1/256\): a geometric sequence. [Certain]

      Why is this a good idea? After softmax, a bias of \(-m\Delta\) multiplies the weight of a key \(\Delta\) tokens back by \(e^{-m\Delta}\). So each head has a built-in preference for recent tokens, with its own “half-life”, while the content scores can still override it. The heads together cover a range of scales, like the many wavelengths of the sinusoidal method:

      \[ \frac{a_{i,\,i-\Delta}}{a_{i,\,i}} = e^{\,s^{c}_{i,i-\Delta} - s^{c}_{i,i}}\cdot e^{-m_h \Delta}, \qquad \Delta_{\text{reach}} \approx \frac{c}{m_h} \tag{13.15} \]
      Read the symbols
      \(a_{i,j}\)
      the attention weight from query \(i\) to key \(j\) after softmax
      \(s^{c}\)
      the content part of the score, \(\mathbf{q} \cdot \mathbf{k}/\sqrt{d_k}\)
      \(\Delta_{\text{reach}}\)
      a rough reach of the head: the distance at which the penalty is \(c\). With \(c = 10\) the weight is cut by \(e^{-10} \approx 1/22{,}000\).
      Worked example: ALiBi weights

      Take \(H = 8\), head 1, so \(m = 1/2\). A query at position 5 sees keys 0 to 5 at distances 5, 4, 3, 2, 1, 0. Suppose all content scores are equal (0). The biases are \((-2.5, -2, -1.5, -1, -0.5, 0)\) and softmax gives the weights \((0.034, 0.056, 0.092, 0.152, 0.251, 0.414)\): strongly recent. For head 8, \(m = 1/256\), the weights are almost flat: from 0.165 to 0.168.

      Reach with \(c = 10\): head 1 reaches about 20 tokens, head 8 about 2,560. With equal content scores over 2,048 keys, head 1 puts 99.3% of its weight on the last 10 tokens, and head 8 puts 63.2% on the last 256.

      Algorithm: one row of ALiBi attention
        Figure 5. ALiBi, formulas (13.13) and (13.14), computed live on 8 tokens. Left: the bias \(-m_h(i - j)\) for every query row \(i\) and key column \(j\); the upper triangle is masked (causal), and darker red means a larger penalty. Right, top: for the chosen query row, the content score \(c_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j / \sqrt{4}\) (outline) and the score after the bias is added (solid); the red line is the bias. Right, bottom: the attention weights with the bias (solid) and without it (dashed). The vectors are fixed toy numbers; key 0 has the highest content score. On a phone the charts are below the matrix.
        How to read Figure 5

        The bias matrix is constant along each diagonal: the penalty depends only on the distance, never on the absolute position. That is why ALiBi is a relative method. Press Play: the player walks along the outlined row, one key per step, and the live panel substitutes the numbers into \(s_{ij} = c_{ij} - m(i - j)\). With head 1 (\(m = 1/2\)) the key 7 back gets a bias of \(-3.5\), so its score falls from 0.850 to \(-2.650\), and after softmax its weight drops from 0.229 (no bias) to 0.021, while the token itself gets 0.442. Move the slider to head 8: the slope is \(1/256\), the red lines almost vanish, and the solid and dashed weights nearly coincide. With \(H = 16\) the slopes start at \(2^{-0.5} = 0.707\) and also end at \(1/256\). Tick “Equal content scores” and set query \(i = 5\) to reproduce the worked example above: weights 0.034 to 0.414 for head 1.

        Past the training length

        A model is trained with sequences up to some length \(L\). What happens when it reads a longer text at test time? This question, called length extrapolation, separates the methods clearly.

        • Learned absolute. There is no row for position \(L\) or beyond. GPT-2 with \(L = 1024\) gives an index error at the 1,025th token. You must cut the input, use a sliding window, or extend the table (copy or interpolate rows) and fine-tune. ViT does the same for images: at a higher resolution there are more patches, so the position table is resized by 2D interpolation. [Certain]
        • Sinusoidal. The formula works for any \(t\), so nothing crashes. But the model has never seen these vectors. The slow pairs, those with a wavelength longer than \(L\), have only turned part of the way round during training; past \(L\) they point in directions never seen before (Figure 6). In the ALiBi paper, the perplexity of a sinusoidal model rises quickly once the input is longer than \(L\). [Certain]
        • T5 bias. Every distance beyond 128 falls in the last bucket, which was trained. So any length gets a valid bias, and the ALiBi paper found that T5’s bias does extrapolate to some degree, but it was slower to compute than the other methods they compared. [Likely]
        • ALiBi. A longer distance just gives a larger penalty of the same form, which the model has already learned to handle. Perplexity stays roughly flat well past \(L\). [Certain] The price: far tokens get tiny weights, so in practice the model mostly uses a recent window. Good perplexity at long lengths does not prove that it can use information from 10,000 tokens back. [Likely]
        • RoPE. Like sinusoidal, it is defined everywhere but extrapolates poorly without help. Position interpolation and YaRN rescale its angles to stretch the context (session 14, session 28).

        The count in Figure 6 has a simple formula. Pair \(i\) completes at least one full turn during training if \(\lambda_i \le L\). Using (13.6), the number of pairs that do not is the number of \(i\) with \(2\pi \cdot 10000^{2i/d} \gt L\). For \(d = 512\) and \(L = 1024\), that is 114 of the 256 pairs; for \(L = 4096\), still 75. [Certain]

        Figure 6. Why sinusoidal positions do not extrapolate, with \(d = 512\) as in the original transformer. Each dial is one sine–cosine pair \(i\) (its wavelength \(\lambda_i\) is printed below it). The colored arc is the set of angles \(\omega_i t\) that the pair took for training positions \(t = 0, \dots, L - 1\). The dot is the angle at the test position. An orange dot lies outside the arc: a value never seen in training.
        How to read Figure 6

        The fast pairs (left) went round many times during training, so their arc is a full circle and every test angle is familiar. The slow pairs (right) covered only part of the circle. Move the test position past \(L\): their dots leave the arc and turn orange. The readout counts how many of the 256 pairs show a new angle. Raise \(L\): fewer pairs are new, but the slowest ones (wavelength 60,611) never complete a turn for any realistic \(L\). Below \(L\) the count is 0 by definition. A learned table would simply have no row there.

        MethodWhere it actsLearned?Past training lengthUsed in
        None (NoPE)only through the causal mask—no hard limit; can generalize on some tasksresearch models
        Sinusoidaladded to input (13.5)nodefined, but unseen values; degrades fastoriginal transformer
        Learned absoluteadded to input (13.4)yes, \(L \times d\)impossible without resizingBERT, GPT-2, ViT
        Shaw relativevector added to keys (13.10)yes, \((2k+1) \times d_k\)clipped: far offsets share one vectorMusic Transformer and others [Likely]
        T5 biasscalar added to scores (13.11)yes, 32 × headsfar offsets share the last bucketT5, Flan-T5
        ALiBilinear penalty on scores (13.13)noworks, mostly as a recency windowBLOOM, MPT
        RoPErotates \(\mathbf{q}\) and \(\mathbf{k}\)noneeds scaling (PI, YaRN)Llama, Qwen, most current LLMs

        Position beyond text: patches, frames and time

        Images and videos have more than one axis, and the same two families appear there.

        • Images. ViT cuts an image into patches and adds a learned 1D position vector per patch index. The paper reports that 2D-aware position embeddings gave no significant gain over 1D ones in their setting. [Certain] Swin uses a learned relative bias, much like T5, inside each local window. [Certain]
        • Video. Order in time decides meaning: “opening a door” played backwards is “closing a door”. Video transformers add a time position on top of the spatial one, for example separate learned spatial and temporal embeddings. [Likely] Without a time signal, a video model is a bag of frames and cannot tell the two actions apart.
        • VLMs. When image tokens are placed inside a text sequence, the model needs both a sequence position (for the LLM) and a 2D position (for the image layout). Qwen2-VL’s M-RoPE gives each token separate time, height and width positions; we preview it in the next session. [Certain]

        Where this is used

        ApplicationWhat position doesLink
        Every LLMTells word order apart (“dog bites man”); modern models use RoPE, older ones learned or ALiBi positionssession 14
        Encoders for classification and searchBERT-style encoders have no causal mask, so without positions a pooled sentence vector would be a bag of words (13.3)session 33
        Long-document models and RAGThe choice of encoding decides how far past the training length a model can readsession 28
        Image encoders in VLMsLearned patch positions are interpolated when the image resolution changessession 164
        Action recognitionA time position lets the model separate actions that differ only in order (open vs close, push vs pull)session 202
        Key-moment detection and video QAFrame positions or timestamps let the model answer “when did it happen?” and point to a timesession 210
        Video fingerprintingMatching two clips uses the order of frame descriptors, not just the set; a bag-of-frames match cannot detect reordered editssession 209
        Encoder–decoder translation and summarizationT5’s bucketed relative bias in both the encoder and the decodersession 24

        Pseudo-code

        Algorithm 1: absolute positions (sinusoidal or learned)
        1. input: token IDs \(\mathrm{id}_0, \dots, \mathrm{id}_{T-1}\), token table \(E\), mode, max length \(L\)
        2. if mode = learned and \(T \gt L\) then error // no row for position L
        3. for \(t = 0\) to \(T - 1\) do
        4. if mode = sinusoidal then
        5. for \(i = 0\) to \(d/2 - 1\) do \(\omega \leftarrow 10000^{-2i/d}\); \(p_{2i} \leftarrow \sin \omega t\); \(p_{2i+1} \leftarrow \cos \omega t\) // (13.5)
        6. else \(\mathbf{p} \leftarrow P_{t,:}\) // learned row
        7. \(\mathbf{x}_t \leftarrow E_{\mathrm{id}_t,:} + \mathbf{p}\) // (13.4)
        8. return \(X\) // then the usual attention (13.1)
        Algorithm 2: relative bias in the scores (T5 or ALiBi), one head
        1. input: \(Q, K, V \in \R^{T \times d_k}\), head \(h\), mode
        2. \(S \leftarrow QK^\top\) // divide by \(\sqrt{d_k}\) for ALiBi; T5 does not
        3. for \(i = 0\) to \(T - 1\) do
        4. for \(j = 0\) to \(T - 1\) do
        5. if causal and \(j \gt i\) then \(S_{ij} \leftarrow -\infty\); continue
        6. if mode = T5 then \(S_{ij} \leftarrow S_{ij} + b_{h,\ \mathrm{bucket}(j - i)}\) // learned scalar, (13.11)–(13.12)
        7. else \(S_{ij} \leftarrow S_{ij} - m_h (i - j)\) // fixed slope \(m_h = 2^{-8h/H}\), (13.13)–(13.14)
        8. return \(\softmax(S)\,V\) // row-wise softmax; no position added to inputs

        The same ideas in PyTorch. Each part checks one claim of this session; the printed values are in the comments:

        import math
        import torch
        torch.manual_seed(0)
        
        # 1) self-attention without positions is permutation-equivariant, formula (13.2)
        T, d = 5, 8
        X = torch.randn(T, d)
        Wq, Wk, Wv = torch.randn(d, d), torch.randn(d, d), torch.randn(d, d)
        def attn(X, bias=None):
            S = (X @ Wq) @ (X @ Wk).T / math.sqrt(d)
            if bias is not None:
                S = S + bias
            return torch.softmax(S, dim=-1) @ (X @ Wv)
        perm = torch.tensor([4, 2, 0, 3, 1])
        print(torch.allclose(attn(X[perm]), attn(X)[perm], atol=1e-6))           # True: outputs just move
        print(torch.allclose(attn(X[perm]).mean(0), attn(X).mean(0), atol=1e-6)) # True: pooled vector identical (13.3)
        
        # 2) sinusoidal positions, formula (13.5)
        def sinusoidal(T, d, base=10000.0):
            pos = torch.arange(T).float()[:, None]                  # (T, 1)
            w = base ** (-torch.arange(0, d, 2).float() / d)        # (d/2,) frequencies
            P = torch.zeros(T, d)
            P[:, 0::2] = torch.sin(pos * w)
            P[:, 1::2] = torch.cos(pos * w)
            return P
        P = sinusoidal(16, 4)
        print(P[1].round(decimals=4), P[2].round(decimals=4))
        # tensor([0.8415, 0.5403, 0.0100, 1.0000]) tensor([ 0.9093, -0.4161,  0.0200,  0.9998])
        print(round((P[1] @ P[3]).item(), 4), round((P[5] @ P[7]).item(), 4))  # 0.5837 0.5837 (13.8)
        pe = sinusoidal(T, d)
        print(torch.allclose(attn(X[perm] + pe), attn(X + pe)[perm], atol=1e-6))  # False: order now matters
        
        # 3) T5 relative position buckets, formula (13.12)
        def t5_bucket(rel, bidirectional=True, num_buckets=32, max_distance=128):
            # rel = key_position - query_position
            out = torch.zeros_like(rel)
            if bidirectional:
                num_buckets //= 2
                out += (rel > 0).long() * num_buckets
                n = rel.abs()
            else:
                n = (-rel).clamp(min=0)
            max_exact = num_buckets // 2
            large = max_exact + (torch.log(n.float().clamp(min=1) / max_exact)
                                 / math.log(max_distance / max_exact)
                                 * (num_buckets - max_exact)).long()
            large = large.clamp(max=num_buckets - 1)
            return out + torch.where(n < max_exact, n, large)
        rel = torch.tensor([0, -1, -7, -8, -12, -32, -128, -1000, 1, 8, 128])
        print(t5_bucket(rel).tolist())      # [0, 1, 7, 8, 9, 12, 15, 15, 17, 24, 31]
        print(t5_bucket(-torch.tensor([0, 15, 16, 19, 113, 5000]), bidirectional=False).tolist())
        # [0, 15, 16, 17, 31, 31]
        
        # 4) ALiBi bias, formulas (13.13) and (13.14)
        def alibi_slopes(H):
            s = 2 ** (-8 / H)
            return torch.tensor([s ** (h + 1) for h in range(H)])
        def alibi_bias(T, H):
            i = torch.arange(T)[:, None]; j = torch.arange(T)[None, :]
            dist = (i - j).clamp(min=0).float()                     # how far back key j is
            causal = torch.full((T, T), float("-inf")).triu(1)
            return -alibi_slopes(H)[:, None, None] * dist + causal  # (H, T, T)
        print(alibi_slopes(8).tolist())     # [0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625, 0.0078125, 0.00390625]
        B = alibi_bias(6, 8)
        print(torch.softmax(B[0, 5], -1).round(decimals=3))
        # tensor([0.0340, 0.0560, 0.0920, 0.1520, 0.2510, 0.4140])  head 1, query at position 5
        print(attn(X, B[0][:T, :T]).shape)  # torch.Size([5, 8]): the bias plugs straight into the scores

        Common mistakes

        • Saying “transformers are permutation-invariant”. Self-attention is equivariant (13.2); only a pooled output is invariant (13.3). And a causal decoder is not fully either, because the mask leaks order.
        • Forgetting positions in a custom encoder. A small transformer over frames, patches or set elements without any position signal treats the input as a bag. It may still train, and quietly fail on order-dependent examples.
        • Running a learned-position model past \(L\). GPT-2 or BERT past 1,024 or 512 tokens is an index error, or, with some code, silent reuse of wrong rows. Truncate, use a sliding window, or resize and fine-tune.
        • Mixing up the sinusoidal layout. Some code interleaves sin and cos (dimensions \(2i, 2i + 1\)); other code puts all sines first and all cosines second. Both are valid, but loading weights trained with one layout into code using the other breaks the model.
        • Getting the sign of the offset wrong. T5 uses \(j - i\) (key minus query); ALiBi penalizes \(i - j\). Flip a sign and a decoder attends to the wrong side, or every distance lands in a wrong bucket.
        • Treating ALiBi slopes as learnable or reusing 8-head slopes for 16 heads. The slopes are fixed and depend on \(H\) (13.14). Changing the head count changes the slopes.
        • Reading flat perplexity past \(L\) as “uses the whole context”. With ALiBi, distant tokens get tiny weights. Test retrieval from far back (a needle-in-a-haystack test), not only perplexity. [Likely]
        • Resizing image position tables naively. When ViT runs at a new resolution, the position grid must be interpolated in 2D, not as a flat 1D list; otherwise rows and columns get mixed.

        Interview questions

        Why does a transformer need position information at all?basic
        Self-attention computes each output from the query of a token and the set of all keys and values; the weights \(W_Q, W_K, W_V\) are shared by all positions, and nothing in (13.1) refers to an index. So shuffling the inputs shuffles the outputs the same way, \(\mathrm{Attn}(PX) = P\,\mathrm{Attn}(X)\) (13.2), and any pooled summary is unchanged (13.3). “Dog bites man” and “man bites dog” would look the same. Positions break this symmetry.
        Is a transformer permutation-invariant or permutation-equivariant? What changes with a causal mask?core
        Without positions, each layer is equivariant: outputs move with the inputs. A mean or sum pooled output is invariant. A causal mask makes the visible set depend on the position (token \(i\) sees \(i + 1\) tokens), so outputs change with order and a deep model can learn to count, which is why NoPE decoders still learn positional information (Haviv et al., 2022). The order inside the visible prefix is still invisible to a single attention layer.
        Write the sinusoidal position encoding. Why use both sine and cosine, and why many frequencies?core
        \(p_{t,2i} = \sin(t \cdot 10000^{-2i/d})\), \(p_{t,2i+1} = \cos(t \cdot 10000^{-2i/d})\) (13.5). A sine–cosine pair is a point on a circle, so shifting by \(k\) is a rotation that depends only on \(k\) (13.7); this makes relative offsets a fixed linear map and gives \(\mathbf{p}_t \cdot \mathbf{p}_s = \sum_i \cos(\omega_i(t - s))\) (13.8). Many frequencies, with wavelengths from \(2\pi\) to about \(2\pi \cdot 10^4\), give fine resolution for nearby tokens and unique codes for far ones, like the hands of a clock.
        Learned vs sinusoidal absolute positions: what are the trade-offs?core
        Learned tables (BERT, GPT-2, ViT) cost \(L \times d\) parameters (786,432 for GPT-2), can fit any pattern, but have a hard maximum length \(L\) and rows for rare late positions get few updates. Sinusoidal has no parameters and is defined for any \(t\), but values past \(L\) were never seen, so quality still drops. Vaswani et al. found nearly identical results on translation. Both only enter at the input, and through (13.9) they also create content–position cross terms.
        Expand the first-layer attention score with absolute positions. What does it tell you?advanced
        \((\mathbf{e}_i + \mathbf{p}_i)W(\mathbf{e}_j + \mathbf{p}_j)^\top\) has four terms: content–content, content–position, position–content, position–position (13.9). Only the last directly encodes order, and for a learned \(W\) it can depend on \(i\) and \(j\) separately, not only on \(i - j\). So absolute encodings allow but do not enforce relative behavior, and the mixed terms add noise. Relative methods (Shaw, T5, ALiBi, RoPE) instead make the score depend on the offset directly.
        How does T5 encode position? Explain the buckets.core
        T5 adds a learned scalar \(b_{h,\mathrm{bucket}(j - i)}\) to each attention logit (13.11), one per head and bucket, shared across the layers of a stack; nothing is added to the inputs. There are 32 buckets. In the encoder, half are for the past and half for the future; in each half, 8 exact buckets cover distances 0–7, and 8 log-spaced buckets cover up to 128; everything farther shares the last bucket (13.12). The decoder uses all 32 for the past (16 exact). Fine detail where it matters, coarse far away, and every length gets a valid bucket.
        What is ALiBi and how are its slopes chosen?core
        ALiBi adds \(-m_h(i - j)\) to the causal attention logits of head \(h\) (13.13), and adds no position vectors to the inputs. The slopes are fixed, a geometric sequence \(m_h = 2^{-8h/H}\) (13.14): for 8 heads \(1/2, \dots, 1/256\). Each head is a soft recency window of reach about \(c/m_h\) (13.15): head 1 about 20 tokens, head 8 about 2,560 for \(c = 10\). There are no position parameters at all.
        Why does ALiBi extrapolate better than sinusoidal positions? What is the catch?advanced
        Past \(L\), ALiBi only produces larger penalties of the same linear form, which the model already handles; the far tokens are simply down-weighted, so nothing out-of-distribution reaches the network. Sinusoidal vectors past \(L\) are new inputs: the slow pairs point to angles never seen (114 of 256 pairs for \(d = 512\), \(L = 1024\)). The catch: ALiBi’s good long-range perplexity largely comes from attending to a recent window; it does not mean the model retrieves facts from far back. Long-context benchmarks with retrieval are needed to check that.
        GPT-2 has 1,024 learned positions. You need to process 4,000 tokens. What are your options?advanced
        (1) Chunk or slide a window of 1,024 tokens, possibly with overlap and a summary carried between chunks. (2) Extend the position table: initialize new rows by copying or interpolating the old ones (stretch the 1,024 rows over 4,000 slots), then fine-tune on long data. (3) Switch to a model with a relative encoding (RoPE with scaling, ALiBi). Option 2 without fine-tuning usually hurts quality, because the model has never seen the new rows. [Likely]
        How do vision transformers and video models handle position, and what happens at a new resolution?core
        ViT adds a learned vector per patch index (1D; 2D-aware variants gave no significant gain in the paper). At a new resolution the number of patches changes, so the table is reshaped into its 2D grid and interpolated, then fine-tuned. Video models add a temporal position on top, so frame order is visible; without it, the model cannot tell “open the door” from its reverse. VLMs like Qwen2-VL use M-RoPE with separate time, height and width positions.
        Why not just concatenate a one-hot position vector to each token?basic
        It works in principle but has the same problems as one-hot tokens: the size grows with the maximum length, every position is equally far from every other (no notion that 5 is close to 6), and it still has a hard limit. Adding a dense \(d\)-dimensional position vector keeps the width fixed and can make nearby positions similar (sinusoidal does so by construction, formula 13.8).

        Exercises

        1. Show that \(\mathrm{softmax}\) applied row-wise satisfies \(\softmax(PSP^\top) = P\,\softmax(S)\,P^\top\) for a permutation matrix \(P\), and use it to prove (13.2).
        2. For \(d = 4\) sinusoidal positions, compute \(\mathbf{p}_0 \cdot \mathbf{p}_1\) using formula (13.8).
        3. For \(d = 512\), what is the wavelength of pair \(i = 128\)? How many positions does pair \(i = 0\) need for one full turn?
        4. In a T5 decoder (32 buckets, max distance 128), which bucket does distance 20 fall in? Distance 500?
        5. An ALiBi model has 16 heads. What are the slopes of heads 1, 2 and 16?
        6. With ALiBi slope \(m = 1/4\) and equal content scores, what is the ratio of the weight on a token 8 back to the weight on the token itself?
        7. BERT-base has \(L = 512\), \(d = 768\). How many parameters is its position table? Why does it not matter for the parameter count but matter a lot for usage?
        Answerscheck
        1. \(PSP^\top\) permutes rows and columns of \(S\) the same way. Softmax on a row is unchanged by reordering that row’s entries (they move along), and reordering rows just reorders the outputs, so the result is \(P\,\softmax(S)\,P^\top\). Then \(\mathrm{Attn}(PX) = P\,\softmax(S)\,P^\top P V = P\,\softmax(S) V\), since \(P^\top P = I\). 2. \(\cos 1 + \cos 0.01 = 0.5403 + 0.99995 = 1.5403\) (the same as \(\mathbf{p}_1 \cdot \mathbf{p}_2\): distance 1 in both). 3. \(\lambda_{128} = 2\pi \cdot 10000^{256/512} = 2\pi \cdot 100 \approx 628.3\); pair 0 has \(\lambda_0 = 2\pi \approx 6.28\) positions. 4. Distance 20: \(16 + \lfloor \log(20/16)/\log 8 \cdot 16 \rfloor = 16 + \lfloor 1.73 \rfloor = 17\). Distance 500 is beyond 113, so bucket 31. 5. \(m_h = 2^{-8h/16} = 2^{-h/2}\): head 1 \(= 0.707\), head 2 \(= 0.5\), head 16 \(= 2^{-8} = 1/256\). 6. \(e^{-8/4} = e^{-2} = 0.135\). 7. \(512 \times 768 = 393{,}216\), a tiny part of 110 M; but it fixes the maximum input at 512 tokens, so longer documents must be cut or chunked.

        Key takeaways

        • Self-attention without positions is permutation-equivariant (13.2): shuffle the tokens and the outputs only move. Pooled outputs are invariant (13.3). “Dog bites man” = “man bites dog”.
        • A causal mask leaks some order, so NoPE decoders learn positions, but encoders, ViTs and video models need explicit positions.
        • Absolute positions add a vector per slot (13.4). Sinusoidal vectors (13.5) use wavelengths from \(2\pi\) to \(2\pi \cdot 10^4\); a shift is a rotation (13.7), so dot products depend only on distance (13.8). Learned tables are simple but stop at \(L\).
        • Relative methods put the distance in the score: Shaw adds a clipped vector to keys (13.10); T5 adds one learned scalar per head and log-spaced bucket (13.11–13.12); ALiBi subtracts a fixed slope times the distance (13.13–13.14).
        • Past the training length: learned tables fail, sinusoidal values are unseen, T5 buckets saturate, ALiBi keeps working mostly as a recency window. RoPE, next session, is the method most current LLMs use.

        Key formulas

        \[ \mathrm{Attn}(PX) = P\,\mathrm{Attn}(X), \qquad \mathbf{x}_t = \mathbf{e}_{\mathrm{id}_t} + \mathbf{p}_t \]
        \[ p_{t,2i} = \sin(\omega_i t),\quad p_{t,2i+1} = \cos(\omega_i t),\quad \omega_i = 10000^{-2i/d},\quad \mathbf{p}_t \cdot \mathbf{p}_s = \sum_i \cos\big(\omega_i (t - s)\big) \]
        \[ \text{T5: } s_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j + b_{h,\,\mathrm{bucket}(j-i)}, \qquad \text{ALiBi: } s_{ij} = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}} - m_h (i - j),\ \ m_h = 2^{-8h/H} \]

        References for this session

        1. Vaswani et al. (2017). Attention Is All You Need
        2. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
        3. Press et al. (2021). Train Short, Test Long: Attention with Linear Biases
        4. Vaswani et al. (2017). Attention Is All You Need
        5. Shaw, Uszkoreit, Vaswani (2018). Self-Attention with Relative Position Representations
        6. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
        7. Press et al. (2021). Train Short, Test Long: Attention with Linear Biases
        8. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
        9. Haviv, Ram, Press, Izsak, Levy (2022). Transformer Language Models without Positional Encodings Still Learn Positional Information
        10. Kazemnejad, Padhi, Natesan Ramamurthy, Das, Reddy (2023). The Impact of Positional Encoding on Length Generalization in Transformers
        11. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
        12. Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding