LLM & VLM courseBlog
21Track 2: Attention and transformers · session 6 of 15

The feed-forward network (MLP) inside each block

Attention moves information between tokens. The feed-forward network (FFN) then works on each token alone: it expands the vector to a wider space, applies a non-linear function, and projects it back. This session shows why that simple shape matters, how GELU and SwiGLU replaced ReLU, why many facts seem to live in these layers, and why they hold about two thirds of a block's weights.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 2, 4 and 11 (matrices, gradients, embeddings); sessions 16 to 20 help (attention)

What you will learn

  • What the FFN computes, formula by formula, and why it is applied to each position separately
  • Why the FFN expands to a wider hidden layer, and why a non-linearity is required (without it two layers collapse into one)
  • ReLU, GELU and SiLU (Swish): their formulas, shapes and gradients
  • Gated units: GLU, GEGLU and SwiGLU, and why modern LLMs use a hidden size of about 8d/3 instead of 4d
  • The FFN as a key–value memory: what the evidence says about where facts are stored, and how edits like ROME use it
  • Counting parameters and FLOPs: why the FFN holds about 2/3 of a dense block, and over 80% in Llama 3 with grouped-query attention
  • Where FFNs appear in VLMs and video models: ViT blocks, projectors, prediction heads and Mixture of Experts

Why this matters

Every transformer block has two parts. The attention part moves information between tokens: each token looks at the others and copies what it needs. The second part, the feed-forward network (FFN, also called the MLP, for multi-layer perceptron), does something different. It takes each token’s vector on its own and transforms it. No token looks at any other token inside the FFN.

This second part is easy to overlook, because its formula is short. But it matters a lot, for four reasons:

  • It holds most of the weights. In a classic dense block, the FFN has about 2/3 of the parameters. In Llama 3 8B, with its smaller attention, it has over 80% of each block. When people quantize, prune or shard a model, the FFN is the biggest target.
  • It does most of the compute at normal context lengths. Each weight costs about 2 FLOPs per token, so the FFN also takes most of the arithmetic, until the context becomes very long.
  • It seems to store much of what the model “knows”. Research suggests that FFN layers act like a large key–value memory: some hidden units fire on specific patterns in the input and push the output toward specific next tokens. Methods that edit single facts in a model change FFN weights.
  • Its design changed in modern LLMs. The original ReLU FFN became GELU in GPT-2 and BERT, and then the gated SwiGLU in PaLM, Llama, Mistral and Qwen. This change is behind the strange hidden sizes you see in model configs, such as 11,008 or 14,336.

Interviewers ask about the FFN often: “What is the shape of the MLP in a transformer?”, “Why expand to 4d?”, “What is SwiGLU and why is the hidden size 8d/3?”, “Where are the parameters in a block?”, “Where are facts stored?”. In Session 20 we finished the attention part of the block. Here we build the other half. In session 22 we will glue the two halves together with residual connections and normalization.

What the FFN computes

Expand, activate, project back

Let \(\mathbf{x} \in \R^d\) be the vector of one token after attention (as a row vector, the same convention as session 11). The original transformer (Vaswani et al., 2017) defines the FFN as two linear layers with a non-linear function between them:

\[ \mathrm{FFN}(\mathbf{x}) = \phi(\mathbf{x} W_1 + \mathbf{b}_1)\, W_2 + \mathbf{b}_2, \qquad W_1 \in \R^{d \times d_{\text{ff}}},\ W_2 \in \R^{d_{\text{ff}} \times d} \tag{21.1} \]
Read the symbols
\(d\)
the model width: the length of each token vector (512 in the original transformer, 4,096 in Llama 3 8B)
\(d_{\text{ff}}\)
the hidden size of the FFN, usually larger than \(d\). Classic choice: \(d_{\text{ff}} = 4d\) (2,048 for \(d = 512\)).
\(W_1, \mathbf{b}_1\)
the up projection (also “expand” or “fc1”) and its bias: maps \(d\) numbers to \(d_{\text{ff}}\) numbers
\(\phi\)
the activation function, a simple non-linear function applied to each number separately: ReLU \(\max(0, u)\) in the original paper, GELU or SiLU in later models
\(W_2, \mathbf{b}_2\)
the down projection (also “project back” or “fc2”) and its bias: maps \(d_{\text{ff}}\) numbers back to \(d\)

So the FFN has three steps: expand the vector from \(d\) to \(d_{\text{ff}}\) numbers, apply \(\phi\) to every number, and project back to \(d\) numbers. The output has the same shape as the input, so it can be added back to the token’s vector (the residual connection, next session).

Position-wise: the same weights for every token

For a sequence of \(T\) tokens, stack the vectors as rows of \(X \in \R^{T \times d}\). The FFN is applied to every row with the same weights:

\[ H = \phi(X W_1 + \mathbf{1}\mathbf{b}_1) \in \R^{T \times d_{\text{ff}}}, \qquad Y = H W_2 + \mathbf{1}\mathbf{b}_2 \in \R^{T \times d} \tag{21.2} \]

Here \(\mathbf{1}\) is a column of \(T\) ones, so the bias is added to every row. Row \(t\) of \(Y\) depends only on row \(t\) of \(X\). This is why the paper calls it a position-wise feed-forward network, and also describes it as “two convolutions with kernel size 1”. [Certain] Two practical facts follow:

  • All tokens go through the FFN as one big matrix multiplication of shape \((T \times d)(d \times d_{\text{ff}})\). In training this is a dense, GPU-friendly operation with no masks.
  • If you shuffle the tokens, the FFN outputs are shuffled in the same way. The FFN cannot mix information between positions; only attention can.
Worked example: one hidden unit by hand

Figure 1 uses \(d = 4\) and \(d_{\text{ff}} = 8\), with GELU and no biases. The token “cat” has \(\mathbf{x} = (1.0, -0.5, 0.3, 0.8)\). Column 1 of \(W_1\) is \((0.5, -0.3, 0.8, 0.1)\). The first hidden pre-activation is \(u_1 = 1.0 \cdot 0.5 + (-0.5)(-0.3) + 0.3 \cdot 0.8 + 0.8 \cdot 0.1 = 0.5 + 0.15 + 0.24 + 0.08 = 0.97\). After GELU, \(h_1 = 0.97\,\Phi(0.97) = 0.809\). The other seven hidden units are computed the same way with their own columns. Then the output is \(\mathbf{y} = \sum_{i=1}^{8} h_i\, (W_2)_{i,:}\) = \((0.481, 0.141, -0.388, 0.357)\), and the block adds it to the input: \(\mathbf{x} + \mathbf{y} = (1.481, -0.359, -0.088, 1.157)\).

Figure 1. One token through a tiny FFN, step by step: input \(\mathbf{x}\) (\(d = 4\)), expand to 8 hidden numbers \(\mathbf{u} = \mathbf{x}W_1\), apply GELU, project back with \(W_2\), and add to the input. Purple bars (to the right) are positive numbers, orange bars (to the left) negative. The weights are fixed and the same for both tokens.
How to read Figure 1

Press Next to walk through the four operations of formula (21.1). In step 2 the 4 input numbers become 8 numbers: each one is the dot product of \(\mathbf{x}\) with one column of \(W_1\). In step 3, GELU keeps the positive numbers almost unchanged and squeezes the negative ones close to zero (but not exactly to zero, unlike ReLU). Step 4 mixes the 8 numbers back into 4. Now switch to the token “sat”: the same weights give a different pattern of active hidden units (for “cat”, units 1, 3 and 4 are the strongest; for “sat”, units 2, 5 and 7). Each token uses the same network, but a different part of it.

Why expand, and why a non-linearity

Without \(\phi\), two layers are only one

Why not just use one \(d \times d\) matrix? And why do we need \(\phi\) at all? Remove \(\phi\) (and the biases, for simplicity) from formula (21.1):

\[ (\mathbf{x} W_1)\, W_2 = \mathbf{x}\, (W_1 W_2), \qquad W_1 W_2 \in \R^{d \times d},\ \ \operatorname{rank}(W_1 W_2) \le d \tag{21.3} \]

The two layers collapse into one \(d \times d\) matrix. All the extra width is wasted: a product of linear maps is just another linear map. A whole stack of transformer blocks without non-linear FFNs would still be mostly linear in each token’s features (attention adds a softmax, but softmax only chooses how much of each value to mix; it does not transform the features themselves). The function \(\phi\) is what lets the network compute things like “this AND that”, thresholds, and curved functions.

What the width buys

With \(\phi\), each hidden unit \(i\) computes \(h_i = \phi(\mathbf{x} \cdot \mathbf{w}_i + b_i)\), where \(\mathbf{w}_i\) is column \(i\) of \(W_1\). With ReLU, this is zero on one side of a flat boundary (a hyperplane) and grows linearly on the other side. The down projection adds up \(d_{\text{ff}}\) such pieces. More pieces allow more complex functions: an old result, the universal approximation theorem, says that one hidden layer with enough units can approximate any continuous function on a bounded region as closely as you want. [Certain] The theorem says nothing about how many units you need, but in practice wider is more expressive.

The choice \(d_{\text{ff}} = 4d\) is not a law. It is the value the original transformer used (2,048 for \(d = 512\)), and GPT-2, BERT and ViT kept it. [Certain] It balances the cost of the FFN with the cost of attention, and later scaling studies found that the loss is not very sensitive to this ratio within a broad range. [Likely]

Figure 2 shows both facts in one dimension: a network with one input, one output and a hidden layer of width \(n\), fitted to a curved target. The hidden weights are random and fixed; only the output weights are fitted (by least squares), which is enough to see the effect of width and of \(\phi\).

Figure 2. Width and non-linearity. Target (grey dots): \(y = \sin(2x) + 0.3x\) at 81 points on \([-3, 3]\). Model: \(\hat{y}(x) = a_0 + \sum_{i=1}^{n} a_i\, \phi(w_i x + b_i)\), with \(w_i, b_i\) drawn once from a fixed random seed (each unit’s bend lies somewhere in \([-3, 3]\)) and \(a_0, \dots, a_n\) fitted by least squares with a tiny ridge penalty \(10^{-3}\). The thin lines mark where each ReLU or GELU unit bends.
How to read Figure 2

Choose “no activation”: the fit is a straight line for every width, with mean squared error (MSE) 0.433, because a sum of linear functions is linear (formula 21.3). The target’s own variance is 0.489, so the line explains very little. Now choose ReLU and move the width: with 4 units the error is still 0.458, with 8 units it drops to 0.045, and with 32 units to 0.0008. The curve is built from straight pieces that bend at the thin lines. GELU gives a smooth curve and gets there with fewer units (0.002 at 8 units). Real FFNs have thousands of units in thousands of input dimensions, and all weights are trained, but the principle is the same.

Activation functions: ReLU, GELU and SiLU

The activation \(\phi\) is applied to each hidden number separately. Three choices matter for LLMs.

ReLU (rectified linear unit), \(\mathrm{ReLU}(u) = \max(0, u)\), was used in the original transformer. It is cheap and its gradient is simple: 1 for positive inputs, 0 for negative ones. That second fact is also its weak point: a unit whose input is negative for every token gets no gradient at all and may never recover (a “dead” unit).

GELU (Gaussian error linear unit, Hendrycks and Gimpel, 2016) multiplies the input by the probability that a standard normal variable is smaller than it:

\[ \mathrm{GELU}(u) = u\, \Phi(u), \qquad \Phi(u) = \tfrac{1}{2}\Big(1 + \operatorname{erf}\big(u/\sqrt{2}\big)\Big) \tag{21.4} \]

For large positive \(u\), \(\Phi(u) \approx 1\) and GELU is almost \(u\). For large negative \(u\), \(\Phi(u) \approx 0\) and GELU is almost 0. In between it is smooth, and it dips slightly below zero, to a minimum of about \(-0.170\) at \(u \approx -0.75\). GPT-2 and many other models use a faster approximation with tanh:

\[ \mathrm{GELU}(u) \approx \tfrac{1}{2}\, u \Big(1 + \tanh\!\big(\sqrt{2/\pi}\,(u + 0.044715\, u^3)\big)\Big) \tag{21.5} \]

The two versions differ by at most about 0.00047 (near \(u = -2.7\)). [Certain, checked numerically] In PyTorch they are F.gelu(u) and F.gelu(u, approximate="tanh"). A model should be run with the version it was trained with.

SiLU (sigmoid linear unit), also called Swish, multiplies the input by its own sigmoid:

\[ \mathrm{SiLU}(u) = u\, \sigma(u) = \frac{u}{1 + e^{-u}}, \qquad \mathrm{Swish}_\beta(u) = u\, \sigma(\beta u) \tag{21.6} \]
Read the symbols
\(\Phi(u)\)
the cumulative distribution function of the standard normal distribution: the probability that a \(\mathcal{N}(0,1)\) number is below \(u\)
\(\operatorname{erf}\)
the error function, a standard special function; \(\Phi\) is written with it
\(\sigma(u)\)
the sigmoid \(1/(1 + e^{-u})\), between 0 and 1
\(\beta\)
a sharpness factor. Swish with \(\beta = 1\) is SiLU, the version used in LLMs. As \(\beta \to \infty\), Swish becomes ReLU.

GELU and SiLU look alike: both are a smooth version of ReLU, both are slightly negative for negative inputs (SiLU’s minimum is about \(-0.278\) at \(u \approx -1.28\)), and both have a non-zero gradient almost everywhere, so units do not die as easily. Swish was found by an automatic search over activation functions (Ramachandran et al., 2017). [Certain] Why smooth functions train a bit better is not fully understood; the usual explanations are smoother gradients and no dead units. [Likely]

Worked example: values and gradients at a few points

At \(u = 1\): ReLU gives 1, GELU gives \(1 \cdot \Phi(1) = 0.841\), SiLU gives \(\sigma(1) = 0.731\). At \(u = -1\): ReLU gives 0, GELU gives \(-0.159\), SiLU gives \(-0.269\). The gradients at \(u = -1\) are 0 (ReLU), \(-0.083\) (GELU) and \(0.072\) (SiLU); at \(u = 0\) both smooth functions have gradient exactly 0.5. So a GELU or SiLU unit with a negative input still receives a small gradient and can move.

Figure 3. ReLU (grey, dashed), GELU (purple) and SiLU (orange) on \([-4, 4]\), or their derivatives when the box is ticked. The dots show the value at the chosen input \(u\).
How to read Figure 3

Move \(u\) from left to right. For \(u \gt 3\) all three curves are almost the same line \(y = u\). The difference is near zero and below it: ReLU is flat at 0, GELU dips to about \(-0.17\) near \(u = -0.75\), and SiLU dips deeper, to about \(-0.28\) near \(u = -1.28\). Tick the gradient box: ReLU’s gradient jumps from 0 to 1 at \(u = 0\), while the smooth functions change gradually and even overshoot 1 a little for positive \(u\) (GELU’s gradient at \(u = 1\) is 1.083). A negative gradient (GELU for every \(u\) below about \(-0.75\), SiLU below about \(-1.28\)) means that making the input more negative raises the output toward zero.

Gated units: GLU and SwiGLU

The gate

A gated linear unit (GLU) computes two projections of the input and multiplies them element by element. One of them goes through a sigmoid and acts as a gate: a number between 0 and 1 that decides how much of the other projection passes. Dauphin et al. (2017) introduced it for convolutional language models: [Certain]

\[ \mathrm{GLU}(\mathbf{x}) = (\mathbf{x} W + \mathbf{b}) \odot \sigma(\mathbf{x} V + \mathbf{c}) \tag{21.7} \]

Here \(\odot\) means element-wise multiplication. Shazeer (2020) put this idea into the transformer FFN and tried different functions in place of the sigmoid: ReLU (ReGLU), GELU (GEGLU), Swish (SwiGLU), and none at all (Bilinear). The SwiGLU FFN, without biases, is:

\[ \mathrm{FFN}_{\text{SwiGLU}}(\mathbf{x}) = \Big(\mathrm{SiLU}(\mathbf{x} W_{\text{gate}}) \odot \mathbf{x} W_{\text{up}}\Big)\, W_{\text{down}} \tag{21.8} \]
Read the symbols
\(W_{\text{gate}} \in \R^{d \times d_{\text{ff}}}\)
the gate projection (called gate_proj or w1 in Llama code). Its output goes through SiLU.
\(W_{\text{up}} \in \R^{d \times d_{\text{ff}}}\)
the up projection (up_proj or w3). Its output is not activated; it is multiplied by the gate.
\(W_{\text{down}} \in \R^{d_{\text{ff}} \times d}\)
the down projection (down_proj or w2), the same role as \(W_2\) in (21.1)
\(\odot\)
element-wise product of two vectors of length \(d_{\text{ff}}\)

Why should a product help? In the plain FFN, hidden unit \(i\) is \(\phi(\mathbf{x} \cdot \mathbf{w}_i)\): a function of one direction of the input. In SwiGLU it is \(\mathrm{SiLU}(\mathbf{x} \cdot \mathbf{g}_i)\,(\mathbf{x} \cdot \mathbf{u}_i)\): one direction decides whether the unit is open, another direction decides what value passes. The product of two linear functions is a quadratic function of \(\mathbf{x}\), so each unit can express interactions between features, such as “feature A matters only when feature B is present”. This is an intuition, not a proof. [Likely]

The 8d/3 rule

A gated FFN has three weight matrices instead of two. To compare fairly at equal parameters and FLOPs, Shazeer reduced the hidden size by a factor of 2/3. [Certain] With \(d_{\text{ff}} = 4d\) for the plain FFN:

\[ 3\, d\, d_{\text{ff}}^{\text{gated}} = 2\, d\, (4d) \quad\Longrightarrow\quad d_{\text{ff}}^{\text{gated}} = \tfrac{2}{3}\cdot 4d = \tfrac{8}{3}\, d \tag{21.9} \]

In the T5 base model of the paper (\(d = 768\)), the plain FFN used \(d_{\text{ff}} = 3{,}072\) and the gated ones used 2,048: \(2 \times 768 \times 3{,}072 = 3 \times 768 \times 2{,}048 = 4{,}718{,}592\) weights in both cases. [Certain] The results on the C4 dataset, as held-out log-perplexity (lower is better):

FFN variantHidden sizeAfter 65,536 stepsAfter 524,288 steps
ReLU (baseline)3,0721.9971.677
GELU3,0721.9831.679
Swish3,0721.9941.683
GLU (sigmoid gate)2,0481.9821.663
Bilinear (no function)2,0481.9601.648
ReGLU2,0481.9531.645
GEGLU2,0481.9421.633
SwiGLU2,0481.9441.636

Numbers from Table 1 of Shazeer (2020). [Certain] Three things to notice. The plain activations (ReLU, GELU, Swish) are within about 0.015 of each other. All gated versions beat all plain ones at equal size. And even the Bilinear version, with no non-linear function except the product itself, beats the plain ones. The gains also survive fine-tuning on GLUE and SuperGLUE in the paper. The paper famously ends: “We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.” [Certain]

PaLM (2022) adopted SwiGLU, noting that it needs three matrix multiplications instead of two, and also removed all biases, which it found gave more stable training for large models. [Certain] Llama, Mistral and Qwen use SwiGLU without biases; Gemma uses GEGLU. [Certain]

Worked example: where 11,008 and 14,336 come from

Llama code computes the hidden size from \(4d\): take 2/3 of it, optionally multiply by a factor, and round up to a multiple of some number \(m\). [Certain for the procedure; the per-model factors below are from the released configurations. Likely]

Llama 2 7B (\(d = 4{,}096\), \(m = 256\), no factor): \(\tfrac{2}{3} \times 16{,}384 = 10{,}922.7\), truncated to 10,922; the next multiple of 256 is \(43 \times 256 = 11{,}008\).

Llama 3 8B (\(d = 4{,}096\), factor 1.3, \(m = 1{,}024\)): \(10{,}922 \times 1.3 = 14{,}198.6\), truncated to 14,198; the next multiple of 1,024 is \(14 \times 1{,}024 = 14{,}336\). This matches intermediate_size: 14336 in the model’s config.json. [Certain]

Llama 3 70B (\(d = 8{,}192\), factor 1.3, \(m = 4{,}096\)): \(21{,}845 \times 1.3 = 28{,}398.5\), so 28,672 \(= 7 \times 4{,}096\). Rounding to a multiple of a large power of two keeps the matrix shapes efficient on GPUs and easy to split across devices.

The FFN as a key–value memory

Rewriting the formula

Write the columns of \(W_1\) as vectors \(\mathbf{k}_1, \dots, \mathbf{k}_{d_{\text{ff}}}\) and the rows of \(W_2\) as \(\mathbf{v}_1, \dots, \mathbf{v}_{d_{\text{ff}}}\). Then formula (21.1), without biases, is a sum:

\[ \mathrm{FFN}(\mathbf{x}) = \sum_{i=1}^{d_{\text{ff}}} \underbrace{\phi(\mathbf{x} \cdot \mathbf{k}_i)}_{m_i:\ \text{how well } \mathbf{x} \text{ matches key } i}\ \mathbf{v}_i \tag{21.10} \]
Read the symbols
\(\mathbf{k}_i \in \R^d\)
the key of memory \(i\): column \(i\) of \(W_1\). A pattern the unit responds to.
\(m_i\)
the memory coefficient: the activation of hidden unit \(i\). Large when the input points in the direction of \(\mathbf{k}_i\).
\(\mathbf{v}_i \in \R^d\)
the value of memory \(i\): row \(i\) of \(W_2\). The vector that gets added to the residual stream when the memory fires.

This looks very much like attention (session 16): compare a query with keys, then sum values weighted by the match. There are two differences. The keys and values here are fixed parameters learned in training, not computed from the context. And the weights \(m_i\) come from \(\phi\), not from a softmax, so they do not need to sum to 1 and many can be zero. Geva et al. (2021) called this view “feed-forward layers are key–value memories”. [Certain]

What the evidence says

  • Keys detect patterns. Geva et al. studied a 16-layer language model with \(d = 1{,}024\) and \(d_{\text{ff}} = 4{,}096\) trained on WikiText-103. For sampled hidden units, they collected the training prefixes that activate them most. Human annotators found recognizable patterns for every sampled key. In lower layers the patterns were mostly shallow (for example, prefixes ending with the same word); in upper layers they were more semantic (prefixes about the same topic, with different words). [Certain]
  • Values predict next tokens. If you multiply a value vector \(\mathbf{v}_i\) by the output embedding matrix, you get a score for every token in the vocabulary. In the upper layers, the top token of a value often agreed with the token that actually followed its key’s trigger prefixes, far more often than chance. [Certain] A later paper by Geva et al. (2022) showed that FFN updates can be read as promoting human-readable concepts in this vocabulary space. [Certain]
  • Outputs are mixtures. A typical input activates hundreds of memories in a layer, and in most cases the layer’s top prediction is not the top prediction of any single memory: the output is a composition of many memories. [Certain]
  • Facts can be located and edited. Dai et al. (2022) found “knowledge neurons” in BERT’s FFN layers whose activation is linked to specific facts; suppressing or amplifying them changed the probability of the correct answer. [Certain] Meng et al. (2022, ROME) traced factual recall in GPT models to the FFN layers in the middle of the network, at the last token of the subject (“Eiffel Tower” in “The Eiffel Tower is in”). They edited one FFN matrix with a rank-one update to change a single fact. [Certain]

A careful reading of this evidence matters, because the key–value view is easy to oversell:

  • Units are often polysemantic. One hidden unit can respond to several unrelated patterns, because the model stores more features than it has units (this is called superposition). Work on sparse autoencoders (Bricken et al., 2023) tries to split activations into cleaner, “monosemantic” features. [Certain]
  • Facts are spread out. A fact is usually supported by many units in several layers, plus attention that moves the subject’s information to the right position. Editing one place can fail to change related questions, or can damage other facts. [Likely]
  • “Where facts are stored” is a simplification. The evidence shows that FFN layers are a main place where the model writes fact-related information into the stream, not that each fact has a single address. [Likely]

Figure 4 is a toy version of formula (21.10), with 6 hand-made memories and a vocabulary of 8 tokens. To keep it readable, each value is shown already multiplied by the output embedding, as a change in the logits of the 8 tokens.

Figure 4. A toy FFN as a key–value memory, formula (21.10) with ReLU and a bias. The input has 5 features (capital, France, Italy, Germany, sky). Each card is one hidden unit: its key, its bias, its value, and its activation \(m_i\) for the chosen prompt (highlighted when it fires). Click a card to switch that unit off (set \(m_i = 0\)). The bars show the softmax over the 8 tokens of the base logits plus \(\sum_i m_i \mathbf{v}_i\).
How to read Figure 4

For “The capital of France is”, three memories fire: memory 1 (capital AND France, which needs both features to pass its bias of \(-1\)), memory 5 (any capital) and memory 6 (anything about France). Paris gets 98.1%. Switch off memory 1: Paris drops to 48.1% but is still first, because memories 5 and 6 still help. Switch off memory 6 too: the four capitals tie at 17.1% each. The fact was stored in more than one place. Now reset and tick the edit: Rome wins the France prompt with 86.8%, while the Italy prompt does not change at all (memory 1 does not fire for it). But Paris keeps 7.1%, pushed by memory 6: an edit in one place leaves related memories behind. This is the same trade-off that real editing methods face.

Counting parameters and FLOPs

Why about 2/3

Count the weight matrices of one block, ignoring biases and normalization (they are tiny). Standard multi-head attention (session 19) has four \(d \times d\) matrices: \(W_Q, W_K, W_V, W_O\). The plain FFN with \(d_{\text{ff}} = 4d\) has two matrices of \(d \times 4d\):

\[ N_{\text{attn}} = 4d^2, \qquad N_{\text{FFN}} = 2\, d\, d_{\text{ff}} = 8d^2, \qquad \frac{N_{\text{FFN}}}{N_{\text{attn}} + N_{\text{FFN}}} = \frac{8}{12} = \frac{2}{3} \tag{21.11} \]

A SwiGLU FFN with \(d_{\text{ff}} = 8d/3\) also has \(3 \cdot d \cdot 8d/3 = 8d^2\) weights, so the share stays 2/3. Two things move it up in modern models. First, grouped-query attention (GQA, session 26) shares keys and values across heads, so \(W_K\) and \(W_V\) become smaller. Second, many models choose \(d_{\text{ff}}\) above \(8d/3\). In general, with \(h\) query heads, \(h_{kv}\) key/value heads and head size \(d_h\):

\[ N_{\text{attn}} = 2\, d\, h\, d_h + 2\, d\, h_{kv}\, d_h, \qquad N_{\text{FFN}} = c\, d\, d_{\text{ff}}, \quad c = 2 \text{ (plain) or } 3 \text{ (gated)} \tag{21.12} \]
Model\(d\)\(d_{\text{ff}}\)FFN typeHeads (query / KV)Attention per blockFFN per blockFFN share
Transformer base (2017)5122,048ReLU8 / 81.05 M2.10 M66.7%
GPT-2 small7683,072GELU12 / 122.36 M4.72 M66.7%
Llama 2 7B4,09611,008SwiGLU32 / 3267.1 M135.3 M66.8%
Llama 3 8B4,09614,336SwiGLU32 / 841.9 M176.2 M80.8%
Llama 3 70B8,19228,672SwiGLU64 / 8151.0 M704.6 M82.4%

Sizes from the papers and public configurations; weights only, no biases or norms. [Certain] The whole-model share is lower, because the embedding tables (session 11) are outside the blocks: the 32 FFNs of Llama 3 8B hold 5.64 B of its 8.03 B parameters (70.2%), and the 32 FFNs of Llama 2 7B hold 4.33 B of 6.74 B (64.2%).

FLOPs per token

A matrix multiplication costs 2 FLOPs (one multiply, one add) per weight per token. So the FFN costs \(2 N_{\text{FFN}}\) FLOPs per token per layer, no matter how long the context is. Attention has two parts: the four projections cost \(2 N_{\text{attn}}\), and the score and value products \(QK^\top\) and \(AV\) cost about \(4 T d\) for a token that attends to \(T\) others (session 25 does the full count):

\[ F_{\text{FFN}} = 2 N_{\text{FFN}}, \qquad F_{\text{attn}} \approx 2 N_{\text{attn}} + 4\, T\, d \tag{21.13} \]

The FFN stays the biggest cost until \(4Td\) catches up. For GPT-2 small, \(4Td = 16d^2\) gives \(T = 4d = 3{,}072\). For Llama 3 8B, \(4Td = 6\, d\, d_{\text{ff}}\) gives \(T = 1.5\, d_{\text{ff}} = 21{,}504\) tokens. Below these lengths, making the FFN cheaper (quantization, sparsity, MoE) is where the compute savings are; above them, attention becomes the main cost. This is the theme of session 93.

Figure 5. Where the weights and the FLOPs of one block go, from formulas (21.11) to (21.13). A preset loads a real model’s \(d\), \(d_{\text{ff}}\), FFN type and head counts. Change \(d_{\text{ff}}\) or the FFN type to see what would happen; the context slider changes only the FLOPs bar. Weights only, no biases or norms.
How to read Figure 5

The top bar splits the block’s parameters, the bottom bar splits the FLOPs for one token that attends to \(T\) tokens. Load GPT-2 small: the FFN has 66.7% of the weights. Load Llama 2 7B: still about 2/3 with SwiGLU at 11,008. Load Llama 3 8B: 80.8%, because grouped-query attention made attention smaller. Now move the context slider: at \(T = 4{,}096\) the FFN still does 70% of the FLOPs, but at \(T = 131{,}072\) the attention scores dominate and the FFN falls to 14%. Untick “Gated” on Llama 3 8B: the FFN loses a third of its weights.

Beyond text: FFNs in vision and video models

The same expand–activate–project block appears all over vision-language models:

  • Vision Transformer blocks. ViT (session 164) uses exactly the GPT-style block: attention, then an MLP with GELU and a 4× hidden size (ViT-Base: \(d = 768\), hidden size 3,072). [Certain] Video transformers such as TimeSformer and ViViT keep this FFN and change only the attention pattern over space and time. [Likely] So for a video encoder too, about 2/3 of each block’s weights are in the FFN.
  • Projectors. A VLM must map vision features to the LLM width. LLaVA-1.5 uses a two-layer MLP for this (LLaVA used a single linear layer). [Certain] That projector is a small FFN with a different output size: vision width in, LLM width out (session 178).
  • Prediction heads. Detection and temporal-grounding models put small MLPs on top of the transformer to predict boxes or time spans. Moment-DETR, for key-moment detection in video, predicts the start and end of a moment with such heads. [Likely]
  • Mixture of Experts in VLMs. Several large VLMs use MoE language backbones, so the FFN of the language model becomes many experts while the vision encoder stays dense. [Likely]

Where this is used

ApplicationWhat the FFN does thereLink
Every decoder LLM (Llama, Mistral, Qwen, Gemma)SwiGLU or GEGLU FFN in every block, formula (21.8); about 2/3 to over 80% of each block’s weightssession 23
Mixture of Experts (Mixtral, DeepSeek-V3)Each FFN is replaced by many expert FFNs and a router; more parameters, same compute per tokensession 29
Quantization, pruning and sparsityFFN matrices are the largest weights, so they get the most attention in compression. ReLU-style activations are often zero, which some inference systems exploit to skip work.session 94
Knowledge editing and interpretabilityFacts are traced to mid-layer FFNs and edited with low-rank weight updates (ROME); units and their values are inspected as key–value memories (21.10)session 157
Vision encoders and video action recognitionViT and video transformer blocks use a GELU MLP with 4× expansion after each attention layer; the clip-level features that classify actions come out of these blockssession 203
VLM projectorsA 2-layer MLP maps vision-encoder outputs to the LLM width (LLaVA-1.5)session 178
Key-moment detection and temporal groundingSmall MLP heads on transformer outputs predict moment start and end times and a confidence scoresession 207
Text-to-video retrieval and Video-RAGThe text and video encoders that produce the embeddings are transformer stacks; their FFNs are most of the encoder weights and the main target when the encoders are compressed for large-scale indexingsession 208

Pseudo-code

Algorithm 1: classic FFN forward, formula (21.2)
  1. input: \(X \in \R^{T \times d}\), weights \(W_1 \in \R^{d \times d_{\text{ff}}}\), \(\mathbf{b}_1\), \(W_2 \in \R^{d_{\text{ff}} \times d}\), \(\mathbf{b}_2\), activation \(\phi\)
  2. \(U \leftarrow X W_1 + \mathbf{b}_1\) // expand: \((T, d)\) to \((T, d_{\text{ff}})\), one big matmul for all tokens
  3. \(H \leftarrow \phi(U)\) // element-wise: ReLU, GELU (21.4) or SiLU (21.6)
  4. \(Y \leftarrow H W_2 + \mathbf{b}_2\) // project back: \((T, d_{\text{ff}})\) to \((T, d)\)
  5. return \(X + Y\) // the block adds the result to the residual stream
Algorithm 2: SwiGLU FFN with Llama-style sizing, formulas (21.8) and (21.9)
  1. sizing: \(h \leftarrow \lfloor \tfrac{2}{3} \cdot 4d \rfloor\); if a factor \(f\) is given then \(h \leftarrow \lfloor f\, h \rfloor\)
  2. \(d_{\text{ff}} \leftarrow m \cdot \lceil h / m \rceil\) // round up to a multiple of \(m\): 11,008 or 14,336
  3. input: \(X \in \R^{T \times d}\), \(W_{\text{gate}}, W_{\text{up}} \in \R^{d \times d_{\text{ff}}}\), \(W_{\text{down}} \in \R^{d_{\text{ff}} \times d}\)
  4. \(G \leftarrow \mathrm{SiLU}(X W_{\text{gate}})\) // the gate, \((T, d_{\text{ff}})\)
  5. \(U \leftarrow X W_{\text{up}}\) // the content, \((T, d_{\text{ff}})\); in practice fused with the gate into one matmul
  6. \(Y \leftarrow (G \odot U)\, W_{\text{down}}\) // element-wise product, then project back
  7. return \(X + Y\)
Algorithm 3: read an FFN as key–value memories, formula (21.10)
  1. input: one token vector \(\mathbf{x}\), FFN weights, output embedding \(E \in \R^{|V| \times d}\), number \(k\)
  2. \(\mathbf{m} \leftarrow \phi(\mathbf{x} W_1 + \mathbf{b}_1)\) // memory coefficients, one per hidden unit
  3. \(I \leftarrow\) indices of the \(k\) largest entries of \(\mathbf{m}\) // the memories that fire most
  4. for \(i \in I\) do
  5. \(\mathbf{s} \leftarrow E\, \mathbf{v}_i\) // value \(i\) in vocabulary space, \(\mathbf{v}_i\) = row \(i\) of \(W_2\)
  6. report \(m_i\) and the tokens with the highest scores in \(\mathbf{s}\) // what this memory promotes

The same ideas in PyTorch. Each block checks one claim from this session (outputs from PyTorch 2.8 on CPU):

import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)

# 1) the classic FFN of formula (21.1): expand 4x, GELU, project back
class FFN(nn.Module):
    def __init__(self, d, d_ff):
        super().__init__()
        self.up = nn.Linear(d, d_ff)       # W1, b1
        self.down = nn.Linear(d_ff, d)     # W2, b2
    def forward(self, x):                  # x: (B, T, d)
        return self.down(F.gelu(self.up(x)))

# 2) the SwiGLU FFN of formula (21.8), as in Llama (no biases)
class SwiGLU(nn.Module):
    def __init__(self, d, d_ff):
        super().__init__()
        self.gate = nn.Linear(d, d_ff, bias=False)   # W_gate
        self.up = nn.Linear(d, d_ff, bias=False)     # W_up
        self.down = nn.Linear(d_ff, d, bias=False)   # W_down
    def forward(self, x):
        return self.down(F.silu(self.gate(x)) * self.up(x))

count = lambda m: sum(p.numel() for p in m.parameters())
print(count(FFN(768, 3072)))                         # 4722432  (GPT-2 small, with biases)
with torch.device("meta"):                           # no memory is allocated
    print(count(SwiGLU(4096, 14336)))                # 176160768 (Llama 3 8B, one layer)

# 3) position-wise: permuting the tokens permutes the outputs
ffn = FFN(16, 64)
x = torch.randn(2, 5, 16)
perm = torch.tensor([3, 0, 4, 1, 2])
print(torch.allclose(ffn(x)[:, perm], ffn(x[:, perm]), atol=1e-6))   # True

# 4) without the activation, two layers collapse into one d x d matrix (21.3)
W1, W2 = torch.randn(16, 64), torch.randn(64, 16)
print(torch.allclose((x @ W1) @ W2, x @ (W1 @ W2), atol=1e-4))       # True
print(torch.linalg.matrix_rank(W1 @ W2).item())                      # 16, never more than d

# 5) key-value view (21.10): the output is a sum of value rows weighted by key matches
K = ffn.up.weight            # (d_ff, d): row i is key k_i
Vt = ffn.down.weight.T       # (d_ff, d): row i is value v_i
t = x[0, 0]
m = F.gelu(K @ t + ffn.up.bias)                       # one coefficient per memory
y = (m[:, None] * Vt).sum(0) + ffn.down.bias
print(torch.allclose(y, ffn(t), atol=1e-6))           # True

# 6) GELU: exact vs tanh approximation (21.4), (21.5)
z = torch.linspace(-6, 6, 120001)
print(F.gelu(torch.tensor(1.0)).item())                                    # 0.8413...
print((F.gelu(z) - F.gelu(z, approximate="tanh")).abs().max().item())      # about 4.7e-4

# 7) Llama's FFN width rule (21.9): 2/3 of 4d, times a factor, rounded up
def llama_ffn_dim(d, multiple_of, mult=None):
    h = int(2 * 4 * d / 3)
    if mult is not None:
        h = int(mult * h)
    return multiple_of * ((h + multiple_of - 1) // multiple_of)
print(llama_ffn_dim(4096, 256), llama_ffn_dim(4096, 1024, 1.3), llama_ffn_dim(8192, 4096, 1.3))
# 11008 14336 28672

Common mistakes

  • Forgetting the third matrix. A SwiGLU FFN with hidden size \(d_{\text{ff}}\) has \(3\,d\,d_{\text{ff}}\) weights, not \(2\,d\,d_{\text{ff}}\). Counting Llama 3 8B’s FFN as two matrices gives 117 M per layer instead of 176 M.
  • Assuming \(d_{\text{ff}} = 4d\) everywhere. Gated models use about \(8d/3\) and then round or scale it (11,008, 14,336, 28,672). Read intermediate_size in the config instead of guessing.
  • Mixing up gate and up projections. In \(\mathrm{SiLU}(\mathbf{x} W_{\text{gate}}) \odot \mathbf{x} W_{\text{up}}\), only one branch is activated. Swapping w1 and w3 when loading weights gives a model that runs but produces nonsense, because the activation is applied to the wrong branch.
  • Using the wrong GELU. The exact (erf) and tanh versions differ by up to about 0.0005 per unit. That is small, but over many layers it can change outputs; when you port a model, use the version it was trained with.
  • Thinking the FFN mixes tokens. It does not: it is applied to each position independently (21.2). Any information that a token needs from other tokens must arrive through attention first.
  • Reading one neuron as one concept. Hidden units are often polysemantic, and facts are spread over many units and layers. “Neuron 3,217 stores the capital of France” is almost never the full story.
  • Ignoring the FFN in memory planning. The hidden activation has shape \((B, T, d_{\text{ff}})\), wider than the residual stream; a gated FFN keeps two of them (gate and up) for the backward pass. This is a large part of activation memory in training (session 38).

Interview questions

What does the FFN in a transformer block compute, and how is it different from attention?basic
It computes \(\phi(\mathbf{x} W_1 + \mathbf{b}_1) W_2 + \mathbf{b}_2\) for each token vector \(\mathbf{x}\) separately (21.1): expand from \(d\) to \(d_{\text{ff}}\) (classically \(4d\)), apply an element-wise non-linearity, project back to \(d\), then add to the residual stream. The same weights are used at every position, so it is position-wise and never mixes tokens. Attention is the opposite: it mixes information across positions using input-dependent weights. Roughly, attention gathers information, the FFN processes it.
Why do we need the non-linearity, and why make the hidden layer wider than d?basic
Without \(\phi\), \((\mathbf{x}W_1)W_2 = \mathbf{x}(W_1W_2)\): the two layers collapse into one \(d \times d\) linear map (21.3), and the width is wasted. With \(\phi\), each hidden unit is a non-linear feature of the input (for ReLU: zero on one side of a hyperplane, linear on the other), and the output is a weighted sum of these features. More units means more features and more complex functions (universal approximation). The 4× width is a convention from the original transformer that balances FFN and attention cost.
Compare ReLU, GELU and SiLU. Why did LLMs move away from ReLU?core
ReLU \(= \max(0,u)\): cheap, exactly sparse, but zero gradient for negative inputs, so units can die. GELU \(= u\,\Phi(u)\) (21.4) and SiLU \(= u\,\sigma(u)\) (21.6) are smooth versions: close to \(u\) for large positive inputs, close to 0 for large negative ones, slightly negative in between (minimum about \(-0.17\) for GELU, \(-0.28\) for SiLU), and with a non-zero gradient almost everywhere. They train slightly better in practice; in Shazeer’s T5 study, plain GELU and Swish were within about 0.015 log-perplexity of ReLU, and the real gain came from gating. GELU is often computed with the tanh approximation (21.5).
What is SwiGLU, and why is Llama’s FFN hidden size 11,008 instead of 16,384?core
SwiGLU is \((\mathrm{SiLU}(\mathbf{x}W_{\text{gate}}) \odot \mathbf{x}W_{\text{up}})W_{\text{down}}\) (21.8): one branch is a smooth gate, the other carries content, multiplied element-wise. It has three matrices, so to match a plain \(4d\) FFN’s parameters and FLOPs the hidden size is reduced to \(\tfrac{2}{3}\cdot 4d = 8d/3\) (21.9). For \(d = 4{,}096\) that is 10,922.7, rounded up to a multiple of 256: 11,008. Llama 3 8B also multiplies by 1.3 and rounds to a multiple of 1,024: 14,336. Shazeer found gated variants beat plain ones at equal size (SwiGLU 1.944 vs ReLU 1.997 log-perplexity after 65k steps).
Count the parameters of one transformer block. Why is the FFN about 2/3? What changes in Llama 3 8B?core
Multi-head attention has \(W_Q, W_K, W_V, W_O\): \(4d^2\). A plain FFN with \(d_{\text{ff}} = 4d\) has \(2 \cdot d \cdot 4d = 8d^2\). So the FFN is \(8/12 = 2/3\) (21.11); a SwiGLU FFN at \(8d/3\) also has \(8d^2\). Llama 3 8B uses grouped-query attention with 8 KV heads of size 128: attention is \(2 \cdot 4096^2 + 2 \cdot 4096 \cdot 1024 = 41.9\) M; the FFN is \(3 \cdot 4096 \cdot 14336 = 176.2\) M, so 80.8% of the block. Over the whole model (with embeddings) the FFNs are 5.64 B of 8.03 B, about 70%.
At what context length does attention cost more FLOPs per token than the FFN?advanced
Per token and layer, the FFN costs \(2N_{\text{FFN}}\) FLOPs, independent of context. The attention score and value products cost about \(4Td\) for a token attending to \(T\) tokens (plus \(2N_{\text{attn}}\) for the projections) (21.13). Setting \(4Td = 2N_{\text{FFN}}\): for a plain FFN with \(8d^2\) weights, \(T = 4d\) (3,072 for GPT-2 small); for Llama 3 8B, \(T = 6\,d\,d_{\text{ff}}/(4d) = 1.5\,d_{\text{ff}} = 21{,}504\). Below that the FFN dominates compute, so FFN optimizations (quantization, sparsity, MoE) pay off most; above it, attention optimizations matter more. Note that decoding is usually limited by memory bandwidth, not FLOPs, which changes the picture again.
Explain the view of the FFN as a key–value memory. What is the evidence, and what are its limits?advanced
Write \(W_1\)’s columns as keys \(\mathbf{k}_i\) and \(W_2\)’s rows as values \(\mathbf{v}_i\): \(\mathrm{FFN}(\mathbf{x}) = \sum_i \phi(\mathbf{x}\cdot\mathbf{k}_i)\,\mathbf{v}_i\) (21.10), like attention with learned, fixed keys and values and no softmax. Geva et al. (2021) found that keys fire on recognizable input patterns (shallow in lower layers, semantic in upper layers) and that values, projected through the output embedding, promote tokens that tend to follow those patterns, mostly in upper layers; layer outputs are compositions of many memories. Knowledge-neuron and ROME studies localize and edit facts in FFN layers. Limits: units are polysemantic (superposition), facts are distributed across units and layers, and edits can be incomplete or have side effects.
How does ROME edit a fact, and why does it target the FFN?advanced
ROME first uses causal tracing: it corrupts the subject tokens, then restores hidden states one at a time to see which restore the correct answer. In GPT models, mid-layer FFN outputs at the last subject token had a large effect. It then treats that FFN’s down projection as a linear key–value store (the key is the hidden activation for the subject, the value is the vector written to the stream) and applies a rank-one update so the subject’s key maps to a new value that produces the new fact. It targets the FFN because that is where the tracing points and because the key–value view makes a closed-form, local update possible.
Why do modern LLMs remove the biases from the FFN?basic
PaLM removed all biases from dense layers and layer norms and reported more stable training for large models; Llama and most later models follow this. The biases are a tiny share of the parameters, so the model loses almost nothing in capacity, and the code becomes simpler (and fits cleanly with RMSNorm, which has no bias either). The original transformer and GPT-2 have biases; models derived from them, such as many ViTs, still do.
Why is the FFN the part that Mixture of Experts replaces?core
Because it holds most of the parameters and FLOPs of a block and it works on each token independently. Independence means each token can be sent to a different FFN (expert) without breaking anything, and a router can pick the top-k experts per token. Parameters then grow with the number of experts while compute per token grows only with k. Mixtral 8x7B has 8 expert FFNs per layer and uses 2 per token: about 47 B parameters in total, about 13 B used per token. Attention, which must see all tokens together, is usually kept dense.
Where do MLPs appear in a vision-language model?core
In every block of the vision encoder (ViT: GELU MLP with 4× expansion, for example 768 to 3,072 in ViT-Base), in every block of the language model (usually SwiGLU), in the projector between them (LLaVA-1.5 uses a two-layer MLP to map vision features to the LLM width), and often in task heads (box or time-span prediction in detection and moment-retrieval models). In an MoE VLM, the language model’s FFNs become experts. Video encoders reuse the ViT block, so their FFNs also hold most of the weights.

Exercises

  1. A model has \(d = 1{,}024\) and a plain GELU FFN with \(d_{\text{ff}} = 4d\), plus standard multi-head attention. How many weights are in the FFN and in attention per block (no biases)? What is the FFN share?
  2. The same model switches to SwiGLU at equal parameters. What hidden size should it use before rounding? What is it after rounding up to a multiple of 256?
  3. Compute GELU and SiLU at \(u = 2\). Which is closer to ReLU?
  4. In Figure 4, for the prompt “The capital of Germany is”, which memories fire? What happens if you switch off memory 3 alone?
  5. A model has \(d = 2{,}048\), 16 query heads and 4 KV heads of size 128, and a SwiGLU FFN with \(d_{\text{ff}} = 8{,}192\). Compute the attention and FFN weights per block and the FFN share.
  6. Show that a SwiGLU FFN with \(d_{\text{ff}} = 4d\) (no reduction) has 3/4 of the block’s weights when attention is \(4d^2\).
  7. For GPT-2 small (\(d = 768\)), at what context length \(T\) does the attention score cost \(4Td\) equal the FFN cost per token?
Answerscheck
1. FFN: \(2 \times 1{,}024 \times 4{,}096 = 8{,}388{,}608\). Attention: \(4 \times 1{,}024^2 = 4{,}194{,}304\). Share \(= 8/12 = 66.7\%\). 2. \(\tfrac{8}{3} \times 1{,}024 = 2{,}730.7\); \(\lfloor 2{,}730.7 \rfloor = 2{,}730\), rounded up to a multiple of 256 is \(11 \times 256 = 2{,}816\). 3. \(\mathrm{GELU}(2) = 2\,\Phi(2) = 1.954\); \(\mathrm{SiLU}(2) = 2\,\sigma(2) = 1.762\). GELU is closer to ReLU’s value 2. 4. Memory 3 (capital AND Germany) and memory 5 (any capital); Berlin gets 91.9%. With memory 3 off, only memory 5 is left, which helps all four capitals equally: they tie at 17.1% each. 5. Attention: \(2 \times 2{,}048 \times 2{,}048 + 2 \times 2{,}048 \times 512 = 8{,}388{,}608 + 2{,}097{,}152 = 10{,}485{,}760\). FFN: \(3 \times 2{,}048 \times 8{,}192 = 50{,}331{,}648\). Share \(= 50.3/60.8 = 82.8\%\). 6. FFN \(= 3 \cdot d \cdot 4d = 12d^2\); share \(= 12/(12+4) = 3/4\). 7. FFN FLOPs \(= 2 \times 8d^2 = 16d^2\); \(4Td = 16d^2\) gives \(T = 4d = 3{,}072\).

Key takeaways

  • The FFN is the second half of every transformer block: expand from \(d\) to \(d_{\text{ff}}\), apply a non-linearity, project back (21.1). It acts on each position separately with the same weights (21.2); only attention mixes tokens.
  • The non-linearity is essential: without it, the two layers collapse into one \(d \times d\) map (21.3). The width \(d_{\text{ff}}\) sets how many non-linear features each layer can compute.
  • ReLU gave way to smooth GELU (21.4, 21.5) and SiLU (21.6), and then to gated units (21.7). SwiGLU (21.8) with \(d_{\text{ff}} \approx 8d/3\) (21.9) is the standard in Llama, Mistral and Qwen; this rule explains sizes like 11,008 and 14,336.
  • The FFN can be read as a key–value memory (21.10): keys detect input patterns, values push toward output tokens. This is a main place where facts are written into the stream, and where editing methods like ROME act, but facts are spread out and units are polysemantic.
  • A dense block puts about 2/3 of its weights in the FFN (21.11); with grouped-query attention it is over 80% (21.12). The FFN also does most of the FLOPs until the context reaches thousands to tens of thousands of tokens (21.13), which is why MoE, quantization and pruning target it first.

Key formulas

\[ \mathrm{FFN}(\mathbf{x}) = \phi(\mathbf{x}W_1 + \mathbf{b}_1)W_2 + \mathbf{b}_2, \qquad \mathrm{FFN}_{\text{SwiGLU}}(\mathbf{x}) = \big(\mathrm{SiLU}(\mathbf{x}W_{\text{gate}}) \odot \mathbf{x}W_{\text{up}}\big)W_{\text{down}} \]
\[ \mathrm{GELU}(u) = u\,\Phi(u), \qquad \mathrm{SiLU}(u) = u\,\sigma(u), \qquad d_{\text{ff}}^{\text{gated}} = \tfrac{8}{3}d \]
\[ \mathrm{FFN}(\mathbf{x}) = \sum_i \phi(\mathbf{x}\cdot\mathbf{k}_i)\,\mathbf{v}_i, \qquad \frac{N_{\text{FFN}}}{N_{\text{attn}} + N_{\text{FFN}}} = \frac{8d^2}{4d^2 + 8d^2} = \frac{2}{3} \]

References for this session

  1. Hendrycks & Gimpel (2016). Gaussian Error Linear Units (GELUs)
  2. Shazeer (2020). GLU Variants Improve Transformer
  3. Geva et al. (2020). Transformer Feed-Forward Layers Are Key-Value Memories
  4. Vaswani et al. (2017). Attention Is All You Need
  5. Hendrycks & Gimpel (2016). Gaussian Error Linear Units (GELUs)
  6. Shazeer (2020). GLU Variants Improve Transformer
  7. Geva et al. (2020). Transformer Feed-Forward Layers Are Key-Value Memories
  8. Dauphin, Fan, Auli, Grangier (2017). Language Modeling with Gated Convolutional Networks
  9. Ramachandran, Zoph, Le (2017). Searching for Activation Functions
  10. Geva, Caciularu, Wang, Goldberg (2022). Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
  11. Dai, Dong, Hao, Sui, Chang, Wei (2022). Knowledge Neurons in Pretrained Transformers
  12. Meng, Bau, Andonian, Belinkov (2022). Locating and Editing Factual Associations in GPT
  13. Bricken et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
  14. Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
  15. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  16. Llama Team, Meta (2024). The Llama 3 Herd of Models
  17. Jiang et al. (2024). Mixtral of Experts
  18. Fedus et al. (2021). Switch Transformers
  19. Zhang et al. (2021). MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
  20. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  21. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  22. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
  23. Song et al. (2023). PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU