The feed-forward network (MLP) inside each block
Attention moves information between tokens. The feed-forward network (FFN) then works on each token alone: it expands the vector to a wider space, applies a non-linear function, and projects it back. This session shows why that simple shape matters, how GELU and SwiGLU replaced ReLU, why many facts seem to live in these layers, and why they hold about two thirds of a block's weights.
What you will learn
- What the FFN computes, formula by formula, and why it is applied to each position separately
- Why the FFN expands to a wider hidden layer, and why a non-linearity is required (without it two layers collapse into one)
- ReLU, GELU and SiLU (Swish): their formulas, shapes and gradients
- Gated units: GLU, GEGLU and SwiGLU, and why modern LLMs use a hidden size of about 8d/3 instead of 4d
- The FFN as a key–value memory: what the evidence says about where facts are stored, and how edits like ROME use it
- Counting parameters and FLOPs: why the FFN holds about 2/3 of a dense block, and over 80% in Llama 3 with grouped-query attention
- Where FFNs appear in VLMs and video models: ViT blocks, projectors, prediction heads and Mixture of Experts
Why this matters
Every transformer block has two parts. The attention part moves information between tokens: each token looks at the others and copies what it needs. The second part, the feed-forward network (FFN, also called the MLP, for multi-layer perceptron), does something different. It takes each token’s vector on its own and transforms it. No token looks at any other token inside the FFN.
This second part is easy to overlook, because its formula is short. But it matters a lot, for four reasons:
- It holds most of the weights. In a classic dense block, the FFN has about 2/3 of the parameters. In Llama 3 8B, with its smaller attention, it has over 80% of each block. When people quantize, prune or shard a model, the FFN is the biggest target.
- It does most of the compute at normal context lengths. Each weight costs about 2 FLOPs per token, so the FFN also takes most of the arithmetic, until the context becomes very long.
- It seems to store much of what the model “knows”. Research suggests that FFN layers act like a large key–value memory: some hidden units fire on specific patterns in the input and push the output toward specific next tokens. Methods that edit single facts in a model change FFN weights.
- Its design changed in modern LLMs. The original ReLU FFN became GELU in GPT-2 and BERT, and then the gated SwiGLU in PaLM, Llama, Mistral and Qwen. This change is behind the strange hidden sizes you see in model configs, such as 11,008 or 14,336.
Interviewers ask about the FFN often: “What is the shape of the MLP in a transformer?”, “Why expand to 4d?”, “What is SwiGLU and why is the hidden size 8d/3?”, “Where are the parameters in a block?”, “Where are facts stored?”. In Session 20 we finished the attention part of the block. Here we build the other half. In session 22 we will glue the two halves together with residual connections and normalization.
What the FFN computes
Expand, activate, project back
Let \(\mathbf{x} \in \R^d\) be the vector of one token after attention (as a row vector, the same convention as session 11). The original transformer (Vaswani et al., 2017) defines the FFN as two linear layers with a non-linear function between them:
- Read the symbols
- \(d\)
- the model width: the length of each token vector (512 in the original transformer, 4,096 in Llama 3 8B)
- \(d_{\text{ff}}\)
- the hidden size of the FFN, usually larger than \(d\). Classic choice: \(d_{\text{ff}} = 4d\) (2,048 for \(d = 512\)).
- \(W_1, \mathbf{b}_1\)
- the up projection (also “expand” or “fc1”) and its bias: maps \(d\) numbers to \(d_{\text{ff}}\) numbers
- \(\phi\)
- the activation function, a simple non-linear function applied to each number separately: ReLU \(\max(0, u)\) in the original paper, GELU or SiLU in later models
- \(W_2, \mathbf{b}_2\)
- the down projection (also “project back” or “fc2”) and its bias: maps \(d_{\text{ff}}\) numbers back to \(d\)
So the FFN has three steps: expand the vector from \(d\) to \(d_{\text{ff}}\) numbers, apply \(\phi\) to every number, and project back to \(d\) numbers. The output has the same shape as the input, so it can be added back to the token’s vector (the residual connection, next session).
Position-wise: the same weights for every token
For a sequence of \(T\) tokens, stack the vectors as rows of \(X \in \R^{T \times d}\). The FFN is applied to every row with the same weights:
Here \(\mathbf{1}\) is a column of \(T\) ones, so the bias is added to every row. Row \(t\) of \(Y\) depends only on row \(t\) of \(X\). This is why the paper calls it a position-wise feed-forward network, and also describes it as “two convolutions with kernel size 1”. [Certain] Two practical facts follow:
- All tokens go through the FFN as one big matrix multiplication of shape \((T \times d)(d \times d_{\text{ff}})\). In training this is a dense, GPU-friendly operation with no masks.
- If you shuffle the tokens, the FFN outputs are shuffled in the same way. The FFN cannot mix information between positions; only attention can.
Figure 1 uses \(d = 4\) and \(d_{\text{ff}} = 8\), with GELU and no biases. The token “cat” has \(\mathbf{x} = (1.0, -0.5, 0.3, 0.8)\). Column 1 of \(W_1\) is \((0.5, -0.3, 0.8, 0.1)\). The first hidden pre-activation is \(u_1 = 1.0 \cdot 0.5 + (-0.5)(-0.3) + 0.3 \cdot 0.8 + 0.8 \cdot 0.1 = 0.5 + 0.15 + 0.24 + 0.08 = 0.97\). After GELU, \(h_1 = 0.97\,\Phi(0.97) = 0.809\). The other seven hidden units are computed the same way with their own columns. Then the output is \(\mathbf{y} = \sum_{i=1}^{8} h_i\, (W_2)_{i,:}\) = \((0.481, 0.141, -0.388, 0.357)\), and the block adds it to the input: \(\mathbf{x} + \mathbf{y} = (1.481, -0.359, -0.088, 1.157)\).
How to read Figure 1
Press Next to walk through the four operations of formula (21.1). In step 2 the 4 input numbers become 8 numbers: each one is the dot product of \(\mathbf{x}\) with one column of \(W_1\). In step 3, GELU keeps the positive numbers almost unchanged and squeezes the negative ones close to zero (but not exactly to zero, unlike ReLU). Step 4 mixes the 8 numbers back into 4. Now switch to the token “sat”: the same weights give a different pattern of active hidden units (for “cat”, units 1, 3 and 4 are the strongest; for “sat”, units 2, 5 and 7). Each token uses the same network, but a different part of it.
Why expand, and why a non-linearity
Without \(\phi\), two layers are only one
Why not just use one \(d \times d\) matrix? And why do we need \(\phi\) at all? Remove \(\phi\) (and the biases, for simplicity) from formula (21.1):
The two layers collapse into one \(d \times d\) matrix. All the extra width is wasted: a product of linear maps is just another linear map. A whole stack of transformer blocks without non-linear FFNs would still be mostly linear in each token’s features (attention adds a softmax, but softmax only chooses how much of each value to mix; it does not transform the features themselves). The function \(\phi\) is what lets the network compute things like “this AND that”, thresholds, and curved functions.
What the width buys
With \(\phi\), each hidden unit \(i\) computes \(h_i = \phi(\mathbf{x} \cdot \mathbf{w}_i + b_i)\), where \(\mathbf{w}_i\) is column \(i\) of \(W_1\). With ReLU, this is zero on one side of a flat boundary (a hyperplane) and grows linearly on the other side. The down projection adds up \(d_{\text{ff}}\) such pieces. More pieces allow more complex functions: an old result, the universal approximation theorem, says that one hidden layer with enough units can approximate any continuous function on a bounded region as closely as you want. [Certain] The theorem says nothing about how many units you need, but in practice wider is more expressive.
The choice \(d_{\text{ff}} = 4d\) is not a law. It is the value the original transformer used (2,048 for \(d = 512\)), and GPT-2, BERT and ViT kept it. [Certain] It balances the cost of the FFN with the cost of attention, and later scaling studies found that the loss is not very sensitive to this ratio within a broad range. [Likely]
Figure 2 shows both facts in one dimension: a network with one input, one output and a hidden layer of width \(n\), fitted to a curved target. The hidden weights are random and fixed; only the output weights are fitted (by least squares), which is enough to see the effect of width and of \(\phi\).
How to read Figure 2
Choose “no activation”: the fit is a straight line for every width, with mean squared error (MSE) 0.433, because a sum of linear functions is linear (formula 21.3). The target’s own variance is 0.489, so the line explains very little. Now choose ReLU and move the width: with 4 units the error is still 0.458, with 8 units it drops to 0.045, and with 32 units to 0.0008. The curve is built from straight pieces that bend at the thin lines. GELU gives a smooth curve and gets there with fewer units (0.002 at 8 units). Real FFNs have thousands of units in thousands of input dimensions, and all weights are trained, but the principle is the same.
Activation functions: ReLU, GELU and SiLU
The activation \(\phi\) is applied to each hidden number separately. Three choices matter for LLMs.
ReLU (rectified linear unit), \(\mathrm{ReLU}(u) = \max(0, u)\), was used in the original transformer. It is cheap and its gradient is simple: 1 for positive inputs, 0 for negative ones. That second fact is also its weak point: a unit whose input is negative for every token gets no gradient at all and may never recover (a “dead” unit).
GELU (Gaussian error linear unit, Hendrycks and Gimpel, 2016) multiplies the input by the probability that a standard normal variable is smaller than it:
For large positive \(u\), \(\Phi(u) \approx 1\) and GELU is almost \(u\). For large negative \(u\), \(\Phi(u) \approx 0\) and GELU is almost 0. In between it is smooth, and it dips slightly below zero, to a minimum of about \(-0.170\) at \(u \approx -0.75\). GPT-2 and many other models use a faster approximation with tanh:
The two versions differ by at most about 0.00047 (near \(u = -2.7\)). [Certain, checked numerically] In PyTorch they are F.gelu(u) and F.gelu(u, approximate="tanh"). A model should be run with the version it was trained with.
SiLU (sigmoid linear unit), also called Swish, multiplies the input by its own sigmoid:
- Read the symbols
- \(\Phi(u)\)
- the cumulative distribution function of the standard normal distribution: the probability that a \(\mathcal{N}(0,1)\) number is below \(u\)
- \(\operatorname{erf}\)
- the error function, a standard special function; \(\Phi\) is written with it
- \(\sigma(u)\)
- the sigmoid \(1/(1 + e^{-u})\), between 0 and 1
- \(\beta\)
- a sharpness factor. Swish with \(\beta = 1\) is SiLU, the version used in LLMs. As \(\beta \to \infty\), Swish becomes ReLU.
GELU and SiLU look alike: both are a smooth version of ReLU, both are slightly negative for negative inputs (SiLU’s minimum is about \(-0.278\) at \(u \approx -1.28\)), and both have a non-zero gradient almost everywhere, so units do not die as easily. Swish was found by an automatic search over activation functions (Ramachandran et al., 2017). [Certain] Why smooth functions train a bit better is not fully understood; the usual explanations are smoother gradients and no dead units. [Likely]
At \(u = 1\): ReLU gives 1, GELU gives \(1 \cdot \Phi(1) = 0.841\), SiLU gives \(\sigma(1) = 0.731\). At \(u = -1\): ReLU gives 0, GELU gives \(-0.159\), SiLU gives \(-0.269\). The gradients at \(u = -1\) are 0 (ReLU), \(-0.083\) (GELU) and \(0.072\) (SiLU); at \(u = 0\) both smooth functions have gradient exactly 0.5. So a GELU or SiLU unit with a negative input still receives a small gradient and can move.
How to read Figure 3
Move \(u\) from left to right. For \(u \gt 3\) all three curves are almost the same line \(y = u\). The difference is near zero and below it: ReLU is flat at 0, GELU dips to about \(-0.17\) near \(u = -0.75\), and SiLU dips deeper, to about \(-0.28\) near \(u = -1.28\). Tick the gradient box: ReLU’s gradient jumps from 0 to 1 at \(u = 0\), while the smooth functions change gradually and even overshoot 1 a little for positive \(u\) (GELU’s gradient at \(u = 1\) is 1.083). A negative gradient (GELU for every \(u\) below about \(-0.75\), SiLU below about \(-1.28\)) means that making the input more negative raises the output toward zero.
Gated units: GLU and SwiGLU
The gate
A gated linear unit (GLU) computes two projections of the input and multiplies them element by element. One of them goes through a sigmoid and acts as a gate: a number between 0 and 1 that decides how much of the other projection passes. Dauphin et al. (2017) introduced it for convolutional language models: [Certain]
Here \(\odot\) means element-wise multiplication. Shazeer (2020) put this idea into the transformer FFN and tried different functions in place of the sigmoid: ReLU (ReGLU), GELU (GEGLU), Swish (SwiGLU), and none at all (Bilinear). The SwiGLU FFN, without biases, is:
- Read the symbols
- \(W_{\text{gate}} \in \R^{d \times d_{\text{ff}}}\)
- the gate projection (called
gate_projorw1in Llama code). Its output goes through SiLU. - \(W_{\text{up}} \in \R^{d \times d_{\text{ff}}}\)
- the up projection (
up_projorw3). Its output is not activated; it is multiplied by the gate. - \(W_{\text{down}} \in \R^{d_{\text{ff}} \times d}\)
- the down projection (
down_projorw2), the same role as \(W_2\) in (21.1) - \(\odot\)
- element-wise product of two vectors of length \(d_{\text{ff}}\)
Why should a product help? In the plain FFN, hidden unit \(i\) is \(\phi(\mathbf{x} \cdot \mathbf{w}_i)\): a function of one direction of the input. In SwiGLU it is \(\mathrm{SiLU}(\mathbf{x} \cdot \mathbf{g}_i)\,(\mathbf{x} \cdot \mathbf{u}_i)\): one direction decides whether the unit is open, another direction decides what value passes. The product of two linear functions is a quadratic function of \(\mathbf{x}\), so each unit can express interactions between features, such as “feature A matters only when feature B is present”. This is an intuition, not a proof. [Likely]
The 8d/3 rule
A gated FFN has three weight matrices instead of two. To compare fairly at equal parameters and FLOPs, Shazeer reduced the hidden size by a factor of 2/3. [Certain] With \(d_{\text{ff}} = 4d\) for the plain FFN:
In the T5 base model of the paper (\(d = 768\)), the plain FFN used \(d_{\text{ff}} = 3{,}072\) and the gated ones used 2,048: \(2 \times 768 \times 3{,}072 = 3 \times 768 \times 2{,}048 = 4{,}718{,}592\) weights in both cases. [Certain] The results on the C4 dataset, as held-out log-perplexity (lower is better):
| FFN variant | Hidden size | After 65,536 steps | After 524,288 steps |
|---|---|---|---|
| ReLU (baseline) | 3,072 | 1.997 | 1.677 |
| GELU | 3,072 | 1.983 | 1.679 |
| Swish | 3,072 | 1.994 | 1.683 |
| GLU (sigmoid gate) | 2,048 | 1.982 | 1.663 |
| Bilinear (no function) | 2,048 | 1.960 | 1.648 |
| ReGLU | 2,048 | 1.953 | 1.645 |
| GEGLU | 2,048 | 1.942 | 1.633 |
| SwiGLU | 2,048 | 1.944 | 1.636 |
Numbers from Table 1 of Shazeer (2020). [Certain] Three things to notice. The plain activations (ReLU, GELU, Swish) are within about 0.015 of each other. All gated versions beat all plain ones at equal size. And even the Bilinear version, with no non-linear function except the product itself, beats the plain ones. The gains also survive fine-tuning on GLUE and SuperGLUE in the paper. The paper famously ends: “We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.” [Certain]
PaLM (2022) adopted SwiGLU, noting that it needs three matrix multiplications instead of two, and also removed all biases, which it found gave more stable training for large models. [Certain] Llama, Mistral and Qwen use SwiGLU without biases; Gemma uses GEGLU. [Certain]
Llama code computes the hidden size from \(4d\): take 2/3 of it, optionally multiply by a factor, and round up to a multiple of some number \(m\). [Certain for the procedure; the per-model factors below are from the released configurations. Likely]
Llama 2 7B (\(d = 4{,}096\), \(m = 256\), no factor): \(\tfrac{2}{3} \times 16{,}384 = 10{,}922.7\), truncated to 10,922; the next multiple of 256 is \(43 \times 256 = 11{,}008\).
Llama 3 8B (\(d = 4{,}096\), factor 1.3, \(m = 1{,}024\)): \(10{,}922 \times 1.3 = 14{,}198.6\), truncated to 14,198; the next multiple of 1,024 is \(14 \times 1{,}024 = 14{,}336\). This matches intermediate_size: 14336 in the model’s config.json. [Certain]
Llama 3 70B (\(d = 8{,}192\), factor 1.3, \(m = 4{,}096\)): \(21{,}845 \times 1.3 = 28{,}398.5\), so 28,672 \(= 7 \times 4{,}096\). Rounding to a multiple of a large power of two keeps the matrix shapes efficient on GPUs and easy to split across devices.
The FFN as a key–value memory
Rewriting the formula
Write the columns of \(W_1\) as vectors \(\mathbf{k}_1, \dots, \mathbf{k}_{d_{\text{ff}}}\) and the rows of \(W_2\) as \(\mathbf{v}_1, \dots, \mathbf{v}_{d_{\text{ff}}}\). Then formula (21.1), without biases, is a sum:
- Read the symbols
- \(\mathbf{k}_i \in \R^d\)
- the key of memory \(i\): column \(i\) of \(W_1\). A pattern the unit responds to.
- \(m_i\)
- the memory coefficient: the activation of hidden unit \(i\). Large when the input points in the direction of \(\mathbf{k}_i\).
- \(\mathbf{v}_i \in \R^d\)
- the value of memory \(i\): row \(i\) of \(W_2\). The vector that gets added to the residual stream when the memory fires.
This looks very much like attention (session 16): compare a query with keys, then sum values weighted by the match. There are two differences. The keys and values here are fixed parameters learned in training, not computed from the context. And the weights \(m_i\) come from \(\phi\), not from a softmax, so they do not need to sum to 1 and many can be zero. Geva et al. (2021) called this view “feed-forward layers are key–value memories”. [Certain]
What the evidence says
- Keys detect patterns. Geva et al. studied a 16-layer language model with \(d = 1{,}024\) and \(d_{\text{ff}} = 4{,}096\) trained on WikiText-103. For sampled hidden units, they collected the training prefixes that activate them most. Human annotators found recognizable patterns for every sampled key. In lower layers the patterns were mostly shallow (for example, prefixes ending with the same word); in upper layers they were more semantic (prefixes about the same topic, with different words). [Certain]
- Values predict next tokens. If you multiply a value vector \(\mathbf{v}_i\) by the output embedding matrix, you get a score for every token in the vocabulary. In the upper layers, the top token of a value often agreed with the token that actually followed its key’s trigger prefixes, far more often than chance. [Certain] A later paper by Geva et al. (2022) showed that FFN updates can be read as promoting human-readable concepts in this vocabulary space. [Certain]
- Outputs are mixtures. A typical input activates hundreds of memories in a layer, and in most cases the layer’s top prediction is not the top prediction of any single memory: the output is a composition of many memories. [Certain]
- Facts can be located and edited. Dai et al. (2022) found “knowledge neurons” in BERT’s FFN layers whose activation is linked to specific facts; suppressing or amplifying them changed the probability of the correct answer. [Certain] Meng et al. (2022, ROME) traced factual recall in GPT models to the FFN layers in the middle of the network, at the last token of the subject (“Eiffel Tower” in “The Eiffel Tower is in”). They edited one FFN matrix with a rank-one update to change a single fact. [Certain]
A careful reading of this evidence matters, because the key–value view is easy to oversell:
- Units are often polysemantic. One hidden unit can respond to several unrelated patterns, because the model stores more features than it has units (this is called superposition). Work on sparse autoencoders (Bricken et al., 2023) tries to split activations into cleaner, “monosemantic” features. [Certain]
- Facts are spread out. A fact is usually supported by many units in several layers, plus attention that moves the subject’s information to the right position. Editing one place can fail to change related questions, or can damage other facts. [Likely]
- “Where facts are stored” is a simplification. The evidence shows that FFN layers are a main place where the model writes fact-related information into the stream, not that each fact has a single address. [Likely]
Figure 4 is a toy version of formula (21.10), with 6 hand-made memories and a vocabulary of 8 tokens. To keep it readable, each value is shown already multiplied by the output embedding, as a change in the logits of the 8 tokens.
How to read Figure 4
For “The capital of France is”, three memories fire: memory 1 (capital AND France, which needs both features to pass its bias of \(-1\)), memory 5 (any capital) and memory 6 (anything about France). Paris gets 98.1%. Switch off memory 1: Paris drops to 48.1% but is still first, because memories 5 and 6 still help. Switch off memory 6 too: the four capitals tie at 17.1% each. The fact was stored in more than one place. Now reset and tick the edit: Rome wins the France prompt with 86.8%, while the Italy prompt does not change at all (memory 1 does not fire for it). But Paris keeps 7.1%, pushed by memory 6: an edit in one place leaves related memories behind. This is the same trade-off that real editing methods face.
Counting parameters and FLOPs
Why about 2/3
Count the weight matrices of one block, ignoring biases and normalization (they are tiny). Standard multi-head attention (session 19) has four \(d \times d\) matrices: \(W_Q, W_K, W_V, W_O\). The plain FFN with \(d_{\text{ff}} = 4d\) has two matrices of \(d \times 4d\):
A SwiGLU FFN with \(d_{\text{ff}} = 8d/3\) also has \(3 \cdot d \cdot 8d/3 = 8d^2\) weights, so the share stays 2/3. Two things move it up in modern models. First, grouped-query attention (GQA, session 26) shares keys and values across heads, so \(W_K\) and \(W_V\) become smaller. Second, many models choose \(d_{\text{ff}}\) above \(8d/3\). In general, with \(h\) query heads, \(h_{kv}\) key/value heads and head size \(d_h\):
| Model | \(d\) | \(d_{\text{ff}}\) | FFN type | Heads (query / KV) | Attention per block | FFN per block | FFN share |
|---|---|---|---|---|---|---|---|
| Transformer base (2017) | 512 | 2,048 | ReLU | 8 / 8 | 1.05 M | 2.10 M | 66.7% |
| GPT-2 small | 768 | 3,072 | GELU | 12 / 12 | 2.36 M | 4.72 M | 66.7% |
| Llama 2 7B | 4,096 | 11,008 | SwiGLU | 32 / 32 | 67.1 M | 135.3 M | 66.8% |
| Llama 3 8B | 4,096 | 14,336 | SwiGLU | 32 / 8 | 41.9 M | 176.2 M | 80.8% |
| Llama 3 70B | 8,192 | 28,672 | SwiGLU | 64 / 8 | 151.0 M | 704.6 M | 82.4% |
Sizes from the papers and public configurations; weights only, no biases or norms. [Certain] The whole-model share is lower, because the embedding tables (session 11) are outside the blocks: the 32 FFNs of Llama 3 8B hold 5.64 B of its 8.03 B parameters (70.2%), and the 32 FFNs of Llama 2 7B hold 4.33 B of 6.74 B (64.2%).
FLOPs per token
A matrix multiplication costs 2 FLOPs (one multiply, one add) per weight per token. So the FFN costs \(2 N_{\text{FFN}}\) FLOPs per token per layer, no matter how long the context is. Attention has two parts: the four projections cost \(2 N_{\text{attn}}\), and the score and value products \(QK^\top\) and \(AV\) cost about \(4 T d\) for a token that attends to \(T\) others (session 25 does the full count):
The FFN stays the biggest cost until \(4Td\) catches up. For GPT-2 small, \(4Td = 16d^2\) gives \(T = 4d = 3{,}072\). For Llama 3 8B, \(4Td = 6\, d\, d_{\text{ff}}\) gives \(T = 1.5\, d_{\text{ff}} = 21{,}504\) tokens. Below these lengths, making the FFN cheaper (quantization, sparsity, MoE) is where the compute savings are; above them, attention becomes the main cost. This is the theme of session 93.
How to read Figure 5
The top bar splits the block’s parameters, the bottom bar splits the FLOPs for one token that attends to \(T\) tokens. Load GPT-2 small: the FFN has 66.7% of the weights. Load Llama 2 7B: still about 2/3 with SwiGLU at 11,008. Load Llama 3 8B: 80.8%, because grouped-query attention made attention smaller. Now move the context slider: at \(T = 4{,}096\) the FFN still does 70% of the FLOPs, but at \(T = 131{,}072\) the attention scores dominate and the FFN falls to 14%. Untick “Gated” on Llama 3 8B: the FFN loses a third of its weights.
Beyond text: FFNs in vision and video models
The same expand–activate–project block appears all over vision-language models:
- Vision Transformer blocks. ViT (session 164) uses exactly the GPT-style block: attention, then an MLP with GELU and a 4× hidden size (ViT-Base: \(d = 768\), hidden size 3,072). [Certain] Video transformers such as TimeSformer and ViViT keep this FFN and change only the attention pattern over space and time. [Likely] So for a video encoder too, about 2/3 of each block’s weights are in the FFN.
- Projectors. A VLM must map vision features to the LLM width. LLaVA-1.5 uses a two-layer MLP for this (LLaVA used a single linear layer). [Certain] That projector is a small FFN with a different output size: vision width in, LLM width out (session 178).
- Prediction heads. Detection and temporal-grounding models put small MLPs on top of the transformer to predict boxes or time spans. Moment-DETR, for key-moment detection in video, predicts the start and end of a moment with such heads. [Likely]
- Mixture of Experts in VLMs. Several large VLMs use MoE language backbones, so the FFN of the language model becomes many experts while the vision encoder stays dense. [Likely]
Where this is used
| Application | What the FFN does there | Link |
|---|---|---|
| Every decoder LLM (Llama, Mistral, Qwen, Gemma) | SwiGLU or GEGLU FFN in every block, formula (21.8); about 2/3 to over 80% of each block’s weights | session 23 |
| Mixture of Experts (Mixtral, DeepSeek-V3) | Each FFN is replaced by many expert FFNs and a router; more parameters, same compute per token | session 29 |
| Quantization, pruning and sparsity | FFN matrices are the largest weights, so they get the most attention in compression. ReLU-style activations are often zero, which some inference systems exploit to skip work. | session 94 |
| Knowledge editing and interpretability | Facts are traced to mid-layer FFNs and edited with low-rank weight updates (ROME); units and their values are inspected as key–value memories (21.10) | session 157 |
| Vision encoders and video action recognition | ViT and video transformer blocks use a GELU MLP with 4× expansion after each attention layer; the clip-level features that classify actions come out of these blocks | session 203 |
| VLM projectors | A 2-layer MLP maps vision-encoder outputs to the LLM width (LLaVA-1.5) | session 178 |
| Key-moment detection and temporal grounding | Small MLP heads on transformer outputs predict moment start and end times and a confidence score | session 207 |
| Text-to-video retrieval and Video-RAG | The text and video encoders that produce the embeddings are transformer stacks; their FFNs are most of the encoder weights and the main target when the encoders are compressed for large-scale indexing | session 208 |
Pseudo-code
- input: \(X \in \R^{T \times d}\), weights \(W_1 \in \R^{d \times d_{\text{ff}}}\), \(\mathbf{b}_1\), \(W_2 \in \R^{d_{\text{ff}} \times d}\), \(\mathbf{b}_2\), activation \(\phi\)
- \(U \leftarrow X W_1 + \mathbf{b}_1\) // expand: \((T, d)\) to \((T, d_{\text{ff}})\), one big matmul for all tokens
- \(H \leftarrow \phi(U)\) // element-wise: ReLU, GELU (21.4) or SiLU (21.6)
- \(Y \leftarrow H W_2 + \mathbf{b}_2\) // project back: \((T, d_{\text{ff}})\) to \((T, d)\)
- return \(X + Y\) // the block adds the result to the residual stream
- sizing: \(h \leftarrow \lfloor \tfrac{2}{3} \cdot 4d \rfloor\); if a factor \(f\) is given then \(h \leftarrow \lfloor f\, h \rfloor\)
- \(d_{\text{ff}} \leftarrow m \cdot \lceil h / m \rceil\) // round up to a multiple of \(m\): 11,008 or 14,336
- input: \(X \in \R^{T \times d}\), \(W_{\text{gate}}, W_{\text{up}} \in \R^{d \times d_{\text{ff}}}\), \(W_{\text{down}} \in \R^{d_{\text{ff}} \times d}\)
- \(G \leftarrow \mathrm{SiLU}(X W_{\text{gate}})\) // the gate, \((T, d_{\text{ff}})\)
- \(U \leftarrow X W_{\text{up}}\) // the content, \((T, d_{\text{ff}})\); in practice fused with the gate into one matmul
- \(Y \leftarrow (G \odot U)\, W_{\text{down}}\) // element-wise product, then project back
- return \(X + Y\)
- input: one token vector \(\mathbf{x}\), FFN weights, output embedding \(E \in \R^{|V| \times d}\), number \(k\)
- \(\mathbf{m} \leftarrow \phi(\mathbf{x} W_1 + \mathbf{b}_1)\) // memory coefficients, one per hidden unit
- \(I \leftarrow\) indices of the \(k\) largest entries of \(\mathbf{m}\) // the memories that fire most
- for \(i \in I\) do
- \(\mathbf{s} \leftarrow E\, \mathbf{v}_i\) // value \(i\) in vocabulary space, \(\mathbf{v}_i\) = row \(i\) of \(W_2\)
- report \(m_i\) and the tokens with the highest scores in \(\mathbf{s}\) // what this memory promotes
The same ideas in PyTorch. Each block checks one claim from this session (outputs from PyTorch 2.8 on CPU):
import torch import torch.nn as nn import torch.nn.functional as F torch.manual_seed(0) # 1) the classic FFN of formula (21.1): expand 4x, GELU, project back class FFN(nn.Module): def __init__(self, d, d_ff): super().__init__() self.up = nn.Linear(d, d_ff) # W1, b1 self.down = nn.Linear(d_ff, d) # W2, b2 def forward(self, x): # x: (B, T, d) return self.down(F.gelu(self.up(x))) # 2) the SwiGLU FFN of formula (21.8), as in Llama (no biases) class SwiGLU(nn.Module): def __init__(self, d, d_ff): super().__init__() self.gate = nn.Linear(d, d_ff, bias=False) # W_gate self.up = nn.Linear(d, d_ff, bias=False) # W_up self.down = nn.Linear(d_ff, d, bias=False) # W_down def forward(self, x): return self.down(F.silu(self.gate(x)) * self.up(x)) count = lambda m: sum(p.numel() for p in m.parameters()) print(count(FFN(768, 3072))) # 4722432 (GPT-2 small, with biases) with torch.device("meta"): # no memory is allocated print(count(SwiGLU(4096, 14336))) # 176160768 (Llama 3 8B, one layer) # 3) position-wise: permuting the tokens permutes the outputs ffn = FFN(16, 64) x = torch.randn(2, 5, 16) perm = torch.tensor([3, 0, 4, 1, 2]) print(torch.allclose(ffn(x)[:, perm], ffn(x[:, perm]), atol=1e-6)) # True # 4) without the activation, two layers collapse into one d x d matrix (21.3) W1, W2 = torch.randn(16, 64), torch.randn(64, 16) print(torch.allclose((x @ W1) @ W2, x @ (W1 @ W2), atol=1e-4)) # True print(torch.linalg.matrix_rank(W1 @ W2).item()) # 16, never more than d # 5) key-value view (21.10): the output is a sum of value rows weighted by key matches K = ffn.up.weight # (d_ff, d): row i is key k_i Vt = ffn.down.weight.T # (d_ff, d): row i is value v_i t = x[0, 0] m = F.gelu(K @ t + ffn.up.bias) # one coefficient per memory y = (m[:, None] * Vt).sum(0) + ffn.down.bias print(torch.allclose(y, ffn(t), atol=1e-6)) # True # 6) GELU: exact vs tanh approximation (21.4), (21.5) z = torch.linspace(-6, 6, 120001) print(F.gelu(torch.tensor(1.0)).item()) # 0.8413... print((F.gelu(z) - F.gelu(z, approximate="tanh")).abs().max().item()) # about 4.7e-4 # 7) Llama's FFN width rule (21.9): 2/3 of 4d, times a factor, rounded up def llama_ffn_dim(d, multiple_of, mult=None): h = int(2 * 4 * d / 3) if mult is not None: h = int(mult * h) return multiple_of * ((h + multiple_of - 1) // multiple_of) print(llama_ffn_dim(4096, 256), llama_ffn_dim(4096, 1024, 1.3), llama_ffn_dim(8192, 4096, 1.3)) # 11008 14336 28672
Common mistakes
- Forgetting the third matrix. A SwiGLU FFN with hidden size \(d_{\text{ff}}\) has \(3\,d\,d_{\text{ff}}\) weights, not \(2\,d\,d_{\text{ff}}\). Counting Llama 3 8B’s FFN as two matrices gives 117 M per layer instead of 176 M.
- Assuming \(d_{\text{ff}} = 4d\) everywhere. Gated models use about \(8d/3\) and then round or scale it (11,008, 14,336, 28,672). Read
intermediate_sizein the config instead of guessing. - Mixing up gate and up projections. In \(\mathrm{SiLU}(\mathbf{x} W_{\text{gate}}) \odot \mathbf{x} W_{\text{up}}\), only one branch is activated. Swapping
w1andw3when loading weights gives a model that runs but produces nonsense, because the activation is applied to the wrong branch. - Using the wrong GELU. The exact (erf) and tanh versions differ by up to about 0.0005 per unit. That is small, but over many layers it can change outputs; when you port a model, use the version it was trained with.
- Thinking the FFN mixes tokens. It does not: it is applied to each position independently (21.2). Any information that a token needs from other tokens must arrive through attention first.
- Reading one neuron as one concept. Hidden units are often polysemantic, and facts are spread over many units and layers. “Neuron 3,217 stores the capital of France” is almost never the full story.
- Ignoring the FFN in memory planning. The hidden activation has shape \((B, T, d_{\text{ff}})\), wider than the residual stream; a gated FFN keeps two of them (gate and up) for the backward pass. This is a large part of activation memory in training (session 38).
Interview questions
What does the FFN in a transformer block compute, and how is it different from attention?basic
Why do we need the non-linearity, and why make the hidden layer wider than d?basic
Compare ReLU, GELU and SiLU. Why did LLMs move away from ReLU?core
What is SwiGLU, and why is Llama’s FFN hidden size 11,008 instead of 16,384?core
Count the parameters of one transformer block. Why is the FFN about 2/3? What changes in Llama 3 8B?core
At what context length does attention cost more FLOPs per token than the FFN?advanced
Explain the view of the FFN as a key–value memory. What is the evidence, and what are its limits?advanced
How does ROME edit a fact, and why does it target the FFN?advanced
Why do modern LLMs remove the biases from the FFN?basic
Why is the FFN the part that Mixture of Experts replaces?core
Where do MLPs appear in a vision-language model?core
Exercises
- A model has \(d = 1{,}024\) and a plain GELU FFN with \(d_{\text{ff}} = 4d\), plus standard multi-head attention. How many weights are in the FFN and in attention per block (no biases)? What is the FFN share?
- The same model switches to SwiGLU at equal parameters. What hidden size should it use before rounding? What is it after rounding up to a multiple of 256?
- Compute GELU and SiLU at \(u = 2\). Which is closer to ReLU?
- In Figure 4, for the prompt “The capital of Germany is”, which memories fire? What happens if you switch off memory 3 alone?
- A model has \(d = 2{,}048\), 16 query heads and 4 KV heads of size 128, and a SwiGLU FFN with \(d_{\text{ff}} = 8{,}192\). Compute the attention and FFN weights per block and the FFN share.
- Show that a SwiGLU FFN with \(d_{\text{ff}} = 4d\) (no reduction) has 3/4 of the block’s weights when attention is \(4d^2\).
- For GPT-2 small (\(d = 768\)), at what context length \(T\) does the attention score cost \(4Td\) equal the FFN cost per token?
Answerscheck
Key takeaways
- The FFN is the second half of every transformer block: expand from \(d\) to \(d_{\text{ff}}\), apply a non-linearity, project back (21.1). It acts on each position separately with the same weights (21.2); only attention mixes tokens.
- The non-linearity is essential: without it, the two layers collapse into one \(d \times d\) map (21.3). The width \(d_{\text{ff}}\) sets how many non-linear features each layer can compute.
- ReLU gave way to smooth GELU (21.4, 21.5) and SiLU (21.6), and then to gated units (21.7). SwiGLU (21.8) with \(d_{\text{ff}} \approx 8d/3\) (21.9) is the standard in Llama, Mistral and Qwen; this rule explains sizes like 11,008 and 14,336.
- The FFN can be read as a key–value memory (21.10): keys detect input patterns, values push toward output tokens. This is a main place where facts are written into the stream, and where editing methods like ROME act, but facts are spread out and units are polysemantic.
- A dense block puts about 2/3 of its weights in the FFN (21.11); with grouped-query attention it is over 80% (21.12). The FFN also does most of the FLOPs until the context reaches thousands to tens of thousands of tokens (21.13), which is why MoE, quantization and pruning target it first.
Key formulas
References for this session
- Hendrycks & Gimpel (2016). Gaussian Error Linear Units (GELUs)
- Shazeer (2020). GLU Variants Improve Transformer
- Geva et al. (2020). Transformer Feed-Forward Layers Are Key-Value Memories
- Vaswani et al. (2017). Attention Is All You Need
- Hendrycks & Gimpel (2016). Gaussian Error Linear Units (GELUs)
- Shazeer (2020). GLU Variants Improve Transformer
- Geva et al. (2020). Transformer Feed-Forward Layers Are Key-Value Memories
- Dauphin, Fan, Auli, Grangier (2017). Language Modeling with Gated Convolutional Networks
- Ramachandran, Zoph, Le (2017). Searching for Activation Functions
- Geva, Caciularu, Wang, Goldberg (2022). Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
- Dai, Dong, Hao, Sui, Chang, Wei (2022). Knowledge Neurons in Pretrained Transformers
- Meng, Bau, Andonian, Belinkov (2022). Locating and Editing Factual Associations in GPT
- Bricken et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Jiang et al. (2024). Mixtral of Experts
- Fedus et al. (2021). Switch Transformers
- Zhang et al. (2021). MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
- Song et al. (2023). PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU