Efficient FFN 1: SwiGLU and parameter sharing (ALBERT)
The feed-forward network (FFN) holds most of the weights of a large language model, so it is the first place to look for savings. This session covers two different kinds of saving. SwiGLU makes each FFN weight more useful, which is why model configs have odd hidden sizes near 8d/3. Parameter sharing (ALBERT) and factorized embeddings store fewer weights, but you will see that they do not reduce compute, and only sometimes reduce the bytes moved per token.
What you will learn
- Three different costs of a layer: weights stored, FLOPs per token, and bytes moved per decode step, and why a technique can cut one and leave the others alone
- SwiGLU: what it computes, why it gives better quality per parameter, and how the hidden size 8d/3 and its rounding (11,008, 14,336, 28,672) come out of one equation
- SwiGLU at inference time: fusing the gate and up matrices, the extra activation memory in prefill, tensor parallelism, and why its activations are not sparse
- ALBERT-style cross-layer sharing: one set of block weights applied many times, the ablation numbers, and the two ways to spend the saving (fewer parameters or more depth)
- What sharing saves (memory, checkpoint size, room for the KV cache) and what it does not save (FLOPs, latency, and usually HBM traffic), with MobileLLM as the case where it does cut traffic
- Factorized embeddings (V×E plus E×d) and tied input/output embeddings, and their effect on the output head’s compute and rank
Why this matters
In session 93 we measured where a model’s weights live. The survey by Zhou et al. reports that the FFN holds 63.01% of the parameters of LLaMA-7B and 71.69% of LLaMA-70B. [Certain] So if you want a smaller or cheaper model, the FFN is the first place to look.
This session covers two very different ideas, and the most important skill is to keep them apart.
- Make each weight more useful. SwiGLU changes the FFN’s formula. At the same number of weights it gives a lower loss. Almost every open LLM uses it (Llama, Mistral, Qwen), and it is the reason configs contain odd hidden sizes such as 11,008 or 14,336.
- Store fewer weights. ALBERT shares one block’s weights across all layers and shrinks the embedding table with a factorization. Small on-device models such as MobileLLM use related tricks.
The trap is that “fewer parameters” sounds like “faster”. It is not, in general. A layer has three separate costs: the bytes it stores, the FLOPs it does per token, and the bytes it moves from memory per decoding step (session 82). Sharing cuts the first one. It leaves the second one exactly the same, and it usually leaves the third one the same too. We will see the one case where it does not.
This is the first of three sessions on efficient FFNs. The next two (session 95 and session 96) cover Mixture of Experts, which takes the opposite approach: it stores more FFN weights, but uses only a few of them per token.
Three costs of one layer
Take one weight matrix with \(P\) numbers, stored in a format with \(b\) bytes per number (\(b = 2\) for BF16). A batch of \(B\) tokens goes through it. Then:
- Read the symbols
- \(P\)
- number of weights in the matrix (for a \(d \times d_{\text{ff}}\) matrix, \(P = d\, d_{\text{ff}}\))
- \(b\)
- bytes per weight: 2 in BF16, 1 in INT8, 0.5 in INT4
- \(B\)
- tokens processed together in this step (the batch in decode, the prompt length in prefill)
- \(2BP\)
- one multiply and one add per weight per token
- moved
- bytes read from GPU memory (HBM). The weights are read once per step, whatever \(B\) is; the activations are small in decode
The ratio FLOPs / bytes moved is the arithmetic intensity. In BF16 decode it is about \(2BP / 2P = B\) FLOPs per byte. An H100 SXM can do 989 TFLOP/s but read only 3.35 TB/s, so its ridge point is about 295 FLOPs per byte (session 82). Below about 295 tokens per step, the FFN waits for memory, not for math. Keep this in mind: it explains why a technique that adds FLOPs can be free, and why a technique that cuts stored bytes can still be slow.
| Technique (this session) | Stored bytes | FLOPs per token | Bytes moved per decode step | Quality |
|---|---|---|---|---|
| SwiGLU at equal parameters | same | same | same | better |
| Cross-layer sharing, same depth (ALBERT-base) | much lower | same | same, unless the shared block stays in on-chip memory | lower |
| Sharing, more depth at same parameters (MobileLLM-LS) | same | higher | same, if repeats are back to back | better |
| Factorized embedding, input side | lower | slightly higher (\(2Ed\)) | lower | slightly lower |
| Factorized, tied output head | lower | lower | lower | lower; rank is capped at \(E\) |
The rest of the session fills in this table with formulas and real numbers.
SwiGLU: more quality per parameter
The formula
Session 21 introduced the gated FFN. Here is the short version. A plain FFN computes \(\phi(\mathbf{x} W_1) W_2\) with an activation \(\phi\) such as GELU. A gated FFN computes two projections of the input, puts one through an activation, and multiplies them element by element. The SwiGLU FFN (Shazeer, 2020) without biases is:
- Read the symbols
- \(\mathbf{x} \in \R^{d}\)
- one token’s vector (after normalization)
- \(W_{\text{gate}}, W_{\text{up}} \in \R^{d \times d_{\text{ff}}}\)
- the gate and up projections (
gate_proj,up_projin Hugging Face Llama code) - \(W_{\text{down}} \in \R^{d_{\text{ff}} \times d}\)
- the down projection back to width \(d\) (
down_proj) - \(\odot\)
- element-wise product of two vectors of length \(d_{\text{ff}}\)
- \(\sigma\)
- the sigmoid function; SiLU is also called Swish
Hidden unit \(i\) outputs \(\mathrm{SiLU}(\mathbf{x}\cdot\mathbf{g}_i)\,(\mathbf{x}\cdot\mathbf{u}_i)\): one direction of the input decides how open the unit is, another decides what value passes. A plain unit can only respond to one direction. That extra freedom is the usual explanation for the better quality; the paper itself offers no explanation. [Certain]
Quality at equal parameters
A gated FFN has three matrices instead of two. To compare fairly, Shazeer gave every variant the same number of weights and the same FLOPs, by shrinking the gated hidden size. On T5-base (\(d = 768\)) trained on C4, the held-out log-perplexity (lower is better) was: [Certain]
| FFN | Hidden size | Weights per FFN | After 65,536 steps | After 524,288 steps |
|---|---|---|---|---|
| ReLU (baseline) | 3,072 | 4,718,592 | 1.997 | 1.677 |
| GELU | 3,072 | 4,718,592 | 1.983 | 1.679 |
| GEGLU | 2,048 | 4,718,592 | 1.942 | 1.633 |
| SwiGLU | 2,048 | 4,718,592 | 1.944 | 1.636 |
Same weights, same FLOPs, about 0.04 lower loss after the long run. The full table is in session 21. The effect also shows up in small modern models: in MobileLLM, switching a 125M model from a plain FFN to SwiGLU raised the average zero-shot reasoning score from 42.6 to 43.9. [Certain] This is what “better quality per parameter” means: for a fixed memory and compute budget you get a better model. It is a free win at inference time, which is why it spread to almost every LLM.
Why the hidden size is about \(8d/3\)
Suppose the plain FFN you want to match has hidden size \(r d\) (classically \(r = 4\)). Equal weights means:
Real code then rounds the result to a hardware-friendly size. The Llama reference code takes \(\lfloor 2 \cdot 4d / 3 \rfloor\), optionally multiplies by a factor \(\alpha\), and rounds up to a multiple of \(m\):
- Read the symbols
- \(\alpha\)
- an extra width factor chosen by the model designers (1.0 for Llama 2 7B, 1.3 for Llama 3 8B and 70B) [Likely]
- \(m\)
- the rounding multiple (256, 1,024 or 4,096 in the Llama family), so that matrix shapes split well over tensor cores and GPUs
- \(\lfloor\cdot\rfloor, \lceil\cdot\rceil\)
- round down, round up
Llama 2 7B, \(d = 4{,}096\), \(\alpha = 1\), \(m = 256\): \(8 \times 4{,}096 / 3 = 10{,}922.7 \to 10{,}922\), rounded up to \(43 \times 256 = 11{,}008\). The SwiGLU FFN has \(3 \times 4{,}096 \times 11{,}008 = 135{,}266{,}304\) weights per layer, only 0.8% more than a plain \(4d\) FFN (\(134{,}217{,}728\)).
Llama 3 8B, \(\alpha = 1.3\), \(m = 1{,}024\): \(\lfloor 1.3 \times 10{,}922 \rfloor = 14{,}198\), rounded up to \(14 \times 1{,}024 = 14{,}336\). Now the FFN has 176,160,768 weights, 31% more than a plain \(4d\) FFN. The model chose to spend extra weights on the FFN.
MobileLLM-125M, \(d = 576\): \(8 \times 576/3 = 1{,}536\) exactly, and its FFN hidden size is 1,536. [Certain]
All three values match the published configs. [Certain]
Do not assume every SwiGLU model uses 8d/3. The ratio \(d_{\text{ff}}/d\) in real configs (from their config.json files on Hugging Face): [Certain]
| Model | \(d\) | \(d_{\text{ff}}\) | \(d_{\text{ff}}/d\) | FFN weights vs plain \(4d\) |
|---|---|---|---|---|
| MobileLLM-125M | 576 | 1,536 | 2.667 | 1.00× |
| Llama 2 7B | 4,096 | 11,008 | 2.688 | 1.01× |
| Llama 3 8B, Mistral 7B | 4,096 | 14,336 | 3.500 | 1.31× |
| Llama 3.2 1B | 2,048 | 8,192 | 4.000 | 1.50× |
| Qwen2.5 7B | 3,584 | 18,944 | 5.286 | 1.98× |
So \(8d/3\) is the starting point that makes a gated FFN cost the same as a classic one. Newer models often go wider, because grouped-query attention made attention cheaper and the FFN is where extra weights pay off best. [Likely] Use Figure 1 to reproduce these numbers.
How to read Figure 1
Each rectangle is one weight matrix; its length is proportional to its number of weights. With \(\alpha = 1\) and no rounding (\(m = 1\)), the two rows have the same total length: three thinner matrices replace two wider ones, formula (94.3). Click the presets: Llama 2 7B gives 11,008 and the bottom row is 0.8% longer than the top; Llama 3 8B (\(\alpha = 1.3\), \(m = 1{,}024\)) gives 14,336 and the bottom row is 31% longer. The readout also gives the FLOPs per token and the BF16 bytes that one decode step must read for this FFN.
SwiGLU at inference time
At equal parameters, SwiGLU has the same FLOPs and the same weight bytes as a plain FFN. But the way you run it matters. Four practical points.
1. Fuse the gate and up matrices
\(W_{\text{gate}}\) and \(W_{\text{up}}\) read the same input \(\mathbf{x}\). Store them side by side as one \(d \times 2d_{\text{ff}}\) matrix and do one matrix multiplication instead of two. Then compute \(\mathrm{SiLU}(g) \odot u\) in one small element-wise kernel, or inside the matmul’s epilogue (the last stage of the kernel, before it writes the result). Serving engines such as vLLM do exactly this, with a merged gate_up_proj and a fused “SiLU and multiply” kernel. [Likely] One big matmul uses the GPU better than two medium ones, and you launch fewer kernels (session 85).
- input: \(X \in \R^{B \times d}\); fused weight \(W_{gu} = [W_{\text{gate}} \,|\, W_{\text{up}}] \in \R^{d \times 2d_{\text{ff}}}\); \(W_{\text{down}} \in \R^{d_{\text{ff}} \times d}\)
- \(Z \leftarrow X W_{gu}\) // one matmul: reads \(2 d\, d_{\text{ff}}\) weights, \(4 B d\, d_{\text{ff}}\) FLOPs
- \(G, U \leftarrow\) first and second halves of the columns of \(Z\)
- \(H \leftarrow \mathrm{SiLU}(G) \odot U\) // element-wise; fused into step 2’s epilogue when possible, so \(G\) and \(U\) never reach HBM
- \(Y \leftarrow H W_{\text{down}}\) // reads \(d\, d_{\text{ff}}\) weights, \(2 B d\, d_{\text{ff}}\) FLOPs
- return \(Y\) // added to the residual stream by the caller
2. More activation memory in prefill
A plain FFN keeps one hidden tensor of shape \(B \times d_{\text{ff}}\). Without fusion, SwiGLU keeps two (\(G\) and \(U\)) before it can form \(H\). In decode this is tiny. In prefill it is not: for an 8,192-token chunk of Llama 3 8B in BF16, one \(B \times d_{\text{ff}}\) tensor is \(8{,}192 \times 14{,}336 \times 2 = 235\) MB, and \(G\) plus \(U\) is 470 MB. They are freed after each layer, so this is a peak, not a total. Fusion removes most of it. In training the cost is real, because \(G\) and \(U\) are kept for the backward pass (session 39).
3. Tensor parallelism works the same way
Megatron-style tensor parallelism splits the first FFN matrix by columns and the second by rows, so each GPU computes a slice of the hidden units and one all-reduce combines the outputs. [Certain] For SwiGLU, split \(W_{\text{gate}}\) and \(W_{\text{up}}\) along \(d_{\text{ff}}\) in the same way, so that GPU \(j\) holds matching gate and up columns. Then \(\mathrm{SiLU}(G) \odot U\) is local and there is still one all-reduce per FFN (session 48). This is one reason \(d_{\text{ff}}\) is rounded to a multiple of a large power of two: it must divide evenly by the number of GPUs.
4. SwiGLU activations are not sparse
ReLU sets every negative input to exactly 0. In ReLU FFNs a large fraction of hidden units are exactly zero for a given token, and systems such as PowerInfer use this to skip the matching rows of \(W_{\text{down}}\). SiLU is almost never exactly zero, so SwiGLU models have much less of this natural sparsity. Mirzadeh et al. (“ReLU Strikes Back”) argue for converting models back to ReLU-style activations to get it back, at a small quality cost. [Likely] So SwiGLU wins on quality per parameter, but closes one door for sparsity-based speed-ups. We return to this in the pruning sessions.
Figure 2 walks through one fused SwiGLU FFN of Llama 3 8B on an H100, kernel by kernel, and shows when it is limited by memory and when by compute.
How to read Figure 2
Step through the kernels. At \(B = 1\) (one user, decode) the whole layer reads 352 MB of weights and takes about 105 µs, while the math alone would take 0.36 µs: the layer is memory-bound by a factor of almost 300. Move \(B\) up: the time hardly changes until \(B\) is in the hundreds, because the weights are read once for all \(B\) tokens. Near \(B = 256\) to 512 the two bars cross, close to the ridge point of 295. At prefill sizes (4,096 or 8,192 tokens) turn off fusion: the unfused version writes and reads \(G\) and \(U\) through HBM. At \(B = 8{,}192\) that adds about 940 MB of traffic and about 0.21 ms per layer (2.92 ms becomes 3.13 ms, 7% slower). At \(B = 1\) the difference is invisible.
Cross-layer parameter sharing (ALBERT)
The idea
A standard transformer with \(L\) layers has \(L\) different sets of block weights. ALBERT (“A Lite BERT”, Lan et al., 2019) uses one set and applies it \(L\) times: the output of the block is fed back into the same block. Session 33 covered ALBERT as an encoder. Here we look at it as a tool for inference cost. The Universal Transformer (Dehghani et al.) had the same idea earlier: one block, applied repeatedly over depth, with an adaptive number of steps. [Certain]
More generally, split the \(L\) layer positions among \(L/k\) unique blocks, each used \(k\) times. Then:
- Read the symbols
- \(P_{\text{block}}\)
- weights of one transformer block: attention, FFN and norms
- \(k\)
- how many layer positions use each unique block; \(k = 1\) is no sharing, \(k = L\) is ALBERT’s all-shared setting
- \(P_{\text{emb}}\)
- the embedding table, plus the output head if it is not tied
- \(P_{\text{head}}\)
- the output projection to the vocabulary; it costs FLOPs even when it is tied to the input table
The second formula has no \(k\) in it. That is the whole lesson: every layer position still multiplies the token by a full block of weights. Sharing changes which weights are used, not how many multiplications happen.
There are two common orders for the repeats. Immediate (block-wise) repeat uses block 0 at positions 0, 1, then block 1 at positions 2, 3, and so on. Repeat-all-over runs the whole stack of unique blocks, then runs it again. ALBERT with \(k = L\) is both at once.
What ALBERT found
The ALBERT paper tested sharing on an ALBERT-base model (\(L = 12\), \(H = 768\)) with embedding size \(E = 128\). Average of five downstream tasks: [Certain]
| Sharing (ALBERT-base, \(E = 128\)) | Parameters | Avg. score |
|---|---|---|
| not shared | 89M | 81.6 |
| attention shared only | 64M | 81.7 |
| FFN shared only | 38M | 80.2 |
| all shared | 12M | 80.1 |
Two lessons. First, sharing the FFN is what hurts; sharing attention cost nothing here. The FFN is where the model stores most of its “knowledge”, so giving every layer the same FFN removes capacity. Second, the paper also tried groups of layers that share within the group, and found that smaller groups give better quality but more parameters. [Certain] It chose all-shared as its default.
Then ALBERT spent the saving on width. ALBERT-xxlarge has \(H = 4{,}096\) and 12 layers, all shared, with 235M parameters, about 70% of BERT-large’s 334M. It beat BERT-large clearly (average 88.7 vs 85.2). But it trained about 3 times slower: the paper lists its data throughput as 0.3× that of BERT-large. [Certain] With formula (94.5), one ALBERT-xxlarge block has 201,379,840 weights, and it is applied 12 times: about 2.42 billion weights touch every token, against 302 million in BERT-large. Eight times the compute, with fewer stored parameters.
One encoder layer of width \(H\) with FFN width \(4H\) has \(4(H^2 + H) + 2H \cdot 4H + 4H + H + 4H\) weights, as in session 33. For \(H = 1{,}024\): 12,596,224. For \(H = 4{,}096\): 201,379,840.
BERT-large: \(24 \times 12{,}596{,}224 = 302{,}309{,}376\) weights used per token, all different.
ALBERT-xxlarge: \(12 \times 201{,}379{,}840 = 2{,}416{,}558{,}080\) weights used per token, but only one block stored.
Ratio: \(2{,}416{,}558{,}080 / 302{,}309{,}376 = 7.99\). The paper’s measured throughput ratio was 3.17× (BERT-large faster), smaller than 8× because the measured time also includes attention over the sequence, the embeddings, and communication, which do not all grow the same way. [Likely]
What sharing saves, and what it does not
What it saves
- Memory for weights, and the size of the checkpoint file you ship. This matters most on phones and small GPUs.
- Room for the KV cache. On a serving GPU, every gigabyte not used by weights can hold more KV cache, so more requests can run at once. That raises throughput, even though each request is not faster.
- Communication in training: fewer distinct gradients to all-reduce. The ALBERT paper credits part of ALBERT-large’s 1.7× training speed-up over BERT-large to “less communication”. [Certain]
What it does not save
- FLOPs. Formula (94.5): same depth, same FLOPs.
- The KV cache. Each layer position computes its own keys and values from its own input, so a model with shared weights still stores one K and one V per layer per token. (Sharing the KV cache itself across layers is a different method, cross-layer attention (CLA); see session 97.)
- Usually, bytes moved per decode step. In decode, each layer position reads its weights from HBM. If the shared block is too big to stay in on-chip memory until it is used again, it is read again, and the traffic is the same as without sharing. One Llama 2 7B block is 202 million weights, 405 MB in BF16. The H100’s L2 cache is about 50 MB. [Likely] So for large models, sharing does not cut decode time.
The exception: weights that stay on chip
MobileLLM (Liu et al., 2024) studies models under one billion parameters for phones. Its block is small: MobileLLM-125M has 30 layers, \(d = 576\), and about 3.5 million weights per block, 7 MB in FP16. The paper notes that on-chip memory (SRAM) for computing is typically around 20 MB, enough for about one block. [Certain] It therefore repeats each block immediately: block \(j\) runs twice in a row, so the second run finds its weights already on chip and does not read them from DRAM again.
The model MobileLLM-LS-125M keeps the 125M parameters but doubles the depth to 60 layer positions. That doubles the FLOPs per token. Yet on an iPhone 13, execution went from 15.6 ms to 16.0 ms, a 2.6% overhead, while a 60-layer model with no sharing was 86% slower. [Certain] And quality went up: average zero-shot accuracy 46.3 → 47.0 for 125M, and 51.3 → 52.1 for 350M. [Certain] In the paper’s ablation, repeat-all-over was slightly more accurate (45.2 vs 45.0 for immediate repeat, baseline 44.6), but the authors chose immediate repeat for its cache behavior. [Certain]
So there are two ways to spend sharing. Same depth, fewer parameters (ALBERT-base): smaller memory, same compute, some quality loss. Same parameters, more depth (ALBERT-xxlarge’s extra width, MobileLLM-LS’s extra depth): better quality, more compute, and time that depends on whether the bytes moved go up.
- input: token vectors \(X\); unique blocks \(\text{blk}_0, \dots, \text{blk}_{U-1}\) with \(U = L/k\); pattern
- for \(i = 0, \dots, L - 1\):
- if pattern = immediate: \(j \leftarrow \lfloor i / k \rfloor\) // \(0, 0, 1, 1, \dots\): weights reused while still on chip
- else if pattern = repeat-all-over: \(j \leftarrow i \bmod U\) // \(0, 1, \dots, U-1, 0, 1, \dots\)
- \(X \leftarrow \text{blk}_j(X,\ \text{KV cache slot } i)\) // the KV cache is indexed by position \(i\), not by block \(j\)
- return \(X\)
- training: the gradient of \(\text{blk}_j\) is the sum of the gradients from all positions that used it (autograd does this)
Figure 3 lets you apply these patterns to real decoder models and see what changes.
How to read Figure 3
Choose Llama 2 7B on the L4 and switch from no sharing to immediate repeat with \(k = 2\). Stored weights fall from 13.48 GB to 7.00 GB, and the KV cache can hold 27,846 tokens instead of 15,493. The FLOPs bar does not move, and neither does the KV cache per token: every position still writes its own K and V. The readout also estimates the weight bytes read per decode step: for Llama, one block (over 400 MB) is far bigger than L2, so the bytes stay the same. Now choose MobileLLM-125M with immediate repeat: one block is 7 MB, so the repeat can be served from on-chip memory, and the bytes read per step drop. Repeat-all-over gives the same memory saving, but never reuses a block right away.
Llama 2 7B: 32 blocks of 202,383,360 weights (attention 67,108,864, SwiGLU FFN 135,266,304, two norms 8,192), plus input and output tables of \(32{,}000 \times 4{,}096 = 131{,}072{,}000\) each and a final norm. Total 6,738,415,616, which matches the published 6.74B.
KV cache per token: \(2 \times 32 \text{ layers} \times 32 \text{ heads} \times 128 \times 2 \text{ bytes} = 524{,}288\) bytes (0.5 MiB).
On an L4 with 90% of 24 GB usable (21.6 GB): weights take 13.48 GB, leaving 8.12 GB, which is 15,493 tokens of KV cache.
Share pairs of blocks (\(k = 2\), 16 unique blocks): \(16 \times 202{,}383{,}360 + 2 \times 131{,}072{,}000 + 4{,}096 = 3{,}500{,}281{,}856\) weights, 7.00 GB. Now 14.60 GB is free: 27,846 tokens, 1.8 times more. FLOPs per token stay \(2 \times 6.61 \times 10^9 = 13.2\) GFLOP. This model would of course need to be trained with sharing; you cannot just delete half the blocks of a trained Llama. [Certain]
Factorized and tied embeddings
Factorization (ALBERT)
ALBERT’s second idea targets the embedding table. A normal table is \(V \times d\). ALBERT stores a smaller \(V \times E\) table and one \(E \times d\) projection:
- Read the symbols
- \(V\)
- vocabulary size (30,000 for ALBERT, 128,256 for Llama 3)
- \(E\)
- the small embedding size (ALBERT uses \(E = 128\) for all model sizes)
- \(d\)
- the model width (\(H\) in the ALBERT paper)
The product of a \(V \times E\) and an \(E \times d\) matrix has rank at most \(E\), so this is a low-rank version of the full table. The argument: input embeddings only need context-free word meaning, which should need fewer dimensions than the hidden states. In ALBERT’s ablation (ALBERT-base, all-shared) \(E = 128\) was best (average 80.1, against 79.8 at \(E = 768\)), while without sharing larger \(E\) was slightly better (81.7 at 128, 82.3 at 768). [Certain]
For inference, look at the two ends of the model separately.
- Input side. A full table lookup costs zero FLOPs: it just copies one row. The factorized version adds a \(1 \times E\) by \(E \times d\) multiplication, \(2Ed\) FLOPs per token. It saves memory and adds a little compute.
- Output side (the LM head). The head maps the final hidden state to \(V\) logits. That is a real matrix multiplication, and with large vocabularies it is a big one. If the head is tied to a factorized table, the cost falls:
\(V = 128{,}256\), \(d = 4{,}096\). The head has \(525{,}336{,}576\) weights and costs \(2 \times 525{,}336{,}576 = 1.05\) GFLOP per token, about 7.0% of the model’s 15.0 GFLOP per token.
A factorized head with \(E = 512\): \(128{,}256 \times 512 + 512 \times 4{,}096 = 67{,}764{,}224\) weights and \(135{,}528{,}448\) FLOPs, 7.75 times less. But the logits are now produced through a 512-dimensional bottleneck: no matter what the hidden state is, the 128,256 logits must lie in a 512-dimensional subspace. A low-rank output layer limits how sharp and varied the next-token distributions can be (the “softmax bottleneck” argument). [Likely] No major open LLM uses a factorized head today. [Likely]
Tying: the cheap cousin that is used
Tied embeddings use the same \(V \times d\) matrix for the input table and the output head (Press and Wolf, 2017). This halves the vocabulary weights without a rank limit. It does not change FLOPs: the head still does \(2dV\) per token. It is common in small models, where the vocabulary is a large share of all weights: Llama 3.2 1B and Qwen2.5 0.5B tie their embeddings; Llama 3 8B and Qwen2.5 7B do not. [Certain] For Llama 3.2 1B the table is \(128{,}256 \times 2{,}048 = 262{,}668{,}288\) weights, 21% of the model’s 1.24B; an untied head would add the same amount again. MobileLLM reports that tying in its 125M model saved about 10 to 12% of the parameters for a 0.2-point drop in average accuracy, which it won back by spending the saved parameters on more layers. [Certain]
How to read Figure 4
Start at the right end of the slider (\(E = d\), no factorization) with Llama 3 8B: 1.05 billion vocabulary weights and 1.05 GFLOP per token in the head. Tick “tie”: the weights halve, the FLOPs do not move. Now slide \(E\) down to 512: weights and head FLOPs both drop by about 7.75 times, but the readout warns that the logits now have rank at most 512. Click ALBERT-xxlarge and set \(E = 128\): the input table shrinks from 122.9M to 4.4M weights, the setting ALBERT actually used.
Other kinds of sharing, briefly
- Sharing inside a Mixture of Experts. The survey’s section on efficient FFN design mentions MPOE, which decomposes each expert’s FFN matrix so that a large central tensor is shared by all experts and each expert keeps only small extra parts. [Likely] This is sharing applied across experts instead of across layers. MoE itself is the topic of session 95.
- Sharing keys and values. Multi-query and grouped-query attention share K and V across heads; cross-layer attention (CLA) shares them across adjacent layers. These cut the KV cache, which weight sharing does not (session 97).
- Sharing low-rank adapters. Serving many LoRA fine-tunes of one base model shares the big base weights across tasks (session 57).
In code
Everything above in one runnable, deterministic script (PyTorch 2.4 or later for nn.RMSNorm). It checks the \(8d/3\) sizing, shows that the fused SwiGLU equals three separate matrices, counts parameters and FLOPs with PyTorch’s own FLOP counter, and builds a factorized, tied head.
import torch, torch.nn as nn, torch.nn.functional as F from torch.utils.flop_counter import FlopCounterMode torch.manual_seed(0) # 1) hidden size of a gated FFN with the budget of a plain r*d FFN, formulas (94.3) and (94.4) def gated_hidden(d, r=4, mult=1.0, m=256): h = int(2 * r * d / 3) h = int(mult * h) return m * ((h + m - 1) // m) print(gated_hidden(4096), gated_hidden(4096, mult=1.3, m=1024)) # 11008 14336 # 2) SwiGLU (94.2) with gate and up fused into one matrix: one matmul instead of two class SwiGLU(nn.Module): def __init__(self, d, f): super().__init__() self.gate_up = nn.Linear(d, 2 * f, bias=False) # [W_gate | W_up] self.down = nn.Linear(f, d, bias=False) def forward(self, x): g, u = self.gate_up(x).chunk(2, dim=-1) return self.down(F.silu(g) * u) class PlainFFN(nn.Module): def __init__(self, d, f): super().__init__() self.up = nn.Linear(d, f, bias=False); self.down = nn.Linear(f, d, bias=False) def forward(self, x): return self.down(F.gelu(self.up(x))) def n_params(m): return sum(p.numel() for p in m.parameters()) # a shared tensor is counted once def flops(m, x): with FlopCounterMode(display=False) as fc: m(x) return fc.get_total_flops() d = 384 sw, pl = SwiGLU(d, gated_hidden(d)), PlainFFN(d, 4 * d) x = torch.randn(8, d) print(gated_hidden(d), n_params(sw), n_params(pl)) # 1024 1179648 1179648 print(flops(sw, x), flops(pl, x)) # 18874368 18874368 (= 2 * 8 tokens * weights) # the fused layer gives the same output as three separate matrices Wg, Wu = sw.gate_up.weight.chunk(2, dim=0) ref = (F.silu(x @ Wg.T) * (x @ Wu.T)) @ sw.down.weight.T print(torch.allclose(sw(x), ref, atol=1e-6)) # True # 3) cross-layer sharing, formula (94.5) and Algorithm 2 class Block(nn.Module): def __init__(self, d): super().__init__(); self.norm = nn.RMSNorm(d); self.ffn = SwiGLU(d, gated_hidden(d)) def forward(self, x): return x + self.ffn(self.norm(x)) class SharedStack(nn.Module): def __init__(self, d, L, k, pattern): super().__init__() self.blocks = nn.ModuleList(Block(d) for _ in range(L // k)) # L/k unique blocks if pattern == "immediate": self.order = [i // k for i in range(L)] # 0,0,1,1,... elif pattern == "all_over": self.order = [i % (L // k) for i in range(L)] # 0,1,...,0,1,... else: self.order = list(range(L)) def forward(self, x): for i in self.order: x = self.blocks[i](x) return x for pat, k in [("none", 1), ("immediate", 2), ("all_over", 2), ("immediate", 12)]: m = SharedStack(d, 12, k, pat) print(pat, k, n_params(m), flops(m, x), m.order[:6]) # none 1 14160384 226492416 [0, 1, 2, 3, 4, 5] # immediate 2 7080192 226492416 [0, 0, 1, 1, 2, 2] half the weights, same FLOPs # all_over 2 7080192 226492416 [0, 1, 2, 3, 4, 5] # immediate 12 1180032 226492416 [0, 0, 0, 0, 0, 0] ALBERT-style: one block # a shared weight receives the SUM of the gradients from every position that used it m = SharedStack(d, 12, 12, "immediate") m(x).pow(2).mean().backward() print(m.blocks[0].ffn.down.weight.grad.norm().item() > 0) # True: one tensor, 12 contributions # 4) factorized embedding (94.6) with a tied, factorized output head (94.7) V, E, D = 32000, 128, 1024 emb, proj = nn.Embedding(V, E), nn.Linear(E, D, bias=False) ids = torch.randint(0, V, (8,)) h = proj(emb(ids)) # [8, D]: lookup, then 2*E*D FLOPs per token logits = (h @ proj.weight) @ emb.weight.T # [8, V]: 2*D*E + 2*E*V FLOPs per token print(h.shape, logits.shape, V * E + E * D, V * D) # ... 4227072 32768000 print(torch.linalg.matrix_rank(proj.weight @ emb.weight.T).item()) # 128: the logit map has rank E
Note that flops is identical for all four stacks, and n_params falls with \(k\). That is formula (94.5) measured by PyTorch, not computed by us.
Where this is used
| Setting | Technique | Why it is used there |
|---|---|---|
| Open decoder LLMs (Llama, Mistral, Qwen) and the language half of most VLMs | SwiGLU FFN, no biases, \(d_{\text{ff}}\) from 8d/3 upward, rounded to a large multiple | Better quality at the same weights, FLOPs and bytes; free at inference [Certain] |
| Serving engines (vLLM and others) | Fused gate/up matmul, fused SiLU·multiply, gate and up split together under tensor parallelism | Fewer, larger kernels; less activation traffic in prefill [Likely] |
| Small text encoders (ALBERT family) for classification, retrieval and search | Cross-layer sharing and factorized embeddings | Small download and memory; compute is not reduced, so use the base or large sizes when latency matters |
| On-device LLMs (MobileLLM) | SwiGLU, tied embeddings, immediate block-wise repeat | Phones are limited by memory size and memory bandwidth; repeats served from on-chip memory are almost free [Certain] |
| Small LLMs with large vocabularies (Llama 3.2 1B, Qwen2.5 0.5B) | Tied input/output embeddings | The vocabulary table is about a fifth of the weights [Certain] |
| Video on a device: action recognition, key-moment detection or Video-RAG captioning with a small LLM next to a vision encoder | A small tied-embedding SwiGLU LLM, maybe with block repeat | The vision encoder and the KV cache of many frame tokens compete for the same memory; weights saved become frame tokens that fit [Speculation] |
| Serving many requests on one GPU | Fewer stored weights (sharing, or more usually quantization) | Freed memory holds more KV cache, so more requests run at once; per-request latency does not improve [Certain] |
Common mistakes
- “Fewer parameters, so faster.” Sharing keeps the FLOPs (formula 94.5). ALBERT-xxlarge has fewer parameters than BERT-large and trains about 3 times slower.
- Counting a SwiGLU FFN as two matrices. It has three: \(3\,d\,d_{\text{ff}}\) weights and \(6\,d\,d_{\text{ff}}\) FLOPs per token.
- Assuming \(d_{\text{ff}} = 8d/3\) in every SwiGLU model. Llama 3 8B uses 3.5d and Qwen2.5 7B uses 5.29d. Read
intermediate_sizefrom the config. - Splitting a fused gate/up matrix naively for tensor parallelism. Each GPU must hold matching gate and up columns.
- Expecting weight sharing to shrink the KV cache. Each layer position still writes its own K and V. Use GQA, MLA or cross-layer attention for that.
- Expecting weight sharing to cut decode time on a large GPU model. A shared block that does not fit on chip is read from HBM again at every use.
- Factorizing the output head to save FLOPs without checking quality. The logits become rank \(E\). Tying is the safer way to save vocabulary memory; it saves weights but not FLOPs.
- Deleting or merging layers of a trained model and calling it “sharing”. ALBERT and MobileLLM are trained with sharing. Forcing it on a trained model is a form of pruning and needs recovery training.
Interview questions
What is SwiGLU, and what does it cost compared with a GELU FFN?basic
Derive Llama 2 7B’s FFN size of 11,008.core
How would you implement SwiGLU efficiently for serving?core
ALBERT-large has 18M parameters, BERT-large 334M. Is ALBERT-large 18 times faster?core
What does cross-layer weight sharing save at inference, and what does it not save?core
When does weight sharing reduce decode latency? Give an example.advanced
Why did sharing the FFN hurt ALBERT more than sharing attention?advanced
Explain ALBERT’s factorized embedding. Would you use it for the LM head of a modern LLM?core
A team wants to serve more concurrent users on one 24 GB GPU with a 7B model. They propose retraining it with pairs of shared layers. Evaluate.advanced
Does SwiGLU help or hurt activation-sparsity methods?advanced
Exercises
- A plain FFN model has \(d = 2{,}048\) and \(d_{\text{ff}} = 4d\). Switch to SwiGLU at equal weights with Llama-style rounding to a multiple of 256. What is \(d_{\text{ff}}\)?
- Llama 2 7B is retrained with repeat-all-over sharing, \(k = 4\). How many weights are stored? How many FLOPs per token in the forward pass (ignore attention over the context)?
- For Llama 3 8B (\(V = 128{,}256\), \(d = 4{,}096\)), compute the weights and FLOPs per token of a factorized, tied head with \(E = 512\), and the reduction factor.
- One Llama 3 8B FFN layer in BF16 is decoded at \(B = 1\) on an H100 (3.35 TB/s). Estimate its time and the time of all 32 FFN layers. Would immediate sharing with \(k = 2\) change it?
- Using the per-layer counts in the worked example, how many weights does each token touch in ALBERT-xxlarge versus BERT-large (encoder layers only)?
- Would MobileLLM’s immediate repeat help a Llama 2 7B on an H100? Compare one block’s size with the L2 cache.
Answerscheck
Key takeaways
- Separate three costs (94.1): stored bytes, FLOPs per token, and bytes moved per step. Decode is memory-bound below about 295 tokens per step on an H100.
- SwiGLU (94.2) gives lower loss at the same weights, FLOPs and bytes. Equal budget gives \(d_{\text{ff}} = 8d/3\) (94.3), then designers scale and round (94.4): 11,008, 14,336, 28,672.
- Serve SwiGLU with a fused gate/up matmul and a fused SiLU·multiply; split gate and up together under tensor parallelism. Its activations are not sparse.
- Cross-layer sharing (94.5) divides stored block weights by \(k\) but leaves FLOPs and the KV cache unchanged. It helps latency only when the shared block stays on chip (MobileLLM).
- Factorized embeddings (94.6) save memory; a factorized tied head also saves FLOPs (94.7) but caps the logit rank at \(E\). Tying is the common, safe version in small models.
Key formulas
References for this session
- Shazeer (2020). GLU Variants Improve Transformer
- Lan et al. (2019). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Shazeer (2020). GLU Variants Improve Transformer
- Lan et al. (2019). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Liu, Zhao, Iandola, Lai, Tian, Fedorov, Xiong, Chang, Shi, Krishnamoorthi, Lai, Chandra (2024). MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- Press & Wolf (2017). Using the Output Embedding to Improve Language Models
- Dehghani, Gouws, Vinyals, Uszkoreit, Kaiser (2019). Universal Transformers
- Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Jiang et al. (2023). Mistral 7B
- Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Song et al. (2023). PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
- Mirzadeh, Alizadeh, Mehta, Del Mundo, Tuzel, Samei, Rastegari, Farajtabar (2023). ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
- Brandon, Mishra, Nrusimha, Panda, Ragan-Kelley (2024). Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
- Gao, Liu, Zhao, Lu, Wen (2022). Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention