Structured pruning: LLM-Pruner, Sheared LLaMA, SliceGPT, Minitron
Unstructured zeros save little on a GPU. Structured pruning removes whole heads, MLP channels, hidden channels, layers or experts, so the result is a smaller dense model that is faster on any hardware. This session shows how to find the weights that must be removed together, how LLM-Pruner, Sheared LLaMA, SliceGPT and Minitron choose what to remove, and how training with distillation brings the quality back.
What you will learn
- Which structures a Llama block has (heads, MLP channels, hidden channels, layers) and how many parameters each one costs
- How coupled structures are found with a dependency graph, and why a head or a channel must be removed from several matrices at once
- LLM-Pruner: Taylor importance of a group, one-shot removal, and a short LoRA recovery
- Sheared LLaMA: learned masks with the hard concrete distribution, a Lagrangian that forces a target shape, and dynamic batch loading during continued pretraining
- SliceGPT: computational invariance of RMSNorm networks, a PCA rotation per layer, and deleting whole rows and columns
- Minitron: activation-based importance, width versus depth pruning, and recovery by logit distillation on about 1% of the original tokens
- Expert pruning in MoE models, and how to read the memory and accuracy numbers these papers report
Why this matters
In session 114 we set half of the weights to zero and kept the quality. But the matrices kept their shape. On a GPU, a \(4096 \times 4096\) matrix with 50% random zeros is still a \(4096 \times 4096\) matrix: the same memory (unless you pay for an index format) and almost the same time. Only the special 2:4 pattern gives some speed, and only on some GPUs.
Structured pruning removes whole pieces instead: an attention head, a channel of the MLP, a channel of the hidden state, a whole layer, or a whole expert. Each removal deletes complete rows and columns, so the matrices become smaller dense matrices. A 7B model becomes, for example, a normal 2.7B model. It runs faster with the same kernels, on any GPU or CPU, and needs no sparse format. The price: removing big pieces hurts much more than removing single weights, so structured pruning almost always needs some training afterwards to recover.
This idea is now a standard way to build small models. Sheared LLaMA cut LLaMA2-7B down to 2.7B and 1.3B models with 50B tokens of continued pretraining [Certain]. NVIDIA's Minitron models (8B and 4B) were cut from Nemotron-4 15B and trained with distillation on 94B tokens, while the 15B model was trained on 8T tokens [Certain]. Later Minitron work did the same to Llama 3.1 8B (to 4B) and Mistral NeMo 12B (to 8B) [Certain].
We will meet four methods that answer the same three questions in different ways: what can be removed (the structure), which pieces to remove (the importance score), and how to recover the quality (fine-tuning, continued pretraining or distillation).
| Method | What it removes | How it chooses | How it recovers |
|---|---|---|---|
| LLM-Pruner (2023) | coupled groups: heads, MLP channels | gradient (Taylor) importance on 10 samples | LoRA on about 50K instruction samples |
| Sheared LLaMA (2023) | layers, heads, MLP channels, hidden channels, down to a fixed target shape | learned masks, trained with a shape constraint | continued pretraining on 50B tokens, with dynamic data mixing |
| SliceGPT (2024) | hidden channels, after a rotation | PCA of the hidden states | none needed; optional short fine-tuning |
| Minitron (2024) | heads, MLP channels, hidden channels, or layers | activation size on 1024 samples | logit distillation from the original model |
What can be removed in a Llama block
Recall the shape of one Llama-style block (session 22). The hidden state (the residual stream) has \(d\) numbers per token. Attention has \(H\) heads of size \(d_h\). The SwiGLU MLP has an intermediate width \(m\) and three matrices: gate, up and down. There are \(L\) blocks, an embedding table and an output head over a vocabulary of size \(V\). With full multi-head attention, the parameter count is:
- Read the symbols
- \(d\)
- hidden size (width of the residual stream); 4096 in LLaMA2-7B
- \(H, d_h\)
- number of attention heads and the size of one head; 32 and 128 in LLaMA2-7B
- \(m\)
- intermediate width of the MLP; 11008 in LLaMA2-7B
- \(L, V\)
- number of blocks (32) and vocabulary size (32000)
- \(4\,d\,Hd_h\)
- the four attention matrices \(W_Q, W_K, W_V\) (each \(d \to Hd_h\)) and \(W_O\) (\(Hd_h \to d\))
Each structure we can remove has a price in parameters. Read it from (115.1):
A head owns \(d_h\) rows of \(W_Q\), \(W_K\), \(W_V\) and \(d_h\) columns of \(W_O\). An MLP channel owns one row of gate, one row of up and one column of down. A hidden channel is special: the residual stream passes through every block, so one hidden channel touches every matrix in every layer, both norms, the embedding table and the output head. A layer (depth pruning) removes everything inside one block.
With \(d = 4096\), \(H = 32\), \(d_h = 128\), \(m = 11008\), \(L = 32\), \(V = 32000\), formula (115.1) gives \(N = 6{,}738{,}415{,}616\), the official 6.74B. Per block: attention \(4 \cdot 4096^2 = 67{,}108{,}864\), MLP \(3 \cdot 4096 \cdot 11008 = 135{,}266{,}304\). So MLPs are 64.2% of the model, attention 31.9% and embeddings plus head 3.9%.
Prices from (115.2): one head costs \(4 \cdot 4096 \cdot 128 = 2{,}097{,}152\) weights. One MLP channel costs \(3 \cdot 4096 = 12{,}288\). One hidden channel costs \(32 \cdot (4 \cdot 4096 + 3 \cdot 11008 + 2) + 2 \cdot 32000 + 1 = 1{,}645{,}121\). Cutting the hidden size from 4096 to 3072 alone removes exactly 25% of all weights, because every matrix has \(d\) on one side.
The Sheared-LLaMA-2.7B shape is \(L = 32\), \(d = 2560\), \(H = 20\), \(m = 6912\). Formula (115.1) gives 2,701,560,320 parameters: 40.1% of the 7B model. The 1.3B shape (\(L = 24\), \(d = 2048\), \(H = 16\), \(m = 5504\)) gives 1,345,423,360: 20.0%.
How to read Figure 1
Start from LLaMA2-7B and move one slider at a time. Removing heads only shrinks the attention part. Removing MLP channels only shrinks the MLP part, which is the biggest. Lowering \(d\) shrinks all three parts at once, because every matrix has \(d\) on one side. Lowering \(L\) removes whole blocks but leaves the embeddings. The readout also shows the price of one more head, channel or layer at the current shape, from formula (115.2).
Which axis should you prune? The papers agree on one thing: thin layers are usually better than fewer layers at the same size. Minitron found that width pruning (heads, MLP channels, hidden channels) beat depth pruning for models up to 15B after retraining [Certain]. But removing layers gives the most speed for small batches, because each layer is a sequential step, so depth pruning is still used (ShortGPT removes layers whose output is almost equal to their input) [Certain].
Coupled structures: what must go together
You cannot remove one row of \(W_Q\) and keep the rest of the model as it is. If output channel \(j\) of a matrix disappears, every matrix that reads channel \(j\) must lose the matching input column, otherwise the shapes do not fit. Weights connected like this form a coupled group: they are kept or removed together.
LLM-Pruner finds these groups automatically with a dependency graph. Each neuron (one input or output channel of a layer) is a node. Two rules decide when a neuron \(N_j\) must follow neuron \(N_i\):
- Read the symbols
- \(\operatorname{Out}(N_i)\), \(\operatorname{In}(N_j)\)
- the neurons that \(N_i\) sends to, and the neurons that send to \(N_j\)
- \(\operatorname{Deg}^{-}\), \(\operatorname{Deg}^{+}\)
- in-degree and out-degree: how many neurons send to, or receive from, this neuron
In words: if a neuron gets its only input from \(N_i\), it is useless without \(N_i\); if a neuron sends its only output to \(N_j\), it is useless without \(N_j\). Start from one neuron, apply both rules until nothing new is added, and you get its group. In a Llama block this gives three kinds of group: an attention head (rows of \(W_Q, W_K, W_V\) and columns of \(W_O\)), an MLP channel (one row of gate and up, one column of down), and a hidden channel (a column or row in every matrix of every block, because of the residual connections). Click through them in Figure 2.
How to read Figure 2
A head is a band of rows in \(W_Q, W_K, W_V\) and the matching band of columns in \(W_O\). An MLP channel is one row in gate and up and one column in down. A hidden channel is a column of every matrix that reads the residual stream (\(W_Q, W_K, W_V\), gate, up), a row of every matrix that writes to it (\(W_O\), down), and one entry of each norm. The readout gives the size of the group in this toy block and in LLaMA2-7B. The hidden channel is the most expensive group by far, and it is shared by all layers.
Scoring a group: LLM-Pruner
Which groups are safe to remove? LLM-Pruner asks: how much would the loss change if this group were set to zero? Write the group's weights as one vector \(\mathbf{w}\). A second-order Taylor expansion of the loss around the current weights gives:
- Read the symbols
- \(\mathbf{g}\)
- the gradient of the loss with respect to \(\mathbf{w}\), computed by one backward pass on a few calibration texts
- \(\mathbf{H}\)
- the Hessian (second derivatives); LLM-Pruner approximates its diagonal by the Fisher information, the sum of squared per-sample gradients
- \(I_{\text{group}}\)
- the importance of a coupled group: the sum of the scores of its parts (the paper also tried product, max and "last only"; sum worked best)
In session 113 the gradient term vanished because a trained model sits at a minimum of its training loss. LLM-Pruner keeps it on purpose: the calibration texts are not the training data, so the gradient there is not zero. The paper uses only 10 samples from BookCorpus of 128 tokens each [Certain]. The first-order part \(|\mathbf{g}^\top\mathbf{w}|\) alone is the "vector-wise" score; adding the diagonal second-order term per element gives the "element-wise" score.
Two MLP channels, each shown with three of its weights. Channel A: \(\mathbf{w} = (0.9, -0.8, 0.7)\), \(\mathbf{g} = (0.02, 0.01, 0.01)\). Channel B: \(\mathbf{w} = (0.2, -0.3, 0.1)\), \(\mathbf{g} = (0.3, -0.2, 0.4)\).
By magnitude, A is bigger: \(\lVert \mathbf{w}_A \rVert = 1.393\), \(\lVert \mathbf{w}_B \rVert = 0.374\). By first-order Taylor, \(|\mathbf{g}_A^\top \mathbf{w}_A| = |0.018 - 0.008 + 0.007| = 0.017\) and \(|\mathbf{g}_B^\top \mathbf{w}_B| = |0.06 + 0.06 + 0.04| = 0.16\). The loss barely notices channel A, so Taylor removes A, even though its weights are large.
- input: model, about 10 calibration texts, target ratio (for example 20% of parameters)
- build the dependency graph and collect all coupled groups (115.3)
- run forward and backward on the calibration texts; keep \(\mathbf{g}\) and the squared per-sample gradients
- score every group with (115.4), summing over its parts
- for each block in the pruned range (the paper keeps the first 3 blocks and the last block):
- remove the lowest-scoring heads and MLP channels until the target ratio is reached // physically slice the matrices
- recover: LoRA (rank 8) on about 50K Alpaca samples, 2 epochs // about 3 hours on one GPU
What did it achieve? On LLaMA-7B with 20% of the parameters removed (6.74B to 5.42B), the average zero-shot accuracy went from 63.25 to 56.82 without tuning and to 60.07 after LoRA: 94.97% of the original [Certain]. WikiText2 perplexity went from 12.62 to 17.37 after tuning [Certain] (these perplexities use the paper's own short-context setup, so they are not comparable with other papers). That is a real drop for a 20% cut, and it is the main lesson of LLM-Pruner: a cheap one-shot cut with a tiny recovery works at small ratios, but at 50% the quality falls much more. Bigger cuts need much more training.
Prune to a target shape: Sheared LLaMA
LLM-Pruner removes the least important groups and accepts whatever shape results. The layers then end up with different widths, which is awkward for kernels and for comparing with other models. Sheared LLaMA does the opposite: it fixes the target shape first (for example the shape of Pythia-1.4B or INCITE-3B, so that the results can be compared fairly) and then learns which parts of the 7B model to keep.
Learned masks with the hard concrete distribution
Every prunable structure gets a mask value \(z\): one per layer (\(z^{\text{layer}}\)), per head (\(z^{\text{head}}\)), per MLP channel (\(z^{\text{int}}\)) and per hidden channel (\(z^{\text{hidden}}\)). The output of a structure is multiplied by its \(z\). We want each \(z\) to be exactly 0 or exactly 1 at the end, but we also want gradients to train them. The hard concrete distribution (from L0 regularization) gives both. Each mask has one learnable number \(\log\alpha\). At each step we sample:
- Read the symbols
- \(\log\alpha\)
- the learned parameter of one mask; large means "keep", very negative means "remove"
- \(\beta\)
- temperature; Sheared LLaMA uses 0.83
- \(\gamma, \zeta\)
- stretch limits \(-0.1\) and \(1.1\): the sample is stretched to \((-0.1, 1.1)\) and then clipped to \([0, 1]\), so the clipped ends become exact 0 and exact 1 with real probability
Because of the clipping, \(z = 0\) exactly when the stretched sample is below 0, which means \(s \le -\gamma/(\zeta-\gamma) = 1/12\), and \(z = 1\) exactly when \(s \ge (1-\gamma)/(\zeta-\gamma) = 11/12\). With these constants the two probabilities have a simple closed form: \(P(z = 0) = \sigma(-\beta \ln 11 - \log\alpha)\) and \(P(z = 1) = 1 - \sigma(\beta \ln 11 - \log\alpha)\). In between, \(z\) is a smooth function of \(\log\alpha\), so gradients flow. Figure 3 shows the shape.
How to read Figure 3
At \(\log\alpha = 0\) the mask is undecided: exact 0 and exact 1 each have probability 0.120, and most samples are in between. Move the slider right: the mass moves to \(z = 1\) (the structure is kept). Move it left: it moves to \(z = 0\). Training pushes every \(\log\alpha\) to one side. The expected number of nonzero masks, \(\sum (1 - P(z=0))\), is smooth in \(\log\alpha\), which is what the shape constraint below uses. The readout compares the exact formula with the samples.
Forcing the target shape
Now we need the masks to end at exactly the target size, for example \(H_T = 20\) heads per layer. Sheared LLaMA adds a constraint with Lagrange multipliers \(\lambda, \phi\) for every kind of structure:
The model weights \(\theta\) and the masks go down the gradient of the language-model loss plus the penalty. The multipliers \(\lambda, \phi\) go up the gradient: as long as the model has too many heads, the penalty grows, until the masks obey. This is a min-max game, the same idea as constrained optimization in any textbook. In the paper, the masks, \(\lambda\) and \(\phi\) all use learning rate 1.0, and the pruning stage uses only 0.4B tokens [Certain]. At the end, the structures with \(z = 0\) are deleted and the masks are folded into the weights.
Suppose one layer still has \(\sum z^{\text{head}} = 24\) heads but the target is \(H_T = 20\), with \(\lambda = -0.5\) and \(\phi = 1.0\). The penalty is \(-0.5 \cdot 4 + 1.0 \cdot 16 = 14\). Its gradient with respect to each \(z\) is \(\lambda + 2\phi \cdot 4 = 7.5\), positive, so gradient descent pushes all head masks down. The multipliers keep changing until the count reaches 20 and the penalty is 0.
Continued pretraining with dynamic batch loading
After pruning, the 2.7B model is trained further on 50B tokens of RedPajama (seven domains: CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, StackExchange) [Certain]. The paper noticed that pruning hurts some domains more than others, so a fixed data mixture wastes tokens on domains that already recovered. Dynamic batch loading changes the mixture during training. Every \(m\) steps, compare the loss on each domain \(i\) with a reference loss:
- Read the symbols
- \(\ell_t[i]\)
- the current validation loss of the pruned model on domain \(i\)
- \(\ell_{\text{ref}}[i]\)
- the loss we hope to reach; by default predicted by a scaling-law fit for a model of the target size
- \(w_t\)
- the fraction of the next batches drawn from each domain
Domains that are far behind get more data; domains that are already at the reference keep their share. Example with three domains, \(w = (0.5, 0.3, 0.2)\) and gaps \(\Delta = (0, 0.2, 0.05)\): \(\alpha = (0.5, 0.366, 0.210)\), and after normalizing \(w = (0.464, 0.340, 0.195)\). The domain with the largest gap gained 4 points.
- input: source model (LLaMA2-7B), target shape \((L_T, d_T, H_T, m_T)\), pretraining data split into domains
- add masks \(z^{\text{layer}}, z^{\text{hidden}}, z^{\text{head}}, z^{\text{int}}\), each with a \(\log\alpha\); set \(\lambda, \phi = 0\)
- repeat for 0.4B tokens: // pruning stage
- sample all \(z\) with (115.5); forward with masked structures; loss = LM loss + penalties (115.6)
- gradient step down for \(\theta\) and \(\log\alpha\); gradient step up for \(\lambda, \phi\); update the domain mix with (115.7)
- keep the structures with the largest masks so the shape is exactly the target; delete the rest
- continued pretraining on 50B tokens, still updating the domain mix with (115.7)
The results show why this is worth it. In the paper's Table 2, Sheared-LLaMA-2.7B (50B tokens of training after pruning) has an average downstream accuracy of 56.7, against 55.7 for OpenLLaMA-3B-v2 trained on 1T tokens and 54.7 for INCITE-Base-3B trained on 800B tokens [Certain]. The 50B tokens are 5% of OpenLLaMA's budget; counting compute, the paper claims about 3% of the cost of training such models from scratch [Certain]. Figure 5 plots these numbers. The pruning stage itself is about 5× slower per token than normal training, because of the masks, but it is short [Certain].
Rotate, then delete: SliceGPT
Deleting hidden channels directly is hard: in a trained model, every channel carries a bit of everything. SliceGPT first rotates the hidden space so that the information is packed into the first channels, and then deletes the last channels. It needs no gradients and no training.
Computational invariance
The trick is a property of RMSNorm. For any orthogonal matrix \(Q\) (\(Q^\top Q = I\)), multiplying a vector by \(Q\) does not change its length, so
First, fold the learned gain of each RMSNorm into the next matrix (LayerNorm models like OPT are first turned into RMSNorm models, by also folding the mean subtraction into nearby matrices). Then choose an orthogonal \(Q\) and transform the weights: the embedding becomes \(W_{\text{embd}} Q\), every matrix that reads the residual stream becomes \(Q^\top W_{\text{in}}\), every matrix that writes to it becomes \(W_{\text{out}} Q\), and the output head becomes \(Q^\top W_{\text{head}}\). The residual stream is now \(\mathbf{x}Q\) everywhere, and by (115.8) the network computes exactly the same output. Nothing changed yet; we only chose a new coordinate system.
Choosing Q with PCA, then slicing
Run a calibration set through the model and collect the hidden states \(X_\ell\) that enter block \(\ell\). The best rotation, in the sense of keeping the most energy in the first \(k\) coordinates, is given by principal component analysis:
- Read the symbols
- \(X_{\ell,i}\)
- the hidden states (one row per token) of calibration sequence \(i\) entering block \(\ell\)
- \(Q_\ell, \Lambda_\ell\)
- eigenvectors and eigenvalues of \(C_\ell\), sorted from the largest eigenvalue down
- \(k\)
- the new hidden width; slicing 25% of \(d = 4096\) gives \(k = 3072\)
Deleting coordinates \(k+1 \ldots d\) after the rotation means: delete the last rows of every \(Q^\top W_{\text{in}}\) and the last columns of every \(W_{\text{out}} Q\). Every matrix becomes a smaller dense matrix. The energy lost is exactly the sum of the deleted eigenvalues, \(\sum_{j \gt k} \lambda_j\). Figure 4 shows this in two dimensions.
How to read Figure 4
At \(\theta = 0\) we keep the original first channel; the cloud is tilted, so a large part of the energy is lost. Turn the slider: the kept energy goes up and down. Press "Use the PCA angle": the kept axis lines up with the long direction of the cloud, and the kept energy is the largest eigenvalue divided by the sum of both, the maximum possible. No other angle does better, which is why SliceGPT uses PCA.
A different rotation per layer
Hidden states change from layer to layer, so one \(Q\) for the whole model is not the best. SliceGPT uses a different \(Q_\ell\) for each block. But the residual connection then carries \(\mathbf{x}Q_{\ell-1}\) into a block that expects \(\mathbf{x}Q_\ell\). The fix is a small extra matrix on the residual path:
After slicing, this adapter is a \(k \times k\) matrix that must be stored and multiplied. So slicing 30% of the hidden size does not remove 30% of the parameters. For LLaMA2-70B (68.98B), 30% slicing leaves about 51.6B parameters, not 48.3B [Certain for 51.6B, from the paper].
- input: model with RMSNorm (convert LayerNorm first), calibration set (the paper: 1024 sequences of 2048 tokens), slicing fraction
- fold every RMSNorm gain into the matrices that follow it
- for \(\ell = 1 \ldots L\):
- run the calibration set up to block \(\ell\); \(C_\ell \leftarrow \sum_i X_{\ell,i}^\top X_{\ell,i}\) // in float64
- \(Q_\ell \leftarrow\) eigenvectors of \(C_\ell\), largest first (115.9)
- \(W_{\text{in}} \leftarrow (Q_\ell^\top W_{\text{in}})[{:}k, :]\), \(W_{\text{out}} \leftarrow (W_{\text{out}} Q_\ell)[:, {:}k]\), residual adapter \(\leftarrow (Q_{\ell-1}^\top Q_\ell)[{:}k, {:}k]\) (115.10)
- slice the embedding and the head with the first and last \(Q\)
How well does it work? On LLaMA2-7B with 1024 calibration sequences from WikiText2, WikiText2 perplexity goes from 5.47 (dense) to 6.64 at 20% slicing, 7.24 at 25% and 8.12 at 30%; SparseGPT with 2:4 sparsity gives 8.69 [Certain]. For LLaMA2-70B the paper reports running on 3 instead of 4 A100 40GB GPUs, with total compute 66% of the dense model (330 vs 500 GPU-milliseconds per token in its Hugging Face measurement) [Certain]. On zero-shot tasks, the paper reports keeping about 99% of the accuracy for LLaMA2-70B and OPT-66B at 25% slicing, and 90% for the small Phi-2 (with a short fine-tune for LLaMA2 and Phi-2) [Certain]. Big models have more redundant directions, so they slice better. [Likely]
Prune, then distill: Minitron
Minitron (NVIDIA) asked a practical question: if we already trained a big model on 8T tokens, can we get the smaller sizes of the family by pruning and a little retraining, instead of training each size from scratch? Its answer: yes, if the retraining is distillation from the big model (session 59).
Importance from activations only
Minitron scores structures with forward passes only, no gradients, on 1024 random training samples [Certain]. For head \(i\), MLP neuron \(i\) and hidden (embedding) channel \(i\):
The sums run over the batch \(B\) and the sequence \(S\). The paper tested several ways to combine them and chose: take the mean over the sequence, then the L2 norm over the batch [Certain]. A structure whose output is large on real data is important; one whose output is always tiny is not. For layers (depth), it uses either the perplexity change when a layer is removed, or block importance:
Here \(\mathbf{x}_{i,t}\) is the hidden state of token \(t\) entering layer \(i\). If a layer's output is almost parallel to its input (cosine near 1), the layer changes little and \(\mathrm{BI}\) is near 0: a candidate for removal. This is the score of ShortGPT.
Retrain with distillation
After pruning (for example 15B to 8B: hidden size 6144 to 4096, MLP width 24576 to 16384, same 32 layers and 48 heads), the student is trained to match the teacher's next-token distribution:
The teacher gives a full distribution over the vocabulary at every position, not one correct token, so each training token carries much more signal. Minitron found forward KL better than reverse KL, MSE or cosine on the logits, with temperature 1.0 [Certain]. When many layers are removed, it also adds losses on intermediate hidden states; for the 8B and 4B models, which kept all 32 layers, logits alone were used [Certain].
- input: trained teacher, target size, 1024 calibration samples
- forward the calibration samples; score heads, neurons and hidden channels with (115.11) (and layers with (115.12))
- list candidate shapes near the target size; for each, keep the top-scoring structures and slice the weights
- train each candidate briefly (about 1.8B tokens) and keep the best one // rankings change after a little training
- distill the chosen student from the teacher with (115.13) on about 94B tokens
- to get a still smaller model, prune this result again (8B, then 4B)
Results: Minitron 8B reached 63.8 on MMLU (5-shot), against 54.7 for Nemotron-3 8B trained from scratch and 66.6 for the 15B teacher; Minitron 4B reached 58.6 [Certain]. Each smaller model used 94B tokens, about 1.2% of the teacher's 8T. The paper reports up to 40× fewer training tokens per model than training from scratch and 1.8× lower compute for the whole 15B + 8B + 4B family [Certain]. The follow-up paper added teacher correction: when the original training data is not available, first fine-tune the teacher briefly on the distillation data, so that it is not distilling from a distribution it has never seen [Certain].
Pruning experts in MoE models
A mixture-of-experts layer (session 29) has \(n\) expert MLPs and a router that sends each token to a few of them. All experts must be in memory, even though each token uses only a few. So a whole expert is a natural structure to remove: it is a large group, and removing it cuts memory directly.
Lu et al. prune Mixtral 8×7B one layer at a time. Run calibration data (128 sequences of 2048 tokens from C4) through the model, store the input \(\mathbf{x}\) and output \(F(\mathbf{x})\) of each MoE layer, and try every subset \(\mathcal{C}\) of \(r\) experts:
\(F'(\mathbf{x}, \mathcal{C})\) is the layer output when the router can only choose experts in \(\mathcal{C}\). With \(n = 8\) there are only \(\binom{8}{6} = 28\) subsets for \(r = 6\) and \(\binom{8}{4} = 70\) for \(r = 4\), so trying them all is cheap: about 30 and 90 minutes [Certain]. Memory falls from 89,926 MB to 68,383 MB (76%) with 6 experts and to 46,879 MB (52%) with 4, so the model fits on one 80 GB GPU instead of two [Certain]. The average zero-shot accuracy over 8 tasks went from 67.58 to 64.22 with 6 experts and 59.57 with 4 [Certain]. The same paper also skips the second expert for a token when its router weight is much smaller than the first one, for a 1.2× to 1.3× speed-up [Certain].
Code: score, slice, distill
The first program trains a tiny Llama-style model on a fixed random Markov chain, scores MLP channels and heads three ways (random, LLM-Pruner Taylor, Minitron activation), physically slices the matrices to half the MLP width and 6 of 8 heads, and then recovers with logit distillation. It runs in about a minute on a laptop CPU and is deterministic.
import copy, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) V, D, H, M, L, T = 32, 64, 8, 176, 2, 32 # vocab, hidden, heads, MLP width, layers, seq len # data: a fixed random 2nd-order Markov chain, so there is something to learn P = torch.softmax(3.0 * torch.randn(V, V, V), -1) def batch(n, g): x = torch.randint(V, (n, 2), generator=g) for _ in range(T - 1): x = torch.cat([x, torch.multinomial(P[x[:, -2], x[:, -1]], 1, generator=g)], 1) return x[:, :-1], x[:, 1:] class RMSNorm(nn.Module): def __init__(s, d): super().__init__(); s.g = nn.Parameter(torch.ones(d)) def forward(s, x): return s.g * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + 1e-6) class Block(nn.Module): def __init__(s): super().__init__(); s.h, s.dh = H, D // H s.n1, s.n2 = RMSNorm(D), RMSNorm(D) s.q, s.k, s.v, s.o = (nn.Linear(D, D, bias=False) for _ in range(4)) s.gate, s.up = nn.Linear(D, M, bias=False), nn.Linear(D, M, bias=False) s.down = nn.Linear(M, D, bias=False) def forward(s, x): B, t, _ = x.shape; a = s.n1(x) sp = lambda z: z.view(B, t, s.h, s.dh).transpose(1, 2) # s.h shrinks after pruning y = F.scaled_dot_product_attention(sp(s.q(a)), sp(s.k(a)), sp(s.v(a)), is_causal=True) s.head_out = y # (B, h, t, dh), kept for scoring x = x + s.o(y.transpose(1, 2).reshape(B, t, s.h * s.dh)) a = s.n2(x); s.mlp_act = F.silu(s.gate(a)) * s.up(a) # (B, t, M), kept for scoring return x + s.down(s.mlp_act) class LM(nn.Module): def __init__(s): super().__init__(); s.emb = nn.Embedding(V, D) s.blocks = nn.ModuleList(Block() for _ in range(L)); s.nf = RMSNorm(D) s.head = nn.Linear(D, V, bias=False) def forward(s, x): h = s.emb(x) for b in s.blocks: h = b(h) return s.head(s.nf(h)) def loss_of(m, x, y): return F.cross_entropy(m(x).flatten(0, 1), y.flatten()) def nparams(m): return sum(p.numel() for p in m.parameters()) # 1) train the dense "teacher" dense, g = LM(), torch.Generator().manual_seed(1) opt = torch.optim.AdamW(dense.parameters(), lr=3e-3) for step in range(800): x, y = batch(64, g); l = loss_of(dense, x, y) opt.zero_grad(); l.backward(); opt.step() xe, ye = batch(256, torch.Generator().manual_seed(2)) # held-out evaluation set xc, yc = batch(32, torch.Generator().manual_seed(3)) # small calibration set with torch.no_grad(): print("dense loss %.3f params %d" % (loss_of(dense, xe, ye).item(), nparams(dense))) # 2) importance of each coupled group def scores(m, kind): m.zero_grad(); l = loss_of(m, xc, yc); l.backward() out = [] for b in m.blocks: if kind == "taylor": # LLM-Pruner, first-order (115.4): |sum over the group of g * w| mlp = (b.gate.weight.grad * b.gate.weight).sum(1) + (b.up.weight.grad * b.up.weight).sum(1) \ + (b.down.weight.grad * b.down.weight).sum(0) hd = sum((w.weight.grad * w.weight).view(H, -1, D).sum((1, 2)) for w in (b.q, b.k, b.v)) \ + (b.o.weight.grad * b.o.weight).view(D, H, -1).sum((0, 2)) out.append((mlp.abs(), hd.abs())) elif kind == "activation": # Minitron (115.11): mean over sequence, then L2 over batch mlp = b.mlp_act.detach().abs().mean(1).norm(dim=0) hd = b.head_out.detach().norm(dim=-1).mean(2).norm(dim=0) out.append((mlp, hd)) else: # random baseline gen = torch.Generator().manual_seed(7 + len(out)) out.append((torch.rand(M, generator=gen), torch.rand(H, generator=gen))) return out # 3) physically remove the groups: smaller dense matrices, no masks def prune(model, kind, keep_mlp, keep_heads): m = copy.deepcopy(model) for b, (sm, sh) in zip(m.blocks, scores(m, kind)): im = sm.topk(keep_mlp).indices.sort().values ih = sh.topk(keep_heads).indices.sort().values rows = (ih[:, None] * b.dh + torch.arange(b.dh)).flatten() # q/k/v rows of the kept heads with torch.no_grad(): for name in ("gate", "up"): old = getattr(b, name); new = nn.Linear(D, keep_mlp, bias=False) new.weight.copy_(old.weight[im]); setattr(b, name, new) new = nn.Linear(keep_mlp, D, bias=False); new.weight.copy_(b.down.weight[:, im]); b.down = new for name in ("q", "k", "v"): old = getattr(b, name); new = nn.Linear(D, len(rows), bias=False) new.weight.copy_(old.weight[rows]); setattr(b, name, new) new = nn.Linear(len(rows), D, bias=False); new.weight.copy_(b.o.weight[:, rows]); b.o = new b.h = keep_heads m.zero_grad(set_to_none=True) return m pruned = {} for kind in ("random", "taylor", "activation"): pruned[kind] = prune(dense, kind, keep_mlp=88, keep_heads=6) # half the MLP, 6 of 8 heads with torch.no_grad(): print("%-10s loss %.3f params %d" % (kind, loss_of(pruned[kind], xe, ye).item(), nparams(pruned[kind]))) # 4) recover with knowledge distillation from the dense teacher (115.13) student = pruned["activation"]; opt = torch.optim.AdamW(student.parameters(), lr=1e-3) g = torch.Generator().manual_seed(4) for step in range(200): x, _ = batch(64, g) with torch.no_grad(): t_logp = F.log_softmax(dense(x), -1) s_logp = F.log_softmax(student(x), -1) kd = F.kl_div(s_logp, t_logp, log_target=True, reduction="batchmean") / T # KL(teacher || student) per token opt.zero_grad(); kd.backward(); opt.step() with torch.no_grad(): print("after distillation loss %.3f" % loss_of(student, xe, ye).item())
Output (PyTorch 2.8, CPU):
dense loss 2.377 params 104768 random loss 3.052 params 62784 taylor loss 3.004 params 62784 activation loss 2.970 params 62784 after distillation loss 2.517
Read it carefully. The pruned model really has 40% fewer parameters (62,784 instead of 104,768): no masks, smaller matrices. Removing half of the MLP and a quarter of the heads in one shot hurts a lot with any score. The two real scores beat random, but only a little. Recovery with 200 distillation steps closes most of the gap: from 2.970 to 2.517 nats. This is the pattern of every paper in this session: the score matters, but the training after pruning matters more.
The second program checks computational invariance (115.8) and compares a PCA slice with a random one on a small residual network whose RMSNorm gains are already folded into the matrices.
import torch torch.manual_seed(0) V, D, Hm, L = 100, 64, 256, 3 # vocab, hidden, MLP width, blocks E = torch.randn(V, D) * torch.logspace(0, -2, D) # embeddings with a few strong directions Win = [torch.randn(D, Hm) / D ** 0.5 for _ in range(L)] Wout = [torch.randn(Hm, D) / Hm ** 0.5 * 0.3 for _ in range(L)] Whead = torch.randn(D, V) / D ** 0.5 def rms(x): # RMSNorm with its gain already folded into Win return x * D ** 0.5 / x.norm(dim=-1, keepdim=True) # D stays the ORIGINAL width after slicing def run(E, Win, Wout, Whead, ids): h = E[ids]; states = [h] for wi, wo in zip(Win, Wout): h = h + torch.nn.functional.gelu(rms(h) @ wi) @ wo states.append(h) return rms(h) @ Whead, states ids = torch.randint(V, (4096,), generator=torch.Generator().manual_seed(1)) logits, states = run(E, Win, Wout, Whead, ids) # 1) computational invariance (115.8): any orthogonal Q gives the same output Q, _ = torch.linalg.qr(torch.randn(D, D)) def rotate(Q): # W_embd Q, Q^T W_in, W_out Q, Q^T W_head return E @ Q, [Q.T @ w for w in Win], [w @ Q for w in Wout], Q.T @ Whead lr, _ = run(*rotate(Q), ids) print("max |difference| after rotation: %.1e" % (lr - logits).abs().max().item()) # 2) PCA rotation (115.9): eigenvectors of C = sum of X^T X over all residual-stream signals X = torch.cat(states) evals, evecs = torch.linalg.eigh(X.T.double() @ X.double()) Qpca = evecs.flip(1).float() # largest eigenvalue first k = int(0.75 * D) # slice 25% of the hidden width print("energy kept in first %d of %d directions: %.4f" % (k, D, (evals.flip(0)[:k].sum() / evals.sum()).item())) # 3) slice: keep only the first k columns of Q, so every matrix gets smaller def sliced_error(Q): Qk = Q[:, :k] ls, _ = run(E @ Qk, [Qk.T @ w for w in Win], [w @ Qk for w in Wout], Qk.T @ Whead, ids) return ((ls - logits).norm() / logits.norm()).item() print("relative logit error, PCA slice: %.4f" % sliced_error(Qpca)) print("relative logit error, random slice: %.4f" % sliced_error(Q))
Output:
max |difference| after rotation: 3.1e-06 energy kept in first 48 of 64 directions: 0.9854 relative logit error, PCA slice: 0.1645 relative logit error, random slice: 0.6592
The rotation changes the output only by float32 rounding (\(3 \times 10^{-6}\)). Slicing 25% after the PCA rotation keeps 98.5% of the energy of the hidden states and gives a 16% logit error; slicing 25% after a random rotation gives 66%. The rotation is what makes slicing work. (This toy uses one \(Q\) for all layers, so it needs no adapters.)
Comparing the results
Figure 5 puts the Sheared LLaMA results next to models trained from scratch: training tokens on the horizontal axis (log scale), average downstream accuracy on the vertical axis. All numbers are from Table 2 of the Sheared LLaMA paper.
Note what the plot does not count: the 2T tokens that trained LLaMA2-7B in the first place. Pruning is cheap only if a good big model already exists. For a model family that is true, and that is the Minitron argument: train the biggest model once, then derive the others.
Where this is used
| Setting | Typical choice | Why |
|---|---|---|
| Building a family of small models (8B, 4B, ...) | Minitron: width pruning plus logit distillation, repeated | About 1% of the original tokens per smaller model; one teacher for the family |
| A small open model with a fixed, standard shape | Sheared LLaMA: masks with a target shape, then continued pretraining | Uniform layers that run with existing kernels and compare fairly with other models |
| Fitting a 70B model on fewer GPUs, with no training budget | SliceGPT at 20 to 30%, maybe plus quantization | No gradients needed; dense smaller matrices; 3 instead of 4 A100 40GB for LLaMA2-70B |
| A quick, task-agnostic compression on one GPU | LLM-Pruner at about 20%, with LoRA recovery | 10 samples to score, a few hours to recover; quality drops quickly above 20 to 30% |
| MoE models that do not fit in memory | Expert pruning (115.14), maybe expert skipping | Memory falls with every expert removed; Mixtral 8×7B fits on one 80 GB GPU with 6 experts |
| Low-latency decoding at small batch | Depth pruning (BI score, 115.12) | Each removed layer removes a sequential step; width pruning saves less time at batch 1 [Likely] |
| Video models: classification, action recognition, video retrieval | Prune heads and MLP channels of the vision encoder or the LLM, then distill on video data | Video means many frames per request; a thinner encoder cuts cost per frame; recover on the target task [Likely] |
| Video-RAG and key-moment detection on edge devices | A structurally pruned small LLM plus 4-bit quantization | Structured pruning and quantization combine: smaller matrices, then fewer bits per weight [Likely] |
Common mistakes
- Removing a row without its partner columns. Prune a head in \(W_Q\) but not in \(W_K\), \(W_V\) and \(W_O\), and the shapes break or the output silently changes. Always prune the whole coupled group (115.3).
- Masking instead of slicing. Setting a head's weights to zero gives no speed-up. You must build new, smaller matrices and update the config (number of heads, intermediate size).
- Ignoring GQA groups. Removing a key/value head without the query heads that share it breaks the model. Prune query heads inside groups, or whole groups.
- Expecting one-shot structured pruning to be free. Unlike 50% unstructured pruning with SparseGPT, removing 20% or more of whole structures needs recovery training. Plan the training budget from the start.
- Fine-tuning instead of distilling. With the original model available, training on its logits (115.13) recovers more per token than training on hard labels, as Minitron found.
- Forgetting the SliceGPT adapters and the norm fold. Slicing without folding the RMSNorm gains, or without the \(Q_{\ell-1}^\top Q_\ell\) residual matrices, breaks invariance. And the adapters mean that "30% sliced" is not 30% fewer parameters.
- Calibrating on the wrong data. All importance scores depend on the calibration set. Scoring on English web text and then serving code or another language can remove structures that matter there.
- Comparing perplexities across papers. Context length, data split and tokenizer differ. LLM-Pruner's WikiText2 perplexity of 12.62 for LLaMA-7B and SliceGPT's 5.47 for LLaMA2-7B are different setups, not different models' quality.
Interview questions
Why does 50% unstructured sparsity not make an LLM 2× faster, and what does structured pruning change?basic
You want to remove one attention head from a Llama block. Which weights go?basic
Why is a hidden (embedding) channel the most expensive structure to prune?core
Explain LLM-Pruner's importance score. Why keep the gradient term?core
How does Sheared LLaMA reach an exact target shape?advanced
What is dynamic batch loading and why does it help after pruning?core
Why is SliceGPT allowed to rotate the hidden space? Where does it fail for LayerNorm?advanced
Why does SliceGPT need extra matrices on the residual path, and what do they cost?advanced
Width or depth: which would you prune to make an 8B model into 4B?core
Describe the Minitron recipe and why it is cheap.core
How would you prune experts in an MoE model like Mixtral?advanced
Exercises
- Using (115.1), how many parameters does LLaMA2-7B lose if you remove 8 heads in every layer (keep \(d\), \(m\), \(L\))? What fraction is that?
- How many MLP channels per layer must you remove from LLaMA2-7B to save the same number of parameters as one head per layer?
- In Figure 3, find \(\log\alpha\) where \(P(z = 1) \approx 0.5\). What is \(P(z = 0)\) there?
- Dynamic batch loading: weights \((0.6, 0.4)\), gaps \((0.1, 0.5)\). Compute the new weights.
- A hidden state cloud in 2D has covariance eigenvalues 4 and 1. What fraction of the energy does SliceGPT keep when it slices the 2D stream to 1D? What if the eigenvalues are equal?
- Mixtral has 32 MoE layers with 8 experts each. Lu et al. try every subset per layer. How many subsets in total for \(r = 6\)? Why do they not try all combinations across layers jointly?
Answerscheck
Key takeaways
- Structured pruning removes heads, MLP channels, hidden channels, layers or experts, so the result is a smaller dense model that is faster with ordinary kernels (115.1, 115.2).
- Weights connected by a single path form coupled groups (115.3); a head is rows of \(W_Q, W_K, W_V\) plus columns of \(W_O\); a hidden channel touches every matrix in the model.
- LLM-Pruner scores groups with a Taylor expansion of the loss (115.4) and recovers with LoRA; it works at about 20%, not at 50%.
- Sheared LLaMA learns hard concrete masks (115.5) under a Lagrangian shape constraint (115.6), then trains 50B tokens with dynamic batch loading (115.7): a better 2.7B model than ones trained on 1T tokens.
- SliceGPT uses RMSNorm invariance (115.8), rotates each layer with PCA (115.9) and deletes the last channels, with small adapters on the residual path (115.10).
- Minitron scores with activations (115.11, 115.12) and recovers with logit distillation (115.13) on about 1% of the original tokens. MoE models can drop whole experts by layer-wise reconstruction (115.14).
Key formulas
References for this session
- Ma et al. (2023). LLM-Pruner: On the Structural Pruning of Large Language Models
- Xia et al. (2023). Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
- Ashkboos et al. (2024). SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- Muralidharan et al. (2024). Compact Language Models via Pruning and Knowledge Distillation (Minitron)
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Ma et al. (2023). LLM-Pruner: On the Structural Pruning of Large Language Models
- Xia et al. (2023). Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
- Ashkboos et al. (2024). SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- Muralidharan et al. (2024). Compact Language Models via Pruning and Knowledge Distillation (Minitron)
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Sreenivas, Muralidharan, Joshi et al. (2024). LLM Pruning and Distillation in Practice: The Minitron Approach
- Lu, Liu, Xu et al. (2024). Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
- Louizos, Welling and Kingma (2017). Learning Sparse Neural Networks through L0 Regularization
- Men, Xu, Zhang et al. (2024). ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
- Sun et al. (2023). A Simple and Effective Pruning Approach for Large Language Models (Wanda)
- Frantar & Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
- Hinton et al. (2015). Distilling the Knowledge in a Neural Network