LLM & VLM courseBlog
115Track 5: LLM inference and serving · session 38 of 65

Structured pruning: LLM-Pruner, Sheared LLaMA, SliceGPT, Minitron

Unstructured zeros save little on a GPU. Structured pruning removes whole heads, MLP channels, hidden channels, layers or experts, so the result is a smaller dense model that is faster on any hardware. This session shows how to find the weights that must be removed together, how LLM-Pruner, Sheared LLaMA, SliceGPT and Minitron choose what to remove, and how training with distillation brings the quality back.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 113 and 114 (pruning basics, SparseGPT and Wanda), session 26 (MHA and GQA), basic linear algebra (orthogonal matrices, PCA)

What you will learn

  • Which structures a Llama block has (heads, MLP channels, hidden channels, layers) and how many parameters each one costs
  • How coupled structures are found with a dependency graph, and why a head or a channel must be removed from several matrices at once
  • LLM-Pruner: Taylor importance of a group, one-shot removal, and a short LoRA recovery
  • Sheared LLaMA: learned masks with the hard concrete distribution, a Lagrangian that forces a target shape, and dynamic batch loading during continued pretraining
  • SliceGPT: computational invariance of RMSNorm networks, a PCA rotation per layer, and deleting whole rows and columns
  • Minitron: activation-based importance, width versus depth pruning, and recovery by logit distillation on about 1% of the original tokens
  • Expert pruning in MoE models, and how to read the memory and accuracy numbers these papers report

Why this matters

In session 114 we set half of the weights to zero and kept the quality. But the matrices kept their shape. On a GPU, a \(4096 \times 4096\) matrix with 50% random zeros is still a \(4096 \times 4096\) matrix: the same memory (unless you pay for an index format) and almost the same time. Only the special 2:4 pattern gives some speed, and only on some GPUs.

Structured pruning removes whole pieces instead: an attention head, a channel of the MLP, a channel of the hidden state, a whole layer, or a whole expert. Each removal deletes complete rows and columns, so the matrices become smaller dense matrices. A 7B model becomes, for example, a normal 2.7B model. It runs faster with the same kernels, on any GPU or CPU, and needs no sparse format. The price: removing big pieces hurts much more than removing single weights, so structured pruning almost always needs some training afterwards to recover.

This idea is now a standard way to build small models. Sheared LLaMA cut LLaMA2-7B down to 2.7B and 1.3B models with 50B tokens of continued pretraining [Certain]. NVIDIA's Minitron models (8B and 4B) were cut from Nemotron-4 15B and trained with distillation on 94B tokens, while the 15B model was trained on 8T tokens [Certain]. Later Minitron work did the same to Llama 3.1 8B (to 4B) and Mistral NeMo 12B (to 8B) [Certain].

We will meet four methods that answer the same three questions in different ways: what can be removed (the structure), which pieces to remove (the importance score), and how to recover the quality (fine-tuning, continued pretraining or distillation).

MethodWhat it removesHow it choosesHow it recovers
LLM-Pruner (2023)coupled groups: heads, MLP channelsgradient (Taylor) importance on 10 samplesLoRA on about 50K instruction samples
Sheared LLaMA (2023)layers, heads, MLP channels, hidden channels, down to a fixed target shapelearned masks, trained with a shape constraintcontinued pretraining on 50B tokens, with dynamic data mixing
SliceGPT (2024)hidden channels, after a rotationPCA of the hidden statesnone needed; optional short fine-tuning
Minitron (2024)heads, MLP channels, hidden channels, or layersactivation size on 1024 sampleslogit distillation from the original model

What can be removed in a Llama block

Recall the shape of one Llama-style block (session 22). The hidden state (the residual stream) has \(d\) numbers per token. Attention has \(H\) heads of size \(d_h\). The SwiGLU MLP has an intermediate width \(m\) and three matrices: gate, up and down. There are \(L\) blocks, an embedding table and an output head over a vocabulary of size \(V\). With full multi-head attention, the parameter count is:

\[ N = \underbrace{2Vd}_{\text{embed + head}} + L\big(\underbrace{4\,d\,H d_h}_{\text{attention}} + \underbrace{3\,d\,m}_{\text{MLP}} + \underbrace{2d}_{\text{norms}}\big) + d \tag{115.1} \]
Read the symbols
\(d\)
hidden size (width of the residual stream); 4096 in LLaMA2-7B
\(H, d_h\)
number of attention heads and the size of one head; 32 and 128 in LLaMA2-7B
\(m\)
intermediate width of the MLP; 11008 in LLaMA2-7B
\(L, V\)
number of blocks (32) and vocabulary size (32000)
\(4\,d\,Hd_h\)
the four attention matrices \(W_Q, W_K, W_V\) (each \(d \to Hd_h\)) and \(W_O\) (\(Hd_h \to d\))

Each structure we can remove has a price in parameters. Read it from (115.1):

\[ \text{one head: } 4\,d\,d_h, \qquad \text{one MLP channel: } 3d, \qquad \text{one hidden channel: } L\,(4Hd_h + 3m + 2) + 2V + 1 \tag{115.2} \]

A head owns \(d_h\) rows of \(W_Q\), \(W_K\), \(W_V\) and \(d_h\) columns of \(W_O\). An MLP channel owns one row of gate, one row of up and one column of down. A hidden channel is special: the residual stream passes through every block, so one hidden channel touches every matrix in every layer, both norms, the embedding table and the output head. A layer (depth pruning) removes everything inside one block.

Worked example: the price list of LLaMA2-7B

With \(d = 4096\), \(H = 32\), \(d_h = 128\), \(m = 11008\), \(L = 32\), \(V = 32000\), formula (115.1) gives \(N = 6{,}738{,}415{,}616\), the official 6.74B. Per block: attention \(4 \cdot 4096^2 = 67{,}108{,}864\), MLP \(3 \cdot 4096 \cdot 11008 = 135{,}266{,}304\). So MLPs are 64.2% of the model, attention 31.9% and embeddings plus head 3.9%.

Prices from (115.2): one head costs \(4 \cdot 4096 \cdot 128 = 2{,}097{,}152\) weights. One MLP channel costs \(3 \cdot 4096 = 12{,}288\). One hidden channel costs \(32 \cdot (4 \cdot 4096 + 3 \cdot 11008 + 2) + 2 \cdot 32000 + 1 = 1{,}645{,}121\). Cutting the hidden size from 4096 to 3072 alone removes exactly 25% of all weights, because every matrix has \(d\) on one side.

The Sheared-LLaMA-2.7B shape is \(L = 32\), \(d = 2560\), \(H = 20\), \(m = 6912\). Formula (115.1) gives 2,701,560,320 parameters: 40.1% of the 7B model. The 1.3B shape (\(L = 24\), \(d = 2048\), \(H = 16\), \(m = 5504\)) gives 1,345,423,360: 20.0%.

attentionMLPembedding + head
Figure 1. The parameter count of a Llama-style model, formula (115.1), for the shape you choose. The top bar is LLaMA2-7B; the bottom bar is your shape, on the same scale. The presets are the two Sheared-LLaMA targets and a 25% cut of the hidden size.
How to read Figure 1

Start from LLaMA2-7B and move one slider at a time. Removing heads only shrinks the attention part. Removing MLP channels only shrinks the MLP part, which is the biggest. Lowering \(d\) shrinks all three parts at once, because every matrix has \(d\) on one side. Lowering \(L\) removes whole blocks but leaves the embeddings. The readout also shows the price of one more head, channel or layer at the current shape, from formula (115.2).

Which axis should you prune? The papers agree on one thing: thin layers are usually better than fewer layers at the same size. Minitron found that width pruning (heads, MLP channels, hidden channels) beat depth pruning for models up to 15B after retraining [Certain]. But removing layers gives the most speed for small batches, because each layer is a sequential step, so depth pruning is still used (ShortGPT removes layers whose output is almost equal to their input) [Certain].

Coupled structures: what must go together

You cannot remove one row of \(W_Q\) and keep the rest of the model as it is. If output channel \(j\) of a matrix disappears, every matrix that reads channel \(j\) must lose the matching input column, otherwise the shapes do not fit. Weights connected like this form a coupled group: they are kept or removed together.

LLM-Pruner finds these groups automatically with a dependency graph. Each neuron (one input or output channel of a layer) is a node. Two rules decide when a neuron \(N_j\) must follow neuron \(N_i\):

\[ N_j \in \operatorname{Out}(N_i) \ \wedge\ \operatorname{Deg}^{-}(N_j) = 1 \ \Rightarrow\ N_j \text{ depends on } N_i, \qquad N_i \in \operatorname{In}(N_j) \ \wedge\ \operatorname{Deg}^{+}(N_i) = 1 \ \Rightarrow\ N_i \text{ depends on } N_j \tag{115.3} \]
Read the symbols
\(\operatorname{Out}(N_i)\), \(\operatorname{In}(N_j)\)
the neurons that \(N_i\) sends to, and the neurons that send to \(N_j\)
\(\operatorname{Deg}^{-}\), \(\operatorname{Deg}^{+}\)
in-degree and out-degree: how many neurons send to, or receive from, this neuron

In words: if a neuron gets its only input from \(N_i\), it is useless without \(N_i\); if a neuron sends its only output to \(N_j\), it is useless without \(N_j\). Start from one neuron, apply both rules until nothing new is added, and you get its group. In a Llama block this gives three kinds of group: an attention head (rows of \(W_Q, W_K, W_V\) and columns of \(W_O\)), an MLP channel (one row of gate and up, one column of down), and a hidden channel (a column or row in every matrix of every block, because of the residual connections). Click through them in Figure 2.

Figure 2. The weight matrices of a tiny block (\(d = 8\), 4 heads of size 2, \(m = 12\)), drawn in the PyTorch layout: rows are outputs, columns are inputs. The colored cells are one coupled group. Click any cell to pick the group that contains it.
How to read Figure 2

A head is a band of rows in \(W_Q, W_K, W_V\) and the matching band of columns in \(W_O\). An MLP channel is one row in gate and up and one column in down. A hidden channel is a column of every matrix that reads the residual stream (\(W_Q, W_K, W_V\), gate, up), a row of every matrix that writes to it (\(W_O\), down), and one entry of each norm. The readout gives the size of the group in this toy block and in LLaMA2-7B. The hidden channel is the most expensive group by far, and it is shared by all layers.

Scoring a group: LLM-Pruner

Which groups are safe to remove? LLM-Pruner asks: how much would the loss change if this group were set to zero? Write the group's weights as one vector \(\mathbf{w}\). A second-order Taylor expansion of the loss around the current weights gives:

\[ I(\mathbf{w}) = \big|\mathcal{L}(\mathbf{w}) - \mathcal{L}(\mathbf{w} = 0)\big| \approx \Big|\, \mathbf{g}^\top \mathbf{w} - \tfrac12\, \mathbf{w}^\top \mathbf{H}\, \mathbf{w} \,\Big|, \qquad I_{\text{group}} = \sum_{\mathbf{w} \in \text{group}} I(\mathbf{w}) \tag{115.4} \]
Read the symbols
\(\mathbf{g}\)
the gradient of the loss with respect to \(\mathbf{w}\), computed by one backward pass on a few calibration texts
\(\mathbf{H}\)
the Hessian (second derivatives); LLM-Pruner approximates its diagonal by the Fisher information, the sum of squared per-sample gradients
\(I_{\text{group}}\)
the importance of a coupled group: the sum of the scores of its parts (the paper also tried product, max and "last only"; sum worked best)

In session 113 the gradient term vanished because a trained model sits at a minimum of its training loss. LLM-Pruner keeps it on purpose: the calibration texts are not the training data, so the gradient there is not zero. The paper uses only 10 samples from BookCorpus of 128 tokens each [Certain]. The first-order part \(|\mathbf{g}^\top\mathbf{w}|\) alone is the "vector-wise" score; adding the diagonal second-order term per element gives the "element-wise" score.

Worked example: magnitude and Taylor disagree

Two MLP channels, each shown with three of its weights. Channel A: \(\mathbf{w} = (0.9, -0.8, 0.7)\), \(\mathbf{g} = (0.02, 0.01, 0.01)\). Channel B: \(\mathbf{w} = (0.2, -0.3, 0.1)\), \(\mathbf{g} = (0.3, -0.2, 0.4)\).

By magnitude, A is bigger: \(\lVert \mathbf{w}_A \rVert = 1.393\), \(\lVert \mathbf{w}_B \rVert = 0.374\). By first-order Taylor, \(|\mathbf{g}_A^\top \mathbf{w}_A| = |0.018 - 0.008 + 0.007| = 0.017\) and \(|\mathbf{g}_B^\top \mathbf{w}_B| = |0.06 + 0.06 + 0.04| = 0.16\). The loss barely notices channel A, so Taylor removes A, even though its weights are large.

Algorithm 1: LLM-Pruner
  1. input: model, about 10 calibration texts, target ratio (for example 20% of parameters)
  2. build the dependency graph and collect all coupled groups (115.3)
  3. run forward and backward on the calibration texts; keep \(\mathbf{g}\) and the squared per-sample gradients
  4. score every group with (115.4), summing over its parts
  5. for each block in the pruned range (the paper keeps the first 3 blocks and the last block):
  6.   remove the lowest-scoring heads and MLP channels until the target ratio is reached // physically slice the matrices
  7. recover: LoRA (rank 8) on about 50K Alpaca samples, 2 epochs // about 3 hours on one GPU

What did it achieve? On LLaMA-7B with 20% of the parameters removed (6.74B to 5.42B), the average zero-shot accuracy went from 63.25 to 56.82 without tuning and to 60.07 after LoRA: 94.97% of the original [Certain]. WikiText2 perplexity went from 12.62 to 17.37 after tuning [Certain] (these perplexities use the paper's own short-context setup, so they are not comparable with other papers). That is a real drop for a 20% cut, and it is the main lesson of LLM-Pruner: a cheap one-shot cut with a tiny recovery works at small ratios, but at 50% the quality falls much more. Bigger cuts need much more training.

Prune to a target shape: Sheared LLaMA

LLM-Pruner removes the least important groups and accepts whatever shape results. The layers then end up with different widths, which is awkward for kernels and for comparing with other models. Sheared LLaMA does the opposite: it fixes the target shape first (for example the shape of Pythia-1.4B or INCITE-3B, so that the results can be compared fairly) and then learns which parts of the 7B model to keep.

Learned masks with the hard concrete distribution

Every prunable structure gets a mask value \(z\): one per layer (\(z^{\text{layer}}\)), per head (\(z^{\text{head}}\)), per MLP channel (\(z^{\text{int}}\)) and per hidden channel (\(z^{\text{hidden}}\)). The output of a structure is multiplied by its \(z\). We want each \(z\) to be exactly 0 or exactly 1 at the end, but we also want gradients to train them. The hard concrete distribution (from L0 regularization) gives both. Each mask has one learnable number \(\log\alpha\). At each step we sample:

\[ u \sim \mathcal{U}(0,1), \quad s = \sigma\!\Big(\frac{\log u - \log(1-u) + \log\alpha}{\beta}\Big), \quad z = \min\!\big(1, \max\!\big(0,\ s(\zeta - \gamma) + \gamma\big)\big) \tag{115.5} \]
Read the symbols
\(\log\alpha\)
the learned parameter of one mask; large means "keep", very negative means "remove"
\(\beta\)
temperature; Sheared LLaMA uses 0.83
\(\gamma, \zeta\)
stretch limits \(-0.1\) and \(1.1\): the sample is stretched to \((-0.1, 1.1)\) and then clipped to \([0, 1]\), so the clipped ends become exact 0 and exact 1 with real probability

Because of the clipping, \(z = 0\) exactly when the stretched sample is below 0, which means \(s \le -\gamma/(\zeta-\gamma) = 1/12\), and \(z = 1\) exactly when \(s \ge (1-\gamma)/(\zeta-\gamma) = 11/12\). With these constants the two probabilities have a simple closed form: \(P(z = 0) = \sigma(-\beta \ln 11 - \log\alpha)\) and \(P(z = 1) = 1 - \sigma(\beta \ln 11 - \log\alpha)\). In between, \(z\) is a smooth function of \(\log\alpha\), so gradients flow. Figure 3 shows the shape.

Figure 3. 20,000 samples of one hard concrete mask \(z\) (formula 115.5) with \(\beta = 0.83\), \(\gamma = -0.1\), \(\zeta = 1.1\). The two bars at the ends (in a different color) are the probability of an exact 0 or an exact 1; the bars between them are the samples strictly between 0 and 1.
How to read Figure 3

At \(\log\alpha = 0\) the mask is undecided: exact 0 and exact 1 each have probability 0.120, and most samples are in between. Move the slider right: the mass moves to \(z = 1\) (the structure is kept). Move it left: it moves to \(z = 0\). Training pushes every \(\log\alpha\) to one side. The expected number of nonzero masks, \(\sum (1 - P(z=0))\), is smooth in \(\log\alpha\), which is what the shape constraint below uses. The readout compares the exact formula with the samples.

Forcing the target shape

Now we need the masks to end at exactly the target size, for example \(H_T = 20\) heads per layer. Sheared LLaMA adds a constraint with Lagrange multipliers \(\lambda, \phi\) for every kind of structure:

\[ \tilde{\mathcal{L}}^{\text{head}} = \lambda^{\text{head}}\Big(\sum z^{\text{head}} - H_T\Big) + \phi^{\text{head}}\Big(\sum z^{\text{head}} - H_T\Big)^2, \qquad \min_{\theta, z}\ \max_{\lambda, \phi}\ \mathcal{L}(\theta, z) + \sum_{\text{kinds}} \tilde{\mathcal{L}} \tag{115.6} \]

The model weights \(\theta\) and the masks go down the gradient of the language-model loss plus the penalty. The multipliers \(\lambda, \phi\) go up the gradient: as long as the model has too many heads, the penalty grows, until the masks obey. This is a min-max game, the same idea as constrained optimization in any textbook. In the paper, the masks, \(\lambda\) and \(\phi\) all use learning rate 1.0, and the pruning stage uses only 0.4B tokens [Certain]. At the end, the structures with \(z = 0\) are deleted and the masks are folded into the weights.

Worked example: one penalty value

Suppose one layer still has \(\sum z^{\text{head}} = 24\) heads but the target is \(H_T = 20\), with \(\lambda = -0.5\) and \(\phi = 1.0\). The penalty is \(-0.5 \cdot 4 + 1.0 \cdot 16 = 14\). Its gradient with respect to each \(z\) is \(\lambda + 2\phi \cdot 4 = 7.5\), positive, so gradient descent pushes all head masks down. The multipliers keep changing until the count reaches 20 and the penalty is 0.

Continued pretraining with dynamic batch loading

After pruning, the 2.7B model is trained further on 50B tokens of RedPajama (seven domains: CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, StackExchange) [Certain]. The paper noticed that pruning hurts some domains more than others, so a fixed data mixture wastes tokens on domains that already recovered. Dynamic batch loading changes the mixture during training. Every \(m\) steps, compare the loss on each domain \(i\) with a reference loss:

\[ \Delta_t[i] = \max\big(\ell_t[i] - \ell_{\text{ref}}[i],\ 0\big), \qquad \alpha_t = w_{t-m} \odot \exp(\Delta_t), \qquad w_t = \frac{\alpha_t}{\sum_i \alpha_t[i]} \tag{115.7} \]
Read the symbols
\(\ell_t[i]\)
the current validation loss of the pruned model on domain \(i\)
\(\ell_{\text{ref}}[i]\)
the loss we hope to reach; by default predicted by a scaling-law fit for a model of the target size
\(w_t\)
the fraction of the next batches drawn from each domain

Domains that are far behind get more data; domains that are already at the reference keep their share. Example with three domains, \(w = (0.5, 0.3, 0.2)\) and gaps \(\Delta = (0, 0.2, 0.05)\): \(\alpha = (0.5, 0.366, 0.210)\), and after normalizing \(w = (0.464, 0.340, 0.195)\). The domain with the largest gap gained 4 points.

Algorithm 2: Sheared LLaMA
  1. input: source model (LLaMA2-7B), target shape \((L_T, d_T, H_T, m_T)\), pretraining data split into domains
  2. add masks \(z^{\text{layer}}, z^{\text{hidden}}, z^{\text{head}}, z^{\text{int}}\), each with a \(\log\alpha\); set \(\lambda, \phi = 0\)
  3. repeat for 0.4B tokens: // pruning stage
  4.   sample all \(z\) with (115.5); forward with masked structures; loss = LM loss + penalties (115.6)
  5.   gradient step down for \(\theta\) and \(\log\alpha\); gradient step up for \(\lambda, \phi\); update the domain mix with (115.7)
  6. keep the structures with the largest masks so the shape is exactly the target; delete the rest
  7. continued pretraining on 50B tokens, still updating the domain mix with (115.7)

The results show why this is worth it. In the paper's Table 2, Sheared-LLaMA-2.7B (50B tokens of training after pruning) has an average downstream accuracy of 56.7, against 55.7 for OpenLLaMA-3B-v2 trained on 1T tokens and 54.7 for INCITE-Base-3B trained on 800B tokens [Certain]. The 50B tokens are 5% of OpenLLaMA's budget; counting compute, the paper claims about 3% of the cost of training such models from scratch [Certain]. Figure 5 plots these numbers. The pruning stage itself is about 5× slower per token than normal training, because of the masks, but it is short [Certain].

Rotate, then delete: SliceGPT

Deleting hidden channels directly is hard: in a trained model, every channel carries a bit of everything. SliceGPT first rotates the hidden space so that the information is packed into the first channels, and then deletes the last channels. It needs no gradients and no training.

Computational invariance

The trick is a property of RMSNorm. For any orthogonal matrix \(Q\) (\(Q^\top Q = I\)), multiplying a vector by \(Q\) does not change its length, so

\[ \operatorname{RMSNorm}(\mathbf{x} Q) = \operatorname{RMSNorm}(\mathbf{x})\, Q \qquad \text{(no learned gain)} \tag{115.8} \]

First, fold the learned gain of each RMSNorm into the next matrix (LayerNorm models like OPT are first turned into RMSNorm models, by also folding the mean subtraction into nearby matrices). Then choose an orthogonal \(Q\) and transform the weights: the embedding becomes \(W_{\text{embd}} Q\), every matrix that reads the residual stream becomes \(Q^\top W_{\text{in}}\), every matrix that writes to it becomes \(W_{\text{out}} Q\), and the output head becomes \(Q^\top W_{\text{head}}\). The residual stream is now \(\mathbf{x}Q\) everywhere, and by (115.8) the network computes exactly the same output. Nothing changed yet; we only chose a new coordinate system.

Choosing Q with PCA, then slicing

Run a calibration set through the model and collect the hidden states \(X_\ell\) that enter block \(\ell\). The best rotation, in the sense of keeping the most energy in the first \(k\) coordinates, is given by principal component analysis:

\[ C_\ell = \sum_{i} X_{\ell,i}^\top X_{\ell,i}, \qquad C_\ell = Q_\ell \Lambda_\ell Q_\ell^\top, \qquad \text{keep } D = Q_\ell[:, 1{:}k] \tag{115.9} \]
Read the symbols
\(X_{\ell,i}\)
the hidden states (one row per token) of calibration sequence \(i\) entering block \(\ell\)
\(Q_\ell, \Lambda_\ell\)
eigenvectors and eigenvalues of \(C_\ell\), sorted from the largest eigenvalue down
\(k\)
the new hidden width; slicing 25% of \(d = 4096\) gives \(k = 3072\)

Deleting coordinates \(k+1 \ldots d\) after the rotation means: delete the last rows of every \(Q^\top W_{\text{in}}\) and the last columns of every \(W_{\text{out}} Q\). Every matrix becomes a smaller dense matrix. The energy lost is exactly the sum of the deleted eigenvalues, \(\sum_{j \gt k} \lambda_j\). Figure 4 shows this in two dimensions.

hidden stateskept axis (solid)states after deleting the dashed axis
Figure 4. 300 hidden states in a 2-D residual stream (fixed seed). We rotate the coordinate system by \(\theta\), keep the first new coordinate (solid line) and delete the second (dashed). The small dots are the states after deletion. SliceGPT does the same in 4096 dimensions and deletes about 1000 of them.
How to read Figure 4

At \(\theta = 0\) we keep the original first channel; the cloud is tilted, so a large part of the energy is lost. Turn the slider: the kept energy goes up and down. Press "Use the PCA angle": the kept axis lines up with the long direction of the cloud, and the kept energy is the largest eigenvalue divided by the sum of both, the maximum possible. No other angle does better, which is why SliceGPT uses PCA.

A different rotation per layer

Hidden states change from layer to layer, so one \(Q\) for the whole model is not the best. SliceGPT uses a different \(Q_\ell\) for each block. But the residual connection then carries \(\mathbf{x}Q_{\ell-1}\) into a block that expects \(\mathbf{x}Q_\ell\). The fix is a small extra matrix on the residual path:

\[ \mathbf{x}\,Q_{\ell-1} \ \longrightarrow\ \mathbf{x}\,Q_{\ell-1}\,\big(Q_{\ell-1}^\top Q_\ell\big) = \mathbf{x}\,Q_\ell \tag{115.10} \]

After slicing, this adapter is a \(k \times k\) matrix that must be stored and multiplied. So slicing 30% of the hidden size does not remove 30% of the parameters. For LLaMA2-70B (68.98B), 30% slicing leaves about 51.6B parameters, not 48.3B [Certain for 51.6B, from the paper].

Algorithm 3: SliceGPT
  1. input: model with RMSNorm (convert LayerNorm first), calibration set (the paper: 1024 sequences of 2048 tokens), slicing fraction
  2. fold every RMSNorm gain into the matrices that follow it
  3. for \(\ell = 1 \ldots L\):
  4.   run the calibration set up to block \(\ell\); \(C_\ell \leftarrow \sum_i X_{\ell,i}^\top X_{\ell,i}\) // in float64
  5.   \(Q_\ell \leftarrow\) eigenvectors of \(C_\ell\), largest first (115.9)
  6.   \(W_{\text{in}} \leftarrow (Q_\ell^\top W_{\text{in}})[{:}k, :]\), \(W_{\text{out}} \leftarrow (W_{\text{out}} Q_\ell)[:, {:}k]\), residual adapter \(\leftarrow (Q_{\ell-1}^\top Q_\ell)[{:}k, {:}k]\) (115.10)
  7. slice the embedding and the head with the first and last \(Q\)

How well does it work? On LLaMA2-7B with 1024 calibration sequences from WikiText2, WikiText2 perplexity goes from 5.47 (dense) to 6.64 at 20% slicing, 7.24 at 25% and 8.12 at 30%; SparseGPT with 2:4 sparsity gives 8.69 [Certain]. For LLaMA2-70B the paper reports running on 3 instead of 4 A100 40GB GPUs, with total compute 66% of the dense model (330 vs 500 GPU-milliseconds per token in its Hugging Face measurement) [Certain]. On zero-shot tasks, the paper reports keeping about 99% of the accuracy for LLaMA2-70B and OPT-66B at 25% slicing, and 90% for the small Phi-2 (with a short fine-tune for LLaMA2 and Phi-2) [Certain]. Big models have more redundant directions, so they slice better. [Likely]

Prune, then distill: Minitron

Minitron (NVIDIA) asked a practical question: if we already trained a big model on 8T tokens, can we get the smaller sizes of the family by pruning and a little retraining, instead of training each size from scratch? Its answer: yes, if the retraining is distillation from the big model (session 59).

Importance from activations only

Minitron scores structures with forward passes only, no gradients, on 1024 random training samples [Certain]. For head \(i\), MLP neuron \(i\) and hidden (embedding) channel \(i\):

\[ F_{\text{head}}^{(i)} = \sum_{B,S} \big\lVert \operatorname{Attn}(XW^{Q,i}, XW^{K,i}, XW^{V,i}) \big\rVert_2, \qquad F_{\text{neuron}}^{(i)} = \sum_{B,S} X\,(W_1^{i})^\top, \qquad F_{\text{emb}}^{(i)} = \sum_{B,S} \operatorname{LN}(X)_i \tag{115.11} \]

The sums run over the batch \(B\) and the sequence \(S\). The paper tested several ways to combine them and chose: take the mean over the sequence, then the L2 norm over the batch [Certain]. A structure whose output is large on real data is important; one whose output is always tiny is not. For layers (depth), it uses either the perplexity change when a layer is removed, or block importance:

\[ \mathrm{BI}_i = 1 - \E_{X,t}\left[ \frac{\mathbf{x}_{i,t}^\top \mathbf{x}_{i+1,t}}{\lVert \mathbf{x}_{i,t} \rVert_2\, \lVert \mathbf{x}_{i+1,t} \rVert_2} \right] \tag{115.12} \]

Here \(\mathbf{x}_{i,t}\) is the hidden state of token \(t\) entering layer \(i\). If a layer's output is almost parallel to its input (cosine near 1), the layer changes little and \(\mathrm{BI}\) is near 0: a candidate for removal. This is the score of ShortGPT.

Retrain with distillation

After pruning (for example 15B to 8B: hidden size 6144 to 4096, MLP width 24576 to 16384, same 32 layers and 48 heads), the student is trained to match the teacher's next-token distribution:

\[ \mathcal{L}_{\text{KD}} = \frac{1}{T} \sum_{t=1}^{T} \KL\big(p_{\text{teacher}}(\cdot \mid x_{\lt t})\ \big\|\ p_{\text{student}}(\cdot \mid x_{\lt t})\big) \tag{115.13} \]

The teacher gives a full distribution over the vocabulary at every position, not one correct token, so each training token carries much more signal. Minitron found forward KL better than reverse KL, MSE or cosine on the logits, with temperature 1.0 [Certain]. When many layers are removed, it also adds losses on intermediate hidden states; for the 8B and 4B models, which kept all 32 layers, logits alone were used [Certain].

Algorithm 4: Minitron (one compression step)
  1. input: trained teacher, target size, 1024 calibration samples
  2. forward the calibration samples; score heads, neurons and hidden channels with (115.11) (and layers with (115.12))
  3. list candidate shapes near the target size; for each, keep the top-scoring structures and slice the weights
  4. train each candidate briefly (about 1.8B tokens) and keep the best one // rankings change after a little training
  5. distill the chosen student from the teacher with (115.13) on about 94B tokens
  6. to get a still smaller model, prune this result again (8B, then 4B)

Results: Minitron 8B reached 63.8 on MMLU (5-shot), against 54.7 for Nemotron-3 8B trained from scratch and 66.6 for the 15B teacher; Minitron 4B reached 58.6 [Certain]. Each smaller model used 94B tokens, about 1.2% of the teacher's 8T. The paper reports up to 40× fewer training tokens per model than training from scratch and 1.8× lower compute for the whole 15B + 8B + 4B family [Certain]. The follow-up paper added teacher correction: when the original training data is not available, first fine-tune the teacher briefly on the distillation data, so that it is not distilling from a distribution it has never seen [Certain].

Pruning experts in MoE models

A mixture-of-experts layer (session 29) has \(n\) expert MLPs and a router that sends each token to a few of them. All experts must be in memory, even though each token uses only a few. So a whole expert is a natural structure to remove: it is a large group, and removing it cuts memory directly.

Lu et al. prune Mixtral 8×7B one layer at a time. Run calibration data (128 sequences of 2048 tokens from C4) through the model, store the input \(\mathbf{x}\) and output \(F(\mathbf{x})\) of each MoE layer, and try every subset \(\mathcal{C}\) of \(r\) experts:

\[ \mathcal{C}^\ast = \argmin_{\mathcal{C} \subset \{1..n\},\ |\mathcal{C}| = r} \big\lVert F'(\mathbf{x}, \mathcal{C}) - F(\mathbf{x}) \big\rVert_F \tag{115.14} \]

\(F'(\mathbf{x}, \mathcal{C})\) is the layer output when the router can only choose experts in \(\mathcal{C}\). With \(n = 8\) there are only \(\binom{8}{6} = 28\) subsets for \(r = 6\) and \(\binom{8}{4} = 70\) for \(r = 4\), so trying them all is cheap: about 30 and 90 minutes [Certain]. Memory falls from 89,926 MB to 68,383 MB (76%) with 6 experts and to 46,879 MB (52%) with 4, so the model fits on one 80 GB GPU instead of two [Certain]. The average zero-shot accuracy over 8 tasks went from 67.58 to 64.22 with 6 experts and 59.57 with 4 [Certain]. The same paper also skips the second expert for a token when its router weight is much smaller than the first one, for a 1.2× to 1.3× speed-up [Certain].

Code: score, slice, distill

The first program trains a tiny Llama-style model on a fixed random Markov chain, scores MLP channels and heads three ways (random, LLM-Pruner Taylor, Minitron activation), physically slices the matrices to half the MLP width and 6 of 8 heads, and then recovers with logit distillation. It runs in about a minute on a laptop CPU and is deterministic.

import copy, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
V, D, H, M, L, T = 32, 64, 8, 176, 2, 32          # vocab, hidden, heads, MLP width, layers, seq len

# data: a fixed random 2nd-order Markov chain, so there is something to learn
P = torch.softmax(3.0 * torch.randn(V, V, V), -1)
def batch(n, g):
    x = torch.randint(V, (n, 2), generator=g)
    for _ in range(T - 1):
        x = torch.cat([x, torch.multinomial(P[x[:, -2], x[:, -1]], 1, generator=g)], 1)
    return x[:, :-1], x[:, 1:]

class RMSNorm(nn.Module):
    def __init__(s, d): super().__init__(); s.g = nn.Parameter(torch.ones(d))
    def forward(s, x): return s.g * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + 1e-6)

class Block(nn.Module):
    def __init__(s):
        super().__init__(); s.h, s.dh = H, D // H
        s.n1, s.n2 = RMSNorm(D), RMSNorm(D)
        s.q, s.k, s.v, s.o = (nn.Linear(D, D, bias=False) for _ in range(4))
        s.gate, s.up = nn.Linear(D, M, bias=False), nn.Linear(D, M, bias=False)
        s.down = nn.Linear(M, D, bias=False)
    def forward(s, x):
        B, t, _ = x.shape; a = s.n1(x)
        sp = lambda z: z.view(B, t, s.h, s.dh).transpose(1, 2)      # s.h shrinks after pruning
        y = F.scaled_dot_product_attention(sp(s.q(a)), sp(s.k(a)), sp(s.v(a)), is_causal=True)
        s.head_out = y                                              # (B, h, t, dh), kept for scoring
        x = x + s.o(y.transpose(1, 2).reshape(B, t, s.h * s.dh))
        a = s.n2(x); s.mlp_act = F.silu(s.gate(a)) * s.up(a)        # (B, t, M), kept for scoring
        return x + s.down(s.mlp_act)

class LM(nn.Module):
    def __init__(s):
        super().__init__(); s.emb = nn.Embedding(V, D)
        s.blocks = nn.ModuleList(Block() for _ in range(L)); s.nf = RMSNorm(D)
        s.head = nn.Linear(D, V, bias=False)
    def forward(s, x):
        h = s.emb(x)
        for b in s.blocks: h = b(h)
        return s.head(s.nf(h))

def loss_of(m, x, y): return F.cross_entropy(m(x).flatten(0, 1), y.flatten())
def nparams(m): return sum(p.numel() for p in m.parameters())

# 1) train the dense "teacher"
dense, g = LM(), torch.Generator().manual_seed(1)
opt = torch.optim.AdamW(dense.parameters(), lr=3e-3)
for step in range(800):
    x, y = batch(64, g); l = loss_of(dense, x, y)
    opt.zero_grad(); l.backward(); opt.step()
xe, ye = batch(256, torch.Generator().manual_seed(2))     # held-out evaluation set
xc, yc = batch(32, torch.Generator().manual_seed(3))      # small calibration set
with torch.no_grad(): print("dense  loss %.3f  params %d" % (loss_of(dense, xe, ye).item(), nparams(dense)))

# 2) importance of each coupled group
def scores(m, kind):
    m.zero_grad(); l = loss_of(m, xc, yc); l.backward()
    out = []
    for b in m.blocks:
        if kind == "taylor":       # LLM-Pruner, first-order (115.4): |sum over the group of g * w|
            mlp = (b.gate.weight.grad * b.gate.weight).sum(1) + (b.up.weight.grad * b.up.weight).sum(1) \
                + (b.down.weight.grad * b.down.weight).sum(0)
            hd = sum((w.weight.grad * w.weight).view(H, -1, D).sum((1, 2)) for w in (b.q, b.k, b.v)) \
               + (b.o.weight.grad * b.o.weight).view(D, H, -1).sum((0, 2))
            out.append((mlp.abs(), hd.abs()))
        elif kind == "activation": # Minitron (115.11): mean over sequence, then L2 over batch
            mlp = b.mlp_act.detach().abs().mean(1).norm(dim=0)
            hd = b.head_out.detach().norm(dim=-1).mean(2).norm(dim=0)
            out.append((mlp, hd))
        else:                      # random baseline
            gen = torch.Generator().manual_seed(7 + len(out))
            out.append((torch.rand(M, generator=gen), torch.rand(H, generator=gen)))
    return out

# 3) physically remove the groups: smaller dense matrices, no masks
def prune(model, kind, keep_mlp, keep_heads):
    m = copy.deepcopy(model)
    for b, (sm, sh) in zip(m.blocks, scores(m, kind)):
        im = sm.topk(keep_mlp).indices.sort().values
        ih = sh.topk(keep_heads).indices.sort().values
        rows = (ih[:, None] * b.dh + torch.arange(b.dh)).flatten()   # q/k/v rows of the kept heads
        with torch.no_grad():
            for name in ("gate", "up"):
                old = getattr(b, name); new = nn.Linear(D, keep_mlp, bias=False)
                new.weight.copy_(old.weight[im]); setattr(b, name, new)
            new = nn.Linear(keep_mlp, D, bias=False); new.weight.copy_(b.down.weight[:, im]); b.down = new
            for name in ("q", "k", "v"):
                old = getattr(b, name); new = nn.Linear(D, len(rows), bias=False)
                new.weight.copy_(old.weight[rows]); setattr(b, name, new)
            new = nn.Linear(len(rows), D, bias=False); new.weight.copy_(b.o.weight[:, rows]); b.o = new
        b.h = keep_heads
    m.zero_grad(set_to_none=True)
    return m

pruned = {}
for kind in ("random", "taylor", "activation"):
    pruned[kind] = prune(dense, kind, keep_mlp=88, keep_heads=6)       # half the MLP, 6 of 8 heads
    with torch.no_grad():
        print("%-10s loss %.3f  params %d" % (kind, loss_of(pruned[kind], xe, ye).item(), nparams(pruned[kind])))

# 4) recover with knowledge distillation from the dense teacher (115.13)
student = pruned["activation"]; opt = torch.optim.AdamW(student.parameters(), lr=1e-3)
g = torch.Generator().manual_seed(4)
for step in range(200):
    x, _ = batch(64, g)
    with torch.no_grad(): t_logp = F.log_softmax(dense(x), -1)
    s_logp = F.log_softmax(student(x), -1)
    kd = F.kl_div(s_logp, t_logp, log_target=True, reduction="batchmean") / T   # KL(teacher || student) per token
    opt.zero_grad(); kd.backward(); opt.step()
with torch.no_grad(): print("after distillation  loss %.3f" % loss_of(student, xe, ye).item())

Output (PyTorch 2.8, CPU):

dense  loss 2.377  params 104768
random     loss 3.052  params 62784
taylor     loss 3.004  params 62784
activation loss 2.970  params 62784
after distillation  loss 2.517

Read it carefully. The pruned model really has 40% fewer parameters (62,784 instead of 104,768): no masks, smaller matrices. Removing half of the MLP and a quarter of the heads in one shot hurts a lot with any score. The two real scores beat random, but only a little. Recovery with 200 distillation steps closes most of the gap: from 2.970 to 2.517 nats. This is the pattern of every paper in this session: the score matters, but the training after pruning matters more.

The second program checks computational invariance (115.8) and compares a PCA slice with a random one on a small residual network whose RMSNorm gains are already folded into the matrices.

import torch
torch.manual_seed(0)
V, D, Hm, L = 100, 64, 256, 3                      # vocab, hidden, MLP width, blocks
E = torch.randn(V, D) * torch.logspace(0, -2, D)   # embeddings with a few strong directions
Win  = [torch.randn(D, Hm) / D ** 0.5 for _ in range(L)]
Wout = [torch.randn(Hm, D) / Hm ** 0.5 * 0.3 for _ in range(L)]
Whead = torch.randn(D, V) / D ** 0.5

def rms(x):                                        # RMSNorm with its gain already folded into Win
    return x * D ** 0.5 / x.norm(dim=-1, keepdim=True)   # D stays the ORIGINAL width after slicing

def run(E, Win, Wout, Whead, ids):
    h = E[ids]; states = [h]
    for wi, wo in zip(Win, Wout):
        h = h + torch.nn.functional.gelu(rms(h) @ wi) @ wo
        states.append(h)
    return rms(h) @ Whead, states

ids = torch.randint(V, (4096,), generator=torch.Generator().manual_seed(1))
logits, states = run(E, Win, Wout, Whead, ids)

# 1) computational invariance (115.8): any orthogonal Q gives the same output
Q, _ = torch.linalg.qr(torch.randn(D, D))
def rotate(Q):  # W_embd Q,  Q^T W_in,  W_out Q,  Q^T W_head
    return E @ Q, [Q.T @ w for w in Win], [w @ Q for w in Wout], Q.T @ Whead
lr, _ = run(*rotate(Q), ids)
print("max |difference| after rotation: %.1e" % (lr - logits).abs().max().item())

# 2) PCA rotation (115.9): eigenvectors of C = sum of X^T X over all residual-stream signals
X = torch.cat(states)
evals, evecs = torch.linalg.eigh(X.T.double() @ X.double())
Qpca = evecs.flip(1).float()                          # largest eigenvalue first
k = int(0.75 * D)                                     # slice 25% of the hidden width
print("energy kept in first %d of %d directions: %.4f" % (k, D, (evals.flip(0)[:k].sum() / evals.sum()).item()))

# 3) slice: keep only the first k columns of Q, so every matrix gets smaller
def sliced_error(Q):
    Qk = Q[:, :k]
    ls, _ = run(E @ Qk, [Qk.T @ w for w in Win], [w @ Qk for w in Wout], Qk.T @ Whead, ids)
    return ((ls - logits).norm() / logits.norm()).item()
print("relative logit error, PCA slice:    %.4f" % sliced_error(Qpca))
print("relative logit error, random slice: %.4f" % sliced_error(Q))

Output:

max |difference| after rotation: 3.1e-06
energy kept in first 48 of 64 directions: 0.9854
relative logit error, PCA slice:    0.1645
relative logit error, random slice: 0.6592

The rotation changes the output only by float32 rounding (\(3 \times 10^{-6}\)). Slicing 25% after the PCA rotation keeps 98.5% of the energy of the hidden states and gives a 16% logit error; slicing 25% after a random rotation gives 66%. The rotation is what makes slicing work. (This toy uses one \(Q\) for all layers, so it needs no adapters.)

Comparing the results

Figure 5 puts the Sheared LLaMA results next to models trained from scratch: training tokens on the horizontal axis (log scale), average downstream accuracy on the vertical axis. All numbers are from Table 2 of the Sheared LLaMA paper.

pruned from LLaMA2-7B, then 50B tokenstrained from scratch
Figure 5. Average downstream accuracy against training tokens, from the Sheared LLaMA paper (Table 2). Circles are about 1.3B parameters, squares about 3B. Hover or tap a point for its numbers.

Note what the plot does not count: the 2T tokens that trained LLaMA2-7B in the first place. Pruning is cheap only if a good big model already exists. For a model family that is true, and that is the Minitron argument: train the biggest model once, then derive the others.

Where this is used

SettingTypical choiceWhy
Building a family of small models (8B, 4B, ...)Minitron: width pruning plus logit distillation, repeatedAbout 1% of the original tokens per smaller model; one teacher for the family
A small open model with a fixed, standard shapeSheared LLaMA: masks with a target shape, then continued pretrainingUniform layers that run with existing kernels and compare fairly with other models
Fitting a 70B model on fewer GPUs, with no training budgetSliceGPT at 20 to 30%, maybe plus quantizationNo gradients needed; dense smaller matrices; 3 instead of 4 A100 40GB for LLaMA2-70B
A quick, task-agnostic compression on one GPULLM-Pruner at about 20%, with LoRA recovery10 samples to score, a few hours to recover; quality drops quickly above 20 to 30%
MoE models that do not fit in memoryExpert pruning (115.14), maybe expert skippingMemory falls with every expert removed; Mixtral 8×7B fits on one 80 GB GPU with 6 experts
Low-latency decoding at small batchDepth pruning (BI score, 115.12)Each removed layer removes a sequential step; width pruning saves less time at batch 1 [Likely]
Video models: classification, action recognition, video retrievalPrune heads and MLP channels of the vision encoder or the LLM, then distill on video dataVideo means many frames per request; a thinner encoder cuts cost per frame; recover on the target task [Likely]
Video-RAG and key-moment detection on edge devicesA structurally pruned small LLM plus 4-bit quantizationStructured pruning and quantization combine: smaller matrices, then fewer bits per weight [Likely]

Common mistakes

  • Removing a row without its partner columns. Prune a head in \(W_Q\) but not in \(W_K\), \(W_V\) and \(W_O\), and the shapes break or the output silently changes. Always prune the whole coupled group (115.3).
  • Masking instead of slicing. Setting a head's weights to zero gives no speed-up. You must build new, smaller matrices and update the config (number of heads, intermediate size).
  • Ignoring GQA groups. Removing a key/value head without the query heads that share it breaks the model. Prune query heads inside groups, or whole groups.
  • Expecting one-shot structured pruning to be free. Unlike 50% unstructured pruning with SparseGPT, removing 20% or more of whole structures needs recovery training. Plan the training budget from the start.
  • Fine-tuning instead of distilling. With the original model available, training on its logits (115.13) recovers more per token than training on hard labels, as Minitron found.
  • Forgetting the SliceGPT adapters and the norm fold. Slicing without folding the RMSNorm gains, or without the \(Q_{\ell-1}^\top Q_\ell\) residual matrices, breaks invariance. And the adapters mean that "30% sliced" is not 30% fewer parameters.
  • Calibrating on the wrong data. All importance scores depend on the calibration set. Scoring on English web text and then serving code or another language can remove structures that matter there.
  • Comparing perplexities across papers. Context length, data split and tokenizer differ. LLM-Pruner's WikiText2 perplexity of 12.62 for LLaMA-7B and SliceGPT's 5.47 for LLaMA2-7B are different setups, not different models' quality.

Interview questions

Why does 50% unstructured sparsity not make an LLM 2× faster, and what does structured pruning change?basic
Dense GPU kernels do the same work whether a weight is zero or not, and decoding is limited by reading memory; random zeros still have to be read unless you store indices, which costs extra memory and irregular access. Only special patterns like 2:4 get hardware support, at most about 2× for the matrix multiply. Structured pruning removes whole rows and columns, so the result is a smaller dense model: fewer bytes to read and fewer FLOPs, with ordinary kernels on any hardware.
You want to remove one attention head from a Llama block. Which weights go?basic
The \(d_h\) rows of \(W_Q\), \(W_K\) and \(W_V\) that produce that head, and the \(d_h\) columns of \(W_O\) that read its output; then the config's head count drops by one. In LLaMA2-7B that is \(4 \cdot 4096 \cdot 128 = 2{,}097{,}152\) weights. With GQA, removing a query head removes only its \(W_Q\) rows and \(W_O\) columns; removing a KV head forces removing all the query heads that share it.
Why is a hidden (embedding) channel the most expensive structure to prune?core
The residual stream connects all blocks, so the hidden channel is one coupled group across the whole model: a column of every matrix that reads the stream (\(W_Q, W_K, W_V\), gate, up), a row of every matrix that writes to it (\(W_O\), down), the norms, the embedding table and the output head. In LLaMA2-7B one hidden channel is 1,645,121 weights, against 12,288 for an MLP channel. Cutting \(d\) by 25% cuts every matrix by 25%.
Explain LLM-Pruner's importance score. Why keep the gradient term?core
It estimates the loss change from zeroing a group with a Taylor expansion: \(|\mathbf{g}^\top\mathbf{w} - \tfrac12\mathbf{w}^\top\mathbf{H}\mathbf{w}|\), with the Hessian diagonal approximated by the Fisher information (squared per-sample gradients), and sums over the parts of the coupled group. In OBD/OBS the gradient term is dropped because the model sits at a minimum of the training loss. LLM-Pruner computes the loss on a few calibration texts that are not the training data, so the gradient there is not zero and carries useful signal.
How does Sheared LLaMA reach an exact target shape?advanced
Every layer, head, MLP channel and hidden channel has a hard concrete mask whose sample is exactly 0 or 1 with positive probability but still has gradients. The loss adds, for each kind of structure, \(\lambda(\sum z - T) + \phi(\sum z - T)^2\). The weights and mask parameters minimize the total; \(\lambda, \phi\) maximize it, so the penalty grows until the expected count equals the target. After 0.4B tokens, the top-scoring structures are kept to exactly match the target shape, the rest are deleted, and the model is trained further on 50B tokens.
What is dynamic batch loading and why does it help after pruning?core
Pruning damages domains unevenly: some recover fast, others stay behind. Every few steps the method measures each domain's loss gap to a reference loss, multiplies the domain weights by \(\exp(\max(\text{gap}, 0))\) and renormalizes. Tokens then go where the model is most behind, instead of being spent on domains that already recovered. In the paper it improved downstream accuracy at the same token budget [Likely].
Why is SliceGPT allowed to rotate the hidden space? Where does it fail for LayerNorm?advanced
RMSNorm only divides by the vector's length, and an orthogonal \(Q\) does not change lengths, so \(\operatorname{RMSNorm}(\mathbf{x}Q) = \operatorname{RMSNorm}(\mathbf{x})Q\) once the learned gain is folded into the next matrix. Then rotating every read matrix by \(Q^\top\) and every write matrix by \(Q\) gives the same function. LayerNorm also subtracts the mean, which does not commute with a general rotation, so SliceGPT first folds the mean subtraction into the neighboring matrices to turn the model into an RMSNorm model.
Why does SliceGPT need extra matrices on the residual path, and what do they cost?advanced
It uses a different PCA rotation \(Q_\ell\) per block, so the residual stream leaves block \(\ell-1\) in the \(Q_{\ell-1}\) basis while block \(\ell\) expects the \(Q_\ell\) basis. Multiplying by \(Q_{\ell-1}^\top Q_\ell\) converts it. After slicing, that is a \(k\times k\) matrix per residual connection, stored and multiplied at runtime. That is why LLaMA2-70B sliced by 30% still has about 51.6B parameters instead of about 48.3B.
Width or depth: which would you prune to make an 8B model into 4B?core
For quality at a given size, width: Minitron found width pruning beat depth pruning after retraining at these scales, and Minitron 8B and 4B kept all 32 layers. Depth pruning is attractive when latency at small batch matters most, because every layer is a sequential step, and it can be scored cheaply with block importance (1 minus the cosine between a layer's input and output). If you cut depth a lot, add distillation losses on hidden states, not only logits. In practice, try both with a short retraining before committing.
Describe the Minitron recipe and why it is cheap.core
Score heads, MLP neurons, hidden channels (and layers) with forward-only activation statistics on 1024 samples (mean over sequence, L2 over batch); list candidate shapes at the target size; train each for about 1.8B tokens and keep the best; then distill from the teacher with forward KL on logits for about 94B tokens. It is cheap because the student starts from good weights, and the teacher's full distributions are a richer signal than one-hot labels. Minitron 8B scored 63.8 MMLU vs 54.7 for Nemotron-3 8B trained from scratch.
How would you prune experts in an MoE model like Mixtral?advanced
Layer by layer, cache MoE inputs and outputs on calibration data, and for each subset of \(r\) out of \(n\) experts compute the output when the router may only pick from that subset; keep the subset with the smallest reconstruction error. With 8 experts that is only 28 subsets for \(r = 6\). It cuts memory almost linearly in experts (Mixtral 8×7B: 76% of memory with 6 experts) but compute per token barely changes, since each token still uses 2 experts. Combine with dynamic skipping of low-weight second experts for speed.

Exercises

  1. Using (115.1), how many parameters does LLaMA2-7B lose if you remove 8 heads in every layer (keep \(d\), \(m\), \(L\))? What fraction is that?
  2. How many MLP channels per layer must you remove from LLaMA2-7B to save the same number of parameters as one head per layer?
  3. In Figure 3, find \(\log\alpha\) where \(P(z = 1) \approx 0.5\). What is \(P(z = 0)\) there?
  4. Dynamic batch loading: weights \((0.6, 0.4)\), gaps \((0.1, 0.5)\). Compute the new weights.
  5. A hidden state cloud in 2D has covariance eigenvalues 4 and 1. What fraction of the energy does SliceGPT keep when it slices the 2D stream to 1D? What if the eigenvalues are equal?
  6. Mixtral has 32 MoE layers with 8 experts each. Lu et al. try every subset per layer. How many subsets in total for \(r = 6\)? Why do they not try all combinations across layers jointly?
Answerscheck
1. Each head costs \(4 \cdot 4096 \cdot 128 = 2{,}097{,}152\); \(8 \cdot 32 = 256\) heads give 536,870,912 parameters, 7.97% of 6,738,415,616. 2. \(2{,}097{,}152 / 12{,}288 \approx 170.7\), so about 171 MLP channels. 3. \(P(z=1) = 1 - \sigma(0.83 \ln 11 - \log\alpha) = 0.5\) when \(\log\alpha = 0.83 \ln 11 = 1.99\); then \(P(z = 0) = \sigma(-1.99 - 1.99) = 0.018\). 4. \(\alpha = (0.6e^{0.1}, 0.4e^{0.5}) = (0.663, 0.659)\); normalized \((0.501, 0.499)\). 5. \(4/(4+1) = 80\%\). With equal eigenvalues every direction is the same, so only 50% is kept, whatever the rotation: SliceGPT works only when the spectrum is uneven. 6. \(32 \cdot 28 = 896\) subsets. Jointly it would be \(28^{32}\), far too many; layer-wise reconstruction makes the search independent per layer.

Key takeaways

  • Structured pruning removes heads, MLP channels, hidden channels, layers or experts, so the result is a smaller dense model that is faster with ordinary kernels (115.1, 115.2).
  • Weights connected by a single path form coupled groups (115.3); a head is rows of \(W_Q, W_K, W_V\) plus columns of \(W_O\); a hidden channel touches every matrix in the model.
  • LLM-Pruner scores groups with a Taylor expansion of the loss (115.4) and recovers with LoRA; it works at about 20%, not at 50%.
  • Sheared LLaMA learns hard concrete masks (115.5) under a Lagrangian shape constraint (115.6), then trains 50B tokens with dynamic batch loading (115.7): a better 2.7B model than ones trained on 1T tokens.
  • SliceGPT uses RMSNorm invariance (115.8), rotates each layer with PCA (115.9) and deletes the last channels, with small adapters on the residual path (115.10).
  • Minitron scores with activations (115.11, 115.12) and recovers with logit distillation (115.13) on about 1% of the original tokens. MoE models can drop whole experts by layer-wise reconstruction (115.14).

Key formulas

\[ I(\mathbf{w}) \approx \big|\mathbf{g}^\top\mathbf{w} - \tfrac12\mathbf{w}^\top\mathbf{H}\mathbf{w}\big|, \qquad \operatorname{RMSNorm}(\mathbf{x}Q) = \operatorname{RMSNorm}(\mathbf{x})Q \]
\[ C = \textstyle\sum_i X_i^\top X_i = Q\Lambda Q^\top, \qquad \mathcal{L}_{\text{KD}} = \tfrac{1}{T}\sum_t \KL\big(p_{\text{teacher}} \,\|\, p_{\text{student}}\big) \]

References for this session

  1. Ma et al. (2023). LLM-Pruner: On the Structural Pruning of Large Language Models
  2. Xia et al. (2023). Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
  3. Ashkboos et al. (2024). SliceGPT: Compress Large Language Models by Deleting Rows and Columns
  4. Muralidharan et al. (2024). Compact Language Models via Pruning and Knowledge Distillation (Minitron)
  5. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  6. Ma et al. (2023). LLM-Pruner: On the Structural Pruning of Large Language Models
  7. Xia et al. (2023). Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
  8. Ashkboos et al. (2024). SliceGPT: Compress Large Language Models by Deleting Rows and Columns
  9. Muralidharan et al. (2024). Compact Language Models via Pruning and Knowledge Distillation (Minitron)
  10. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  11. Sreenivas, Muralidharan, Joshi et al. (2024). LLM Pruning and Distillation in Practice: The Minitron Approach
  12. Lu, Liu, Xu et al. (2024). Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  13. Louizos, Welling and Kingma (2017). Learning Sparse Neural Networks through L0 Regularization
  14. Men, Xu, Zhang et al. (2024). ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
  15. Sun et al. (2023). A Simple and Effective Pruning Approach for Large Language Models (Wanda)
  16. Frantar & Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
  17. Hinton et al. (2015). Distilling the Knowledge in a Neural Network