Unstructured and 2:4 pruning: SparseGPT, Wanda, OWL
Magnitude pruning breaks large language models at 50% sparsity, but two one-shot methods do not. SparseGPT removes weights and moves the remaining ones to cancel the error, using the same inverse-Hessian machinery as GPTQ. Wanda only multiplies each weight by the size of its input and needs no weight update at all. This session builds both, adds OWL (a different sparsity for each layer) and shows how NVIDIA's 2:4 pattern turns zeros into real speed.
What you will learn
- Why pruning an LLM is solved one layer at a time with the objective ‖WX − (M ⊙ W)X‖², and why magnitude pruning fails on LLMs
- SparseGPT: OBS-style mask selection and weight compensation, made fast with GPTQ's fixed column order, lazy blocks and a Cholesky factor
- Wanda: the score |W_ij| · ‖X_j‖₂, why it is OBS with a diagonal Hessian, and why it compares weights inside each output row
- OWL: measure outliers per layer and give layers with many outliers a lower sparsity, at the same average
- 2:4 semi-structured sparsity: the storage format, the sparse tensor core dot product, and how much speed it really gives
- How to read memory formulas for sparse weights, and why unstructured 50% sparsity saves no memory on a GPU
- What the papers measured: perplexity at 50%, 4:8, 2:4 and 70% sparsity, pruning time and speed-up
Why this matters
A pruned model has many weights set to exactly zero. Zeros need no multiplication and, in the right format, no storage. The question of this session is simple: how do we set half of the weights of a 7B or 175B model to zero, in one shot, without retraining, and without breaking the model? And when do those zeros make the GPU faster?
In session 113 we saw that the simplest rule, magnitude pruning (remove the weights with the smallest \(|w|\)), works well for small networks with retraining. For LLMs it fails badly. In the Wanda paper, 50% magnitude pruning raises the WikiText perplexity of LLaMA-7B from 5.68 to 17.29. Two methods from 2023 get the same 50% almost for free: SparseGPT reaches 7.22 and Wanda reaches 7.26. [Certain] On OPT-175B, SparseGPT at 50% sparsity even has a slightly lower perplexity than the dense model (8.21 against 8.35), while magnitude pruning gives about 43,000. [Certain]
The two methods start from the same idea and then go in opposite directions. Both care about the output of each layer, not about the weights alone. SparseGPT is the "Optimal Brain Surgeon" of session 113, made fast with the GPTQ tricks of session 106: it removes weights and moves the remaining weights to cancel the error. Wanda is the opposite: one multiplication per weight, no weight update, and hundreds of times faster to run. [Certain] Then OWL asks a third question: should every layer get the same sparsity? It says no, and it matters a lot at 70% sparsity.
Finally, zeros only help if the hardware can skip them. NVIDIA GPUs since the Ampere generation (A100) have sparse tensor cores that skip half of the weights if every group of 4 consecutive weights has at most 2 non-zeros. This is the 2:4 pattern. It is the main way that pruned LLMs get a real speed-up on GPUs today. [Likely] After this session, session 115 removes whole heads, channels and layers instead of single weights.
The layer-wise pruning problem
Retraining a 7B model after pruning costs a lot of GPU time. So, as in GPTQ, both methods solve one small problem per linear layer. Take a layer with weights \(W \in \R^{d_{out} \times d_{in}}\). Run a small calibration set (both papers use 128 sequences of web text from C4) through the model and stack the inputs of this layer as the columns of \(X \in \R^{d_{in} \times n}\). [Certain] A mask \(M\) has the same shape as \(W\): 1 means keep the weight, 0 means remove it. We want a mask with the target sparsity, and new values \(\hat W\) for the kept weights, so that the layer output does not change much:
under the constraint that a fraction \(s\) of the entries of \(M\) are 0.
- Read the symbols
- \(W, \hat W\)
- original weights and new weights of the layer (one row per output channel)
- \(M\)
- the mask; \(\odot\) is element-wise multiplication, so \(M \odot \hat W\) has zeros where the mask is 0
- \(X\)
- calibration inputs of this layer, one column per token (about 262,000 tokens: 128 × 2,048)
- \(s\)
- the sparsity: the fraction of weights that become zero (for example 0.5)
Problem (114.1) has two parts. Choosing the mask is a hard combinatorial problem. If the mask is fixed, finding \(\hat W\) is an ordinary least-squares problem, one per row. Magnitude pruning chooses the mask from \(|W|\) alone and keeps \(\hat W = W\). Wanda chooses a better mask and keeps \(\hat W = W\). SparseGPT chooses the mask and \(\hat W\) together.
As in session 106, the error of one row \(\mathbf{w}\) with change \(\delta = \hat{\mathbf{w}} - \mathbf{w}\) is exactly quadratic:
\(H\) is the Hessian of the row error. It depends only on the inputs, so all rows share it. The small \(\lambda I\) ("dampening", 1% of the mean diagonal) keeps \(H\) invertible when some input channels are almost always zero.
Why magnitude pruning fails on LLMs
The diagonal entry \(H_{jj} = 2 \lVert X_j \rVert^2\) is the energy of input channel \(j\) (\(X_j\) is row \(j\) of \(X\): the values of input feature \(j\) over all tokens). LLMs have a few outlier features: input channels whose values are 10 to 100 times larger than the others, in every layer (Dettmers et al., LLM.int8()). [Certain] A small weight on such a channel can carry a large part of the output. Magnitude pruning looks only at \(|w|\), so it removes exactly these weights. This is the main reason Wanda gives for the failure. [Certain, as the paper's explanation]
One output row has weights \(\mathbf{w} = (1.0,\ -0.6,\ 0.2,\ 0.9)\). The input norms are \(\lVert X_j \rVert = (0.5,\ 1.0,\ 8.0,\ 0.4)\): channel 3 is an outlier. Assume the input channels are uncorrelated, so the error of removing a set \(P\) of weights is \(\sum_{j \in P} w_j^2 \lVert X_j \rVert^2\). The full output energy is \(\sum_j w_j^2 \lVert X_j\rVert^2 = 0.25 + 0.36 + 2.56 + 0.1296 = 3.2996\).
Magnitude removes the two smallest \(|w|\): weights 3 (0.2) and 2 (0.6). Error \(= 2.56 + 0.36 = 2.92\), which is 88.5% of the output energy.
Wanda computes \(|w_j| \cdot \lVert X_j \rVert = (0.5,\ 0.6,\ 1.6,\ 0.36)\) and removes the two smallest: weights 4 and 1. Error \(= 0.1296 + 0.25 = 0.3796\), which is 11.5%. Same sparsity, 7.7 times less error. The tiny weight 0.2 is the most important one in the row.
SparseGPT: OBS that scales to 175B
Optimal Brain Surgeon (OBS) (Hassibi and Stork, 1993) answers this question: if we must set weight \(m\) to zero, how should the other weights move, and how much error is left? For the quadratic error (114.2), a Lagrange multiplier gives the exact answer, the same derivation as the GPTQ step with "round" replaced by "set to 0":
- Read the symbols
- \(w_m\)
- the weight we remove; after the update, its new value is exactly 0
- \(H^{-1}\)
- the inverse Hessian of the weights that are still free (\(m\) included)
- \((H^{-1})_{m,:}\)
- row \(m\) of the inverse: the direction in which all other weights move to cancel the error
- \(\varepsilon_m\)
- the OBS saliency: the error that is left after the best compensation. Small \(\varepsilon_m\) = safe to remove
Exact OBS is far too expensive for LLMs. Each row has its own mask, so each row needs its own sequence of shrinking inverse Hessians: \(O(d_{in}^3)\) per row and \(O(d_{out}\, d_{in}^3)\) per layer. The previous best method of this kind (OBC, Frantar and Alistarh, 2022) is accurate on models of 100 million parameters but cannot run on billions. [Certain]
SparseGPT (Frantar and Alistarh, 2023) makes three changes. They are the same as in GPTQ, and in fact SparseGPT and GPTQ come from the same authors, and the SparseGPT code is built on the GPTQ code. [Likely]
- One fixed column order for all rows. Go through the columns from left to right. At column \(j\), each row decides whether its weight \(j\) is kept; if not, it is set to 0 and the error is spread over the columns to its right only. Because the set of "free" columns is the same for every row, all rows share one sequence of inverse Hessians. The rows can have different masks: a pruned weight is just a weight whose target value is 0, and a kept weight has error 0.
- The Cholesky form. The row of the inverse needed at step \(j\) is row \(j\) of the upper Cholesky factor \(U\) of \(H^{-1}\) (see session 106, formula 106.11). So \(U\) is computed once.
- Lazy updates in blocks of 128 columns, applied to the rest of the matrix with one matrix multiplication.
With \(U\), the step for column \(j\) is short. Let \(m_{ij} \in \{0, 1\}\) be the mask entry of row \(i\):
- Read the symbols
- \(U\)
- upper triangular, with \(H^{-1} = U^\top U\); \(U_{jj}^2\) equals \([H_{F}^{-1}]_{jj}\) for the free set \(F = \{j, j+1, \dots\}\)
- \(e_i\)
- the scaled error of row \(i\) at column \(j\): zero if the weight is kept, \(w_{ij} / U_{jj}\) if it is removed
- \(\mathbf{w}_{i,\,j:}\)
- the part of row \(i\) from column \(j\) to the end; the update sets \(w_{ij}\) to its target and moves the later weights
- \(\text{score}_{ij}\)
- the OBS saliency (114.3) in Cholesky form, used to choose the mask
Adaptive mask selection
When is the mask chosen? Choosing it all at the start ignores how the weights move. Choosing each weight one by one is too slow. SparseGPT chooses the mask for a block of \(B_s = 128\) columns at a time: when the algorithm reaches the start of a block, it computes the score (114.4) on the already updated weights of the next 128 columns and, in each row, removes the fraction \(s\) with the smallest score. [Certain] So the mask can follow the compensation updates, and different rows and different blocks can have different sparsity, as long as the average is \(s\). For the \(n{:}m\) patterns (such as 2:4), the choice is made for every group of \(m\) columns: in each row, keep the \(n\) largest scores among the \(m\) weights.
- input: \(W \in \R^{d_{out} \times d_{in}}\), calibration inputs \(X\), sparsity \(s\) or pattern \(n{:}m\), block size \(B = 128\)
- \(H \leftarrow 2XX^\top + \lambda I\) // formula (114.2), \(\lambda\) = 1% of the mean diagonal
- \(U \leftarrow \operatorname{Cholesky}(H^{-1})\), upper triangular // computed once
- for \(b = 0, B, 2B, \dots\): // one block of columns
- if unstructured: in each row, mask the fraction \(s\) of block columns with the smallest \(w^2 / U_{jj}^2\)
- for \(j = b, \dots, b + B - 1\):
- if \(n{:}m\) and \(j\) is the first column of a group: in each row, keep the \(n\) of \(m\) columns with the largest score
- \(E_{:,j} \leftarrow (W_{:,j} - M_{:,j} \odot W_{:,j}) / U_{jj}\) // formula (114.4); 0 for kept weights
- \(W_{:,j:b+B} \leftarrow W_{:,j:b+B} - E_{:,j}\, U_{j,\,j:b+B}\) // compensate inside the block
- \(W_{:,b+B:} \leftarrow W_{:,b+B:} - E_{:,b:b+B}\, U_{b:b+B,\; b+B:}\) // lazy update, one matmul
- return \(M \odot W\)
Figure 1 runs Algorithm 1 on a tiny layer: 4 output rows, 8 inputs, and 16 calibration tokens in which inputs 2 and 6 (and inputs 3 and 7) are correlated. Every number is computed in your browser: \(H\), its inverse, the Cholesky factor \(U\), the mask scores, the errors \(e_i\) and the updates of formula (114.4). With 8 columns, one block holds the whole row, so the lazy update (line 10 of Algorithm 1) has nothing to do here.
- \(H \leftarrow 2XX^\top + \lambda I\) // (114.2), \(\lambda\) = 1% of mean diagonal
- \(U \leftarrow \operatorname{Cholesky}(H^{-1})\), upper // once
- for \(j = 1, \dots, 8\):
- if \(j\) starts a mask block: in each row keep the largest \(w^2 / U_{jj}^2\) // 2 of 4, or 4 of 8
- \(e_i \leftarrow (w_{ij} - m_{ij} w_{ij}) / U_{jj}\) for every row \(i\) // (114.4)
- \(\mathbf{w}_{i,\,j:} \leftarrow \mathbf{w}_{i,\,j:} - e_i\, U_{j,\,j:}\) // compensate
- return \(M \odot W\)
How to read Figure 1
Press Play, or Step by hand. Step 3 chooses the mask of the first group of 4 columns: for row 1 (click it to follow it in the panel) the scores \(w^2/U_{jj}^2\) are 3.26, 1.20, 0.25 and 24.72, so it keeps inputs 1 and 4. Look at input 4: its weight is small (0.35), but \(U_{44}\) is small too, so its score is the largest. Then each column takes two steps: the errors \(e_i\) (zero for kept weights), then the update, which moves every later weight of the row (green outline). The weights of later columns change before their own mask is chosen, which is what "adaptive mask" means. At the end the readout compares the error with and without updates: 2:4 gives 6.28% with updates and 9.97% with "updates off" (same scores, but no weight moves); magnitude 2:4 on the same data gives 14.30% (the purple mark on the bar). The 50% pattern with one mask block of 8 is freer: 3.61% with updates, 7.70% without, 12.27% for magnitude.
The cost is the same as GPTQ: \(O(d_{in}^3)\) for the Cholesky plus \(O(d_{out}\, d_{in}^2)\) for the updates, for the whole layer. The whole model is processed one transformer block at a time, each block calibrated on the outputs of the already pruned blocks. SparseGPT prunes OPT-175B and BLOOM-176B in about 4 hours on one A100 80 GB GPU. [Certain] The paper also observes that larger models are easier to prune: the gap to the dense model shrinks as the model grows. [Certain]
A bonus of the shared framework: the column step can prune and quantize at the same time (remove the weight, or round it to the grid). On OPT-175B, 50% sparsity plus 4-bit weights gives a perplexity of 8.29, better than 3-bit GPTQ (8.68) at a similar storage size. [Certain]
Wanda: no Hessian, no update
Wanda ("pruning by Weights and activations", Sun, Liu, Bair and Kolter, 2023) asks: how much of SparseGPT do we really need? Its answer is a single score per weight:
- Read the symbols
- \(W_{ij}\)
- the weight from input \(j\) to output \(i\)
- \(\lVert X_j \rVert_2\)
- the L2 norm of input feature \(j\) over all \(n\) calibration tokens: \(\sqrt{\sum_t x_{jt}^2}\)
- \(S_{ij}\)
- the importance of the weight; Wanda removes the weights with the smallest \(S_{ij}\) and does not change the others
Where does this score come from? Take the OBS saliency (114.3) and pretend the Hessian is diagonal (inputs uncorrelated, dampening ignored). Then \([H^{-1}]_{jj} = 1 / H_{jj} = 1 / (2 \lVert X_j\rVert^2)\), and
So Wanda is OBS with a diagonal Hessian, and with a diagonal Hessian OBS also says that the other weights should not move (the off-diagonal entries of \(H^{-1}\) are zero). Wanda is exactly right when the inputs are uncorrelated, which the worked example above assumed. In real LLMs the inputs are correlated, so SparseGPT's compensation helps, and it helps most when the pruning is hard (2:4, high sparsity). [Certain, from the paper tables below]
The comparison group matters
Which weights compete with each other? Magnitude pruning usually ranks all weights of a layer together. Wanda ranks the weights within each output row (all weights that feed the same output neuron) and removes the fraction \(s\) of every row. The paper reports that for LLMs this per-output grouping is clearly better than per-layer ranking, for Wanda and also for magnitude pruning. [Likely] One reason: with per-layer ranking, a few rows with small weights can lose almost all of their inputs, and that output neuron dies.
- input: model, 128 calibration sequences, sparsity \(s\) (or pattern \(n{:}m\))
- \(Z \leftarrow\) hidden states after the embedding layer, for all calibration tokens
- for each transformer block:
- run \(Z\) through the block with hooks; for each linear layer collect \(\lVert X_j \rVert_2\) for every input \(j\) // one forward pass
- for each linear layer: \(S \leftarrow |W| \odot \mathbf{1}\, \lVert X \rVert^\top\) // formula (114.5), broadcast over rows
- unstructured: in each row, set the fraction \(s\) of weights with the smallest \(S\) to 0
- \(n{:}m\): in each row and each group of \(m\) consecutive inputs, keep the \(n\) largest \(S\)
- \(Z \leftarrow\) output of the pruned block on \(Z\)
- return the pruned model (no weight was changed except to 0)
The cost is tiny: one forward pass and a sort per row. Computing the pruning metric takes 0.54 s for LLaMA-7B and 5.6 s for LLaMA-65B, against 203.1 s and 1353.4 s for SparseGPT (Wanda paper, same hardware). [Certain] That is about 240 to 380 times faster. Because no weights move, Wanda also works well as the first step before a short fine-tuning run, and it is easy to apply during training.
Figure 2 runs Algorithm 2 on one layer with the same 4 × 8 weights. The calibration set is small enough to show in full: 6 tokens. Input 3 is an outlier channel whose size you control, and its weights are small in every row.
- input: \(W\) (4 × 8), calibration inputs \(X\) (8 inputs × 6 tokens)
- for each input \(j\): \(\lVert X_j \rVert_2 \leftarrow \sqrt{\textstyle\sum_t x_{jt}^2}\)
- for each row \(i\): \(S_{ij} \leftarrow |W_{ij}| \cdot \lVert X_j \rVert_2\) // (114.5)
- if 2:4: for each row and group \(g\): keep the 2 largest \(S\) of inputs \(4g+1 \dots 4g+4\)
- else: for each row: keep the 4 largest \(S\) of 8
- \(\hat W \leftarrow M \odot W\) // no weight moves
How to read Figure 2
Steps 2 to 9 compute one norm each; the formula panel shows the sum of squares with the real numbers. With size ×4, input 3 has norm 6.58 while the others are between 1.72 and 2.12. Steps 10 to 13 compute the scores of one row each: row 1 gets \(S = (1.32, 0.84, 0.79, 0.64 \mid 0.49, 1.22, 0.15, 0.76)\). Then each group of 4 keeps its 2 largest scores. At the end the readout gives the output error on the 6 tokens: at ×4 and 2:4, Wanda 16.80% against magnitude 23.94%. Move the size to ×1: all inputs are similar, and Wanda picks the same 2:4 mask as magnitude (both 14.18%). Move it to ×6: now Wanda prunes 4 weights that magnitude would keep (and keeps 4 that magnitude would prune), and the errors are 20.28% against 33.14%.
Figure 3 runs all three methods on a tiny layer of 6 outputs and 8 inputs. Input 3 is an outlier channel whose size you control.
How to read Figure 3
Start at size 1: all inputs are similar, and all three methods are close; SparseGPT is a little better because it compensates (its changed weights carry a *). Slide the size of input 3 up. The weights on input 3 are small, so magnitude pruning still removes them in 5 of 6 rows, and its error grows to about 40%. Wanda and SparseGPT see the large input and keep those weights; their error falls. Switch to 2:4: the vertical line splits the inputs into two groups of 4, and every row keeps exactly 2 per group. 2:4 is harder than free 50% because good weights in the same group must compete.
What the papers measured
The Wanda paper compares all three methods on LLaMA and LLaMA-2 with the same calibration data (WikiText perplexity, lower is better). [Certain] Figure 5 in the next section plots all of them; here is the core of the table.
| Model (dense) | Pattern | Magnitude | SparseGPT | Wanda |
|---|---|---|---|---|
| LLaMA-7B (5.68) | 50% | 17.29 | 7.22 | 7.26 |
| 4:8 | 16.84 | 8.61 | 8.57 | |
| 2:4 | 42.13 | 11.00 | 11.53 | |
| LLaMA-2-13B (4.57) | 50% | 6.37 | 5.63 | 5.56 |
| 4:8 | 6.76 | 6.60 | 6.55 | |
| 2:4 | 8.33 | 8.32 | 8.27 |
Three lessons. First, at 50% unstructured both methods lose little, and Wanda is as good as SparseGPT. Second, the 2:4 pattern costs much more than free 50%: LLaMA-7B goes from 7.2 to about 11. The constraint "2 in every 4" forces the removal of some important weights. Third, on the hardest setting (2:4 on a 7B model) SparseGPT's compensation wins (11.00 against 11.53). Larger models lose less in every column. [Certain]
OWL: a different sparsity for each layer
Both SparseGPT and Wanda give every layer the same sparsity \(s\). At 50% that is fine. At 70% both break down: Wanda's LLaMA-7B perplexity is 85.77. [Certain] OWL ("Outlier Weighed Layerwise sparsity", Yin et al., 2023) notices that outlier features are not spread evenly over the depth of the model: some blocks have many more outliers than others. Its rule: blocks with more outliers keep more of their weights.
OWL first counts outliers in each layer \(l\), with the Wanda score \(A_{ij} = \lVert X_j\rVert_2 \, |W_{ij}|\):
- Read the symbols
- \(D^l\)
- the Layerwise Outlier Distribution (LOD): the fraction of weights in layer \(l\) whose score is "large"
- \(\bar A^l\)
- the mean score in layer \(l\)
- \(M\)
- the outlier threshold; the paper uses 5 for LLaMA-7B (values 3 to 10 were tried)
- \(C_{in}, C_{out}\)
- input and output size of the layer, so \(C_{in} C_{out}\) is its number of weights
Then it maps the outlier ratio to a sparsity for each transformer block (the paper found per-block better than per-linear-layer). Sparsity goes down linearly as \(D^l\) goes up, the spread between the most and least pruned block is \(2\lambda\), and the average stays at the target \(S\). [Certain] The official code does this as follows (we write it as a formula): [Likely]
Here \(\bar r\) is the mean of the \(r_l\), so the average of the \(S_l\) is exactly \(S\). The block with the most outliers gets \(S_l\) at the bottom of the range, the one with the fewest at the top. For LLaMA-7B the paper uses \(\lambda = 8\%\): at a target of 70%, the blocks range over about 16 points. [Certain] Then Wanda or SparseGPT prunes each block with its own \(S_l\). OWL costs almost nothing extra: the scores are the Wanda scores.
How to read Figure 4
With \(\lambda = 0\) all blocks get the target sparsity: this is plain Wanda or SparseGPT. Raise \(\lambda\): the blocks with tall orange bars (many outliers) get shorter purple bars (lower sparsity), and the others are pruned more. The dashed line is the target; the readout shows that the average stays on it for any \(\lambda\). The spread between the lowest and highest block is always \(2\lambda\).
The effect at 70% sparsity is large (OWL paper, Table 3, WikiText perplexity): [Certain]
| Model (dense) | Magnitude | Wanda | OWL + Wanda | SparseGPT | OWL + SparseGPT |
|---|---|---|---|---|---|
| LLaMA-7B (5.68) | 48419.12 | 85.77 | 24.55 | 26.30 | 19.49 |
| LLaMA-13B (5.09) | 84539.45 | 55.90 | 17.17 | 19.24 | 14.55 |
| LLaMA-30B | 977.73 | 17.37 | 10.75 | 12.56 | 10.28 |
How to read Figure 5
For each model, three groups of bars: 50%, 4:8 and 2:4. The grey magnitude bars are far above the dashed dense line, except on LLaMA-2-13B. SparseGPT and Wanda stay close to the line at 50% and move up as the pattern gets stricter. The OWL view uses its own colors (green = with OWL) and shows the 70% setting, where layer-wise sparsity matters most.
OWL + Wanda improves Wanda by 61.22 perplexity points on LLaMA-7B, with the same total number of zeros. Note that 70% is still far from the dense model: one-shot pruning at that level needs recovery training to be useful. [Likely] And the gain is a perplexity gain; it does not change the speed, because unstructured 70% sparsity still needs a sparse kernel (next section).
2:4 semi-structured sparsity: zeros that the GPU can skip
Unstructured sparsity puts the zeros anywhere. A GPU tensor core multiplies dense tiles of numbers; it cannot skip random zeros. To store random zeros we also need their positions, and those indices cost memory. Semi-structured (or \(n{:}m\)) sparsity is the middle ground: in every group of \(m\) consecutive weights along the input dimension, at most \(n\) are non-zero. NVIDIA Ampere and later GPUs support \(2{:}4\) in hardware: sparse tensor cores deliver twice the math throughput of dense ones for 2:4 matrices. [Certain]
The storage format is simple. In each group of 4, keep the 2 values and a 2-bit index (0 to 3) for each of them. With \(b\)-bit values:
For FP16 (\(b = 16\)) that is 9 bits per weight, 56.25% of dense. For INT8 it is 5 bits (62.5%). [Certain, arithmetic] At compute time, the indices select which entries of the input vector to read:
- Read the symbols
- \(g\)
- the group of 4 inputs, \(4g\) to \(4g + 3\)
- \(v_{i,g,k}\)
- the \(k\)-th kept value of row \(i\) in group \(g\) (stored compressed)
- \(\text{idx}_{i,g,k}\)
- its 2-bit position inside the group (the "metadata")
- \(y_i\)
- the same result as the dense dot product of the masked row, with half the multiply-adds
Step through it in Figure 6.
How to read Figure 6
Use Next and Back. The groups are along the input dimension, the one that is summed. The 2-bit codes under the compressed values are the positions 0 to 3 inside each group. In the last step only the highlighted inputs are read. The sparse result equals the dense product of the masked row exactly; the difference to the unpruned row is the pruning error, which the pruning method (Wanda, SparseGPT) tries to keep small.
How much faster is it really?
"Twice the math" does not mean "twice as fast". There are two regimes, as in session 82:
- Compute-bound (prefill, large batches): the upper limit is 2× for the matrix multiplications only. Attention, normalization and the rest do not get faster.
- Memory-bound (decoding with a small batch): the time is set by bytes read. FP16 2:4 reads 9 bits instead of 16 per weight, so the upper limit is \(16 / 9 \approx 1.78\times\) for the weight reads.
Measured numbers are lower. In the Wanda paper, the matrix multiplications of LLaMA-65B at batch size 1 run 1.54× to 1.63× faster with 2:4 (for example 3.49 ms to 2.14 ms for the q/k/v/o projections), and the end-to-end latency of LLaMA-7B goes from 312 ms to 251 ms: 1.24×. [Certain] So 2:4 gives a real but modest speed-up, at a real accuracy cost (the 2:4 row of the table above). This trade-off is why 2:4 is often combined with quantization and with a short recovery fine-tuning. [Likely]
Why unstructured 50% saves no memory on a GPU
A general sparse format must store the position of every non-zero. In CSR (compressed sparse row), each non-zero has a value and a column index; with 32-bit indices and \(b\)-bit values, the bytes per weight at sparsity \(s\) are
For FP16 the break-even is \(32/48 = 66.7\%\) sparsity: at 50%, CSR uses 3 bytes per weight, more than the 2 bytes of dense. A bitmap (1 bit per position, then the values) is better: 1.125 bytes per weight at 50%, the same as 2:4. But a bitmap needs a special kernel to decode, and on GPUs such kernels (for example Flash-LLM) pay off mainly for memory-bound decoding at high sparsity. [Likely] On CPUs, sparse engines such as DeepSparse do turn unstructured sparsity into speed: the OWL paper measures 2.6× at 70% sparsity on LLaMA-2-7B-chat (213.83 ms to 81.89 ms per token, Intel Xeon, 36 cores). [Certain]
How to read Figure 7
Move the slider. CSR starts at 6 bytes per weight at 0% sparsity (FP16 value + 4-byte index) and crosses the dense line at 66.7%. The bitmap line is always close to the values alone plus 1/8 byte. The green square is 2:4: at its fixed 50%, it costs the same as a bitmap, but the GPU can use it directly in tensor cores. Switch to INT8: the index overhead is relatively larger, so CSR needs 80% sparsity to save memory.
The linear layers of LLaMA-7B hold \(32 \times (4 \cdot 4096^2 + 3 \cdot 4096 \cdot 11008) \approx 6.48 \times 10^9\) weights. In FP16 that is 12.95 GB. In FP16 2:4 (9 bits per weight, formula 114.9) it is 7.29 GB: 5.7 GB less, 56.25% of dense. As CSR with 32-bit indices at the same 50%, it would be 19.4 GB, more than dense. Embeddings and the output layer are not pruned and stay as they were.
Pruning in PyTorch
The code builds one layer of 256 × 512 weights with 2,048 calibration tokens, 8 of whose input channels are outliers (20 times larger), and compares magnitude, Wanda and SparseGPT at 50% per row and at 2:4. The SparseGPT function follows Algorithm 1, the Wanda function follows formula (114.5). It runs on CPU in about 3 seconds; we ran it with PyTorch 2.15.
import torch torch.manual_seed(0) d_out, d_in, n_tok = 256, 512, 2048 W = torch.randn(d_out, d_in) / d_in ** 0.5 # calibration inputs with a few outlier features (as in real LLMs) X = torch.randn(d_in, n_tok) X[torch.randperm(d_in)[:8]] *= 20.0 # 8 outlier input channels at random places def rel_err(W, Wp, X): return ((W @ X - Wp @ X).norm() ** 2 / (W @ X).norm() ** 2).item() def nm_mask(score, n=2, m=4): # keep the n largest scores in every group of m consecutive inputs s = score.reshape(score.shape[0], -1, m) idx = s.topk(n, dim=-1).indices mask = torch.zeros_like(s, dtype=torch.bool).scatter_(-1, idx, True) return mask.reshape(score.shape) def row_mask(score, sparsity): # per-output comparison group: prune the same fraction in every row k = int(score.shape[1] * sparsity) idx = score.argsort(dim=1)[:, :k] return ~torch.zeros_like(score, dtype=torch.bool).scatter_(1, idx, True) def magnitude(W, X, sparsity=0.5, nm=False): s = W.abs() return W * (nm_mask(s) if nm else row_mask(s, sparsity)) def wanda(W, X, sparsity=0.5, nm=False): s = W.abs() * X.norm(dim=1)[None, :] # formula (114.5) return W * (nm_mask(s) if nm else row_mask(s, sparsity)) def sparsegpt(W, X, sparsity=0.5, nm=False, block=128, damp=0.01): W = W.clone(); d = W.shape[1] H = 2 * X @ X.T # formula (114.2) H += damp * H.diagonal().mean() * torch.eye(d) Hinv = torch.linalg.cholesky(torch.cholesky_inverse(torch.linalg.cholesky(H)), upper=True) for b0 in range(0, d, block): b1 = min(b0 + block, d); W1 = W[:, b0:b1].clone(); E = torch.zeros_like(W1) Hb = Hinv[b0:b1, b0:b1] if not nm: # adaptive mask for this block of columns score = W1 ** 2 / Hb.diagonal()[None, :] ** 2 mask = row_mask(score, sparsity) for j in range(b1 - b0): w = W1[:, j]; h = Hb[j, j] if nm and j % 4 == 0: # choose 2 of the next 4 columns, per row sc = W1[:, j:j + 4] ** 2 / Hb.diagonal()[j:j + 4][None, :] ** 2 m4 = nm_mask(sc) keep = m4[:, j % 4] if nm else mask[:, j] q = w * keep # pruned weights become 0 e = (w - q) / h # formula (114.4), scaled error W1[:, j:] -= e[:, None] * Hb[j, j:][None, :] # fix the remaining columns W1[:, j] = q; E[:, j] = e W[:, b0:b1] = W1 W[:, b1:] -= E @ Hinv[b0:b1, b1:] # lazy update of later blocks return W for name, f in [("magnitude", magnitude), ("wanda", wanda), ("sparsegpt", sparsegpt)]: u = f(W, X); s = f(W, X, nm=True) print(f"{name:10s} 50%: {rel_err(W, u, X):.4f} 2:4: {rel_err(W, s, X):.4f} zeros: {(s == 0).float().mean():.2f}") s = sparsegpt(W, X, nm=True) ok = ((s != 0).reshape(d_out, -1, 4).sum(-1) == 2).all().item() print("every group of 4 has exactly 2 non-zeros:", ok) # Output (PyTorch 2.15, CPU): # magnitude 50%: 0.0669 2:4: 0.1305 zeros: 0.50 # wanda 50%: 0.0101 2:4: 0.0187 zeros: 0.50 # sparsegpt 50%: 0.0093 2:4: 0.0167 zeros: 0.50 # every group of 4 has exactly 2 non-zeros: True
Read the output like the paper tables: magnitude is about 7 times worse than the others because it removes weights on the outlier channels; Wanda and SparseGPT are close, and SparseGPT's compensation helps more at 2:4. Here the inputs are uncorrelated apart from scale, so Wanda is almost optimal; with correlated inputs the gap to SparseGPT grows.
To run a 2:4 model fast on a GPU, the weights must be converted to the compressed format and used with a sparse kernel. PyTorch has the function to_sparse_semi_structured in torch.sparse for this (CUDA, Ampere or newer), and NVIDIA cuSPARSELt and CUTLASS provide the kernels; serving engines such as vLLM can load 2:4 checkpoints. [Likely] We did not run these here (no GPU); check the current documentation of your library version.
Where this is used
| Setting | Typical use | Why |
|---|---|---|
| Serving on A100 / H100 class GPUs | 2:4 sparse LLM weights, often with FP8 or INT4, after a short recovery fine-tuning [Likely] | Sparse tensor cores: up to 2× matmul math, about 44% less weight memory in FP16 |
| CPU and edge inference | Unstructured 50% to 80% sparsity with a sparse CPU engine | CPUs can skip arbitrary zeros; 2.6× at 70% measured in the OWL paper |
| Fast compression experiments | Wanda as a baseline that takes seconds; SparseGPT when accuracy matters more than pruning time | 0.54 s against 203.1 s for LLaMA-7B |
| High sparsity (60% to 70%) | OWL layer-wise ratios on top of Wanda or SparseGPT | Same zeros, much lower perplexity |
| VLMs (vision-language models) | Prune the language model; keep the vision encoder and the projector dense or prune them lightly [Likely] | The language model holds most weights and dominates decode time |
| Video-RAG and video QA on one GPU | A 2:4 + quantized VLM answers questions over retrieved clips | Freed weight memory goes to the long KV cache of many frame tokens |
| Video classification and action recognition | 2:4 pruning of a video transformer backbone, calibrated on in-domain clips | The input-aware score (114.5) uses real activation statistics of video tokens |
| Starting point for structured pruning and distillation | Wanda scores to rank channels or as the first step before training (session 115, session 122) | Activation-aware importance is cheap and transfers to larger units |
Common mistakes
- Expecting speed from unstructured sparsity on a GPU. Random zeros are multiplied like any other number by dense kernels. Without a sparse format and kernel, 50% unstructured sparsity gives no speed and no memory gain, and CSR at 50% even uses more memory (formula 114.11).
- Groups along the wrong dimension. The 2:4 groups must be 4 consecutive weights along the input (reduction) dimension, the one the dot product sums over. Groups along the output dimension do not match the hardware.
- Ranking all weights of a layer together. For LLMs, compare weights inside each output row (Wanda's grouping). Per-layer ranking can empty whole rows.
- Calibrating every layer on the dense model's inputs. Process the model block by block and feed each block the outputs of the already pruned blocks, so later layers see the inputs they will really get.
- Computing the Wanda norm per sequence or per batch only. \(\lVert X_j\rVert_2\) must be accumulated over all calibration tokens (sum of squares, then the square root at the end).
- Forgetting dampening in SparseGPT. Dead input channels make \(2XX^\top\) singular; the Cholesky then fails or the updates explode.
- Judging only by WikiText perplexity. Pruned models can lose more on reasoning, long-context and knowledge-heavy tasks than perplexity suggests. Evaluate on the tasks you care about. [Likely]
- Promising 2× speed for 2:4. The 2× is for the matrix math only; measured end-to-end gains are closer to 1.2× to 1.6×.
Interview questions
Why does magnitude pruning fail on LLMs while it works on CNNs?basic
What is the Wanda score, and how can you derive it from OBS?core
Explain SparseGPT. How does it relate to GPTQ?core
Why can SparseGPT use one sequence of inverse Hessians when every row has a different mask?advanced
Why does Wanda compare weights per output row instead of per layer?core
What is 2:4 sparsity, and how is it stored?basic
A colleague says "2:4 makes our model 2× faster". What do you answer?core
Why does 50% unstructured sparsity save no memory on a GPU?core
What does OWL do and why does it help at high sparsity?advanced
When would you choose SparseGPT over Wanda, and the other way round?core
Can you combine 2:4 sparsity with quantization? What is the memory?advanced
Exercises
- How many bits per weight does 2:4 need with INT4 values (ignore scales)? What fraction of dense INT4 is that?
- With FP16 values and 16-bit (instead of 32-bit) column indices, at what sparsity does CSR start to save memory?
- One row: \(\mathbf{w} = (0.3, -1.2, 0.5, 0.1)\), \(\lVert X_j\rVert = (10, 1, 2, 4)\), uncorrelated inputs. Which 2 weights do magnitude and Wanda keep under 2:4? Compute both errors \(\sum_{\text{pruned}} w_j^2 \lVert X_j\rVert^2\).
- OWL with target \(S = 70\%\), \(\lambda = 8\%\) and three blocks with LOD 9%, 5% and 1%. Compute the sparsity of each block with formula (114.8). Check the average.
- How many different 2:4 patterns does one group of 4 have? What is the minimum number of index bits per group, and how many does the hardware format use?
- A decode step reads FP16 weights and is completely memory-bound. What is the best possible speed-up from 2:4 for the weight reads? Compare with the measured 1.24× end to end and explain the gap.
Answerscheck
Key takeaways
- LLM pruning is solved layer by layer: keep \(WX\) close to \((M\odot \hat W)X\) on about 128 calibration sequences (114.1). The error is quadratic with \(H = 2XX^\top\) (114.2).
- Magnitude pruning fails because of outlier features: a small weight on a large input can matter most.
- SparseGPT = OBS (114.3) with GPTQ's fixed column order, Cholesky factor and lazy blocks (114.4), plus adaptive masks per 128 columns. 175B models in about 4 GPU hours.
- Wanda = OBS with a diagonal Hessian: score \(|W_{ij}|\,\lVert X_j\rVert_2\) (114.5, 114.6), per-output ranking, no update, seconds instead of minutes. As good as SparseGPT at 50%; slightly worse on 2:4.
- OWL gives blocks with many outliers a lower sparsity (114.7, 114.8); at 70% it cuts LLaMA-7B perplexity from 85.77 to 24.55 with Wanda.
- 2:4 is the GPU-friendly pattern: \(b/2 + 1\) bits per weight (114.9), 2× peak matmul math, about 1.2× to 1.6× measured. Unstructured sparsity needs index storage (114.11) and special kernels.
Key formulas
References for this session
- Frantar & Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
- Sun et al. (2023). A Simple and Effective Pruning Approach for Large Language Models (Wanda)
- Yin et al. (2023). Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
- Mishra et al. (2021). Accelerating Sparse Deep Neural Networks
- Frantar & Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
- Sun et al. (2023). A Simple and Effective Pruning Approach for Large Language Models (Wanda)
- Yin et al. (2023). Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
- Mishra et al. (2021). Accelerating Sparse Deep Neural Networks
- Hassibi & Stork (1993). Second order derivatives for network pruning: Optimal Brain Surgeon
- Frantar & Alistarh (2022). Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models