LLM & VLM courseBlog
114Track 5: LLM inference and serving · session 37 of 65

Unstructured and 2:4 pruning: SparseGPT, Wanda, OWL

Magnitude pruning breaks large language models at 50% sparsity, but two one-shot methods do not. SparseGPT removes weights and moves the remaining ones to cancel the error, using the same inverse-Hessian machinery as GPTQ. Wanda only multiplies each weight by the size of its input and needs no weight update at all. This session builds both, adds OWL (a different sparsity for each layer) and shows how NVIDIA's 2:4 pattern turns zeros into real speed.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Pruning basics (session 113), Weight PTQ 1: GPTQ (session 106), basic linear algebra

What you will learn

  • Why pruning an LLM is solved one layer at a time with the objective ‖WX − (M ⊙ W)X‖², and why magnitude pruning fails on LLMs
  • SparseGPT: OBS-style mask selection and weight compensation, made fast with GPTQ's fixed column order, lazy blocks and a Cholesky factor
  • Wanda: the score |W_ij| · ‖X_j‖₂, why it is OBS with a diagonal Hessian, and why it compares weights inside each output row
  • OWL: measure outliers per layer and give layers with many outliers a lower sparsity, at the same average
  • 2:4 semi-structured sparsity: the storage format, the sparse tensor core dot product, and how much speed it really gives
  • How to read memory formulas for sparse weights, and why unstructured 50% sparsity saves no memory on a GPU
  • What the papers measured: perplexity at 50%, 4:8, 2:4 and 70% sparsity, pruning time and speed-up

Why this matters

A pruned model has many weights set to exactly zero. Zeros need no multiplication and, in the right format, no storage. The question of this session is simple: how do we set half of the weights of a 7B or 175B model to zero, in one shot, without retraining, and without breaking the model? And when do those zeros make the GPU faster?

In session 113 we saw that the simplest rule, magnitude pruning (remove the weights with the smallest \(|w|\)), works well for small networks with retraining. For LLMs it fails badly. In the Wanda paper, 50% magnitude pruning raises the WikiText perplexity of LLaMA-7B from 5.68 to 17.29. Two methods from 2023 get the same 50% almost for free: SparseGPT reaches 7.22 and Wanda reaches 7.26. [Certain] On OPT-175B, SparseGPT at 50% sparsity even has a slightly lower perplexity than the dense model (8.21 against 8.35), while magnitude pruning gives about 43,000. [Certain]

The two methods start from the same idea and then go in opposite directions. Both care about the output of each layer, not about the weights alone. SparseGPT is the "Optimal Brain Surgeon" of session 113, made fast with the GPTQ tricks of session 106: it removes weights and moves the remaining weights to cancel the error. Wanda is the opposite: one multiplication per weight, no weight update, and hundreds of times faster to run. [Certain] Then OWL asks a third question: should every layer get the same sparsity? It says no, and it matters a lot at 70% sparsity.

Finally, zeros only help if the hardware can skip them. NVIDIA GPUs since the Ampere generation (A100) have sparse tensor cores that skip half of the weights if every group of 4 consecutive weights has at most 2 non-zeros. This is the 2:4 pattern. It is the main way that pruned LLMs get a real speed-up on GPUs today. [Likely] After this session, session 115 removes whole heads, channels and layers instead of single weights.

The layer-wise pruning problem

Retraining a 7B model after pruning costs a lot of GPU time. So, as in GPTQ, both methods solve one small problem per linear layer. Take a layer with weights \(W \in \R^{d_{out} \times d_{in}}\). Run a small calibration set (both papers use 128 sequences of web text from C4) through the model and stack the inputs of this layer as the columns of \(X \in \R^{d_{in} \times n}\). [Certain] A mask \(M\) has the same shape as \(W\): 1 means keep the weight, 0 means remove it. We want a mask with the target sparsity, and new values \(\hat W\) for the kept weights, so that the layer output does not change much:

\[ \min_{M,\ \hat W} \; \big\lVert W X - (M \odot \hat W)\, X \big\rVert_F^2 \tag{114.1} \]

under the constraint that a fraction \(s\) of the entries of \(M\) are 0.

Read the symbols
\(W, \hat W\)
original weights and new weights of the layer (one row per output channel)
\(M\)
the mask; \(\odot\) is element-wise multiplication, so \(M \odot \hat W\) has zeros where the mask is 0
\(X\)
calibration inputs of this layer, one column per token (about 262,000 tokens: 128 × 2,048)
\(s\)
the sparsity: the fraction of weights that become zero (for example 0.5)

Problem (114.1) has two parts. Choosing the mask is a hard combinatorial problem. If the mask is fixed, finding \(\hat W\) is an ordinary least-squares problem, one per row. Magnitude pruning chooses the mask from \(|W|\) alone and keeps \(\hat W = W\). Wanda chooses a better mask and keeps \(\hat W = W\). SparseGPT chooses the mask and \(\hat W\) together.

As in session 106, the error of one row \(\mathbf{w}\) with change \(\delta = \hat{\mathbf{w}} - \mathbf{w}\) is exactly quadratic:

\[ E(\delta) = \delta\, X X^\top \delta^\top = \tfrac12\, \delta\, H\, \delta^\top, \qquad H = 2 X X^\top + \lambda I \tag{114.2} \]

\(H\) is the Hessian of the row error. It depends only on the inputs, so all rows share it. The small \(\lambda I\) ("dampening", 1% of the mean diagonal) keeps \(H\) invertible when some input channels are almost always zero.

Why magnitude pruning fails on LLMs

The diagonal entry \(H_{jj} = 2 \lVert X_j \rVert^2\) is the energy of input channel \(j\) (\(X_j\) is row \(j\) of \(X\): the values of input feature \(j\) over all tokens). LLMs have a few outlier features: input channels whose values are 10 to 100 times larger than the others, in every layer (Dettmers et al., LLM.int8()). [Certain] A small weight on such a channel can carry a large part of the output. Magnitude pruning looks only at \(|w|\), so it removes exactly these weights. This is the main reason Wanda gives for the failure. [Certain, as the paper's explanation]

Worked example: one row, four weights

One output row has weights \(\mathbf{w} = (1.0,\ -0.6,\ 0.2,\ 0.9)\). The input norms are \(\lVert X_j \rVert = (0.5,\ 1.0,\ 8.0,\ 0.4)\): channel 3 is an outlier. Assume the input channels are uncorrelated, so the error of removing a set \(P\) of weights is \(\sum_{j \in P} w_j^2 \lVert X_j \rVert^2\). The full output energy is \(\sum_j w_j^2 \lVert X_j\rVert^2 = 0.25 + 0.36 + 2.56 + 0.1296 = 3.2996\).

Magnitude removes the two smallest \(|w|\): weights 3 (0.2) and 2 (0.6). Error \(= 2.56 + 0.36 = 2.92\), which is 88.5% of the output energy.

Wanda computes \(|w_j| \cdot \lVert X_j \rVert = (0.5,\ 0.6,\ 1.6,\ 0.36)\) and removes the two smallest: weights 4 and 1. Error \(= 0.1296 + 0.25 = 0.3796\), which is 11.5%. Same sparsity, 7.7 times less error. The tiny weight 0.2 is the most important one in the row.

SparseGPT: OBS that scales to 175B

Optimal Brain Surgeon (OBS) (Hassibi and Stork, 1993) answers this question: if we must set weight \(m\) to zero, how should the other weights move, and how much error is left? For the quadratic error (114.2), a Lagrange multiplier gives the exact answer, the same derivation as the GPTQ step with "round" replaced by "set to 0":

\[ \delta = -\,\frac{w_m}{[H^{-1}]_{mm}}\, (H^{-1})_{m,:}, \qquad \varepsilon_m = \frac{w_m^2}{2\,[H^{-1}]_{mm}} \tag{114.3} \]
Read the symbols
\(w_m\)
the weight we remove; after the update, its new value is exactly 0
\(H^{-1}\)
the inverse Hessian of the weights that are still free (\(m\) included)
\((H^{-1})_{m,:}\)
row \(m\) of the inverse: the direction in which all other weights move to cancel the error
\(\varepsilon_m\)
the OBS saliency: the error that is left after the best compensation. Small \(\varepsilon_m\) = safe to remove

Exact OBS is far too expensive for LLMs. Each row has its own mask, so each row needs its own sequence of shrinking inverse Hessians: \(O(d_{in}^3)\) per row and \(O(d_{out}\, d_{in}^3)\) per layer. The previous best method of this kind (OBC, Frantar and Alistarh, 2022) is accurate on models of 100 million parameters but cannot run on billions. [Certain]

SparseGPT (Frantar and Alistarh, 2023) makes three changes. They are the same as in GPTQ, and in fact SparseGPT and GPTQ come from the same authors, and the SparseGPT code is built on the GPTQ code. [Likely]

  • One fixed column order for all rows. Go through the columns from left to right. At column \(j\), each row decides whether its weight \(j\) is kept; if not, it is set to 0 and the error is spread over the columns to its right only. Because the set of "free" columns is the same for every row, all rows share one sequence of inverse Hessians. The rows can have different masks: a pruned weight is just a weight whose target value is 0, and a kept weight has error 0.
  • The Cholesky form. The row of the inverse needed at step \(j\) is row \(j\) of the upper Cholesky factor \(U\) of \(H^{-1}\) (see session 106, formula 106.11). So \(U\) is computed once.
  • Lazy updates in blocks of 128 columns, applied to the rest of the matrix with one matrix multiplication.

With \(U\), the step for column \(j\) is short. Let \(m_{ij} \in \{0, 1\}\) be the mask entry of row \(i\):

\[ \begin{aligned} e_{i} &= \frac{w_{ij} - m_{ij}\, w_{ij}}{U_{jj}}, \qquad \text{score}_{ij} = \frac{w_{ij}^2}{U_{jj}^2} \\ \mathbf{w}_{i,\,j:} &\leftarrow \mathbf{w}_{i,\,j:} - e_{i}\, U_{j,\,j:} \end{aligned} \tag{114.4} \]
Read the symbols
\(U\)
upper triangular, with \(H^{-1} = U^\top U\); \(U_{jj}^2\) equals \([H_{F}^{-1}]_{jj}\) for the free set \(F = \{j, j+1, \dots\}\)
\(e_i\)
the scaled error of row \(i\) at column \(j\): zero if the weight is kept, \(w_{ij} / U_{jj}\) if it is removed
\(\mathbf{w}_{i,\,j:}\)
the part of row \(i\) from column \(j\) to the end; the update sets \(w_{ij}\) to its target and moves the later weights
\(\text{score}_{ij}\)
the OBS saliency (114.3) in Cholesky form, used to choose the mask

Adaptive mask selection

When is the mask chosen? Choosing it all at the start ignores how the weights move. Choosing each weight one by one is too slow. SparseGPT chooses the mask for a block of \(B_s = 128\) columns at a time: when the algorithm reaches the start of a block, it computes the score (114.4) on the already updated weights of the next 128 columns and, in each row, removes the fraction \(s\) with the smallest score. [Certain] So the mask can follow the compensation updates, and different rows and different blocks can have different sparsity, as long as the average is \(s\). For the \(n{:}m\) patterns (such as 2:4), the choice is made for every group of \(m\) columns: in each row, keep the \(n\) largest scores among the \(m\) weights.

Algorithm 1: SparseGPT for one layer (unstructured or n:m)
  1. input: \(W \in \R^{d_{out} \times d_{in}}\), calibration inputs \(X\), sparsity \(s\) or pattern \(n{:}m\), block size \(B = 128\)
  2. \(H \leftarrow 2XX^\top + \lambda I\) // formula (114.2), \(\lambda\) = 1% of the mean diagonal
  3. \(U \leftarrow \operatorname{Cholesky}(H^{-1})\), upper triangular // computed once
  4. for \(b = 0, B, 2B, \dots\): // one block of columns
  5.   if unstructured: in each row, mask the fraction \(s\) of block columns with the smallest \(w^2 / U_{jj}^2\)
  6.   for \(j = b, \dots, b + B - 1\):
  7.     if \(n{:}m\) and \(j\) is the first column of a group: in each row, keep the \(n\) of \(m\) columns with the largest score
  8.     \(E_{:,j} \leftarrow (W_{:,j} - M_{:,j} \odot W_{:,j}) / U_{jj}\) // formula (114.4); 0 for kept weights
  9.     \(W_{:,j:b+B} \leftarrow W_{:,j:b+B} - E_{:,j}\, U_{j,\,j:b+B}\) // compensate inside the block
  10.   \(W_{:,b+B:} \leftarrow W_{:,b+B:} - E_{:,b:b+B}\, U_{b:b+B,\; b+B:}\) // lazy update, one matmul
  11. return \(M \odot W\)

Figure 1 runs Algorithm 1 on a tiny layer: 4 output rows, 8 inputs, and 16 calibration tokens in which inputs 2 and 6 (and inputs 3 and 7) are correlated. Every number is computed in your browser: \(H\), its inverse, the Cholesky factor \(U\), the mask scores, the errors \(e_i\) and the updates of formula (114.4). With 8 columns, one block holds the whole row, so the lazy update (line 10 of Algorithm 1) has nothing to do here.

  1. \(H \leftarrow 2XX^\top + \lambda I\) // (114.2), \(\lambda\) = 1% of mean diagonal
  2. \(U \leftarrow \operatorname{Cholesky}(H^{-1})\), upper // once
  3. for \(j = 1, \dots, 8\):
  4. if \(j\) starts a mask block: in each row keep the largest \(w^2 / U_{jj}^2\) // 2 of 4, or 4 of 8
  5. \(e_i \leftarrow (w_{ij} - m_{ij} w_{ij}) / U_{jj}\) for every row \(i\) // (114.4)
  6. \(\mathbf{w}_{i,\,j:} \leftarrow \mathbf{w}_{i,\,j:} - e_i\, U_{j,\,j:}\) // compensate
  7. return \(M \odot W\)
kept weightpruned (set to 0)moved by the last updatecurrent column
Figure 1. SparseGPT (Algorithm 1) on a 4 × 8 layer, one operation per step. The top strip is the diagonal \(U_{jj}\); the bottom strip is the row \(U_{j,\,j:}\) used by the update; the column on the right holds the scaled errors \(e_i\) of the current column. The bar at the bottom is the relative output error \(\lVert WX - \hat W X\rVert^2 / \lVert WX\rVert^2\) of the matrix as it is now. Click a row to follow it in the formula panel.
How to read Figure 1

Press Play, or Step by hand. Step 3 chooses the mask of the first group of 4 columns: for row 1 (click it to follow it in the panel) the scores \(w^2/U_{jj}^2\) are 3.26, 1.20, 0.25 and 24.72, so it keeps inputs 1 and 4. Look at input 4: its weight is small (0.35), but \(U_{44}\) is small too, so its score is the largest. Then each column takes two steps: the errors \(e_i\) (zero for kept weights), then the update, which moves every later weight of the row (green outline). The weights of later columns change before their own mask is chosen, which is what "adaptive mask" means. At the end the readout compares the error with and without updates: 2:4 gives 6.28% with updates and 9.97% with "updates off" (same scores, but no weight moves); magnitude 2:4 on the same data gives 14.30% (the purple mark on the bar). The 50% pattern with one mask block of 8 is freer: 3.61% with updates, 7.70% without, 12.27% for magnitude.

The cost is the same as GPTQ: \(O(d_{in}^3)\) for the Cholesky plus \(O(d_{out}\, d_{in}^2)\) for the updates, for the whole layer. The whole model is processed one transformer block at a time, each block calibrated on the outputs of the already pruned blocks. SparseGPT prunes OPT-175B and BLOOM-176B in about 4 hours on one A100 80 GB GPU. [Certain] The paper also observes that larger models are easier to prune: the gap to the dense model shrinks as the model grows. [Certain]

A bonus of the shared framework: the column step can prune and quantize at the same time (remove the weight, or round it to the grid). On OPT-175B, 50% sparsity plus 4-bit weights gives a perplexity of 8.29, better than 3-bit GPTQ (8.68) at a similar storage size. [Certain]

Wanda: no Hessian, no update

Wanda ("pruning by Weights and activations", Sun, Liu, Bair and Kolter, 2023) asks: how much of SparseGPT do we really need? Its answer is a single score per weight:

\[ S_{ij} = |W_{ij}| \cdot \lVert X_j \rVert_2 \tag{114.5} \]
Read the symbols
\(W_{ij}\)
the weight from input \(j\) to output \(i\)
\(\lVert X_j \rVert_2\)
the L2 norm of input feature \(j\) over all \(n\) calibration tokens: \(\sqrt{\sum_t x_{jt}^2}\)
\(S_{ij}\)
the importance of the weight; Wanda removes the weights with the smallest \(S_{ij}\) and does not change the others

Where does this score come from? Take the OBS saliency (114.3) and pretend the Hessian is diagonal (inputs uncorrelated, dampening ignored). Then \([H^{-1}]_{jj} = 1 / H_{jj} = 1 / (2 \lVert X_j\rVert^2)\), and

\[ \varepsilon_{ij} = \frac{w_{ij}^2}{2\,[H^{-1}]_{jj}} = w_{ij}^2\, \lVert X_j \rVert^2 = S_{ij}^2 \tag{114.6} \]

So Wanda is OBS with a diagonal Hessian, and with a diagonal Hessian OBS also says that the other weights should not move (the off-diagonal entries of \(H^{-1}\) are zero). Wanda is exactly right when the inputs are uncorrelated, which the worked example above assumed. In real LLMs the inputs are correlated, so SparseGPT's compensation helps, and it helps most when the pruning is hard (2:4, high sparsity). [Certain, from the paper tables below]

The comparison group matters

Which weights compete with each other? Magnitude pruning usually ranks all weights of a layer together. Wanda ranks the weights within each output row (all weights that feed the same output neuron) and removes the fraction \(s\) of every row. The paper reports that for LLMs this per-output grouping is clearly better than per-layer ranking, for Wanda and also for magnitude pruning. [Likely] One reason: with per-layer ranking, a few rows with small weights can lose almost all of their inputs, and that output neuron dies.

Algorithm 2: Wanda for a whole model
  1. input: model, 128 calibration sequences, sparsity \(s\) (or pattern \(n{:}m\))
  2. \(Z \leftarrow\) hidden states after the embedding layer, for all calibration tokens
  3. for each transformer block:
  4.   run \(Z\) through the block with hooks; for each linear layer collect \(\lVert X_j \rVert_2\) for every input \(j\) // one forward pass
  5.   for each linear layer: \(S \leftarrow |W| \odot \mathbf{1}\, \lVert X \rVert^\top\) // formula (114.5), broadcast over rows
  6.     unstructured: in each row, set the fraction \(s\) of weights with the smallest \(S\) to 0
  7.     \(n{:}m\): in each row and each group of \(m\) consecutive inputs, keep the \(n\) largest \(S\)
  8.   \(Z \leftarrow\) output of the pruned block on \(Z\)
  9. return the pruned model (no weight was changed except to 0)

The cost is tiny: one forward pass and a sort per row. Computing the pruning metric takes 0.54 s for LLaMA-7B and 5.6 s for LLaMA-65B, against 203.1 s and 1353.4 s for SparseGPT (Wanda paper, same hardware). [Certain] That is about 240 to 380 times faster. Because no weights move, Wanda also works well as the first step before a short fine-tuning run, and it is easy to apply during training.

Figure 2 runs Algorithm 2 on one layer with the same 4 × 8 weights. The calibration set is small enough to show in full: 6 tokens. Input 3 is an outlier channel whose size you control, and its weights are small in every row.

  1. input: \(W\) (4 × 8), calibration inputs \(X\) (8 inputs × 6 tokens)
  2. for each input \(j\): \(\lVert X_j \rVert_2 \leftarrow \sqrt{\textstyle\sum_t x_{jt}^2}\)
  3. for each row \(i\): \(S_{ij} \leftarrow |W_{ij}| \cdot \lVert X_j \rVert_2\) // (114.5)
  4. if 2:4: for each row and group \(g\): keep the 2 largest \(S\) of inputs \(4g+1 \dots 4g+4\)
  5. else: for each row: keep the 4 largest \(S\) of 8
  6. \(\hat W \leftarrow M \odot W\) // no weight moves
kept by Wandaprunedbeing computed nowmagnitude would decide differently
Figure 2. Wanda (Algorithm 2) step by step. Top: the 6 calibration tokens of each input; the orange column is the one whose norm is being computed. Middle: the norms \(\lVert X_j\rVert_2\). Bottom: the weights; once scored, each cell shows \(S_{ij}\) large and \(w_{ij}\) small. In the selection steps the purple outline marks cells where magnitude pruning would make the opposite choice.
How to read Figure 2

Steps 2 to 9 compute one norm each; the formula panel shows the sum of squares with the real numbers. With size ×4, input 3 has norm 6.58 while the others are between 1.72 and 2.12. Steps 10 to 13 compute the scores of one row each: row 1 gets \(S = (1.32, 0.84, 0.79, 0.64 \mid 0.49, 1.22, 0.15, 0.76)\). Then each group of 4 keeps its 2 largest scores. At the end the readout gives the output error on the 6 tokens: at ×4 and 2:4, Wanda 16.80% against magnitude 23.94%. Move the size to ×1: all inputs are similar, and Wanda picks the same 2:4 mask as magnitude (both 14.18%). Move it to ×6: now Wanda prunes 4 weights that magnitude would keep (and keeps 4 that magnitude would prune), and the errors are 20.28% against 33.14%.

Figure 3 runs all three methods on a tiny layer of 6 outputs and 8 inputs. Input 3 is an outlier channel whose size you control.

kept, positivekept, negativeremoved (original value, ✕)bar: the outlier input
Figure 3. A layer \(W\) of 6 × 8 random weights pruned to 50% with each method. The bars show \(\lVert X_j \rVert\) of the 32 calibration tokens; inputs 2 and 6 are correlated. The readout gives the relative output error \(\lVert WX - \hat W X\rVert^2 / \lVert WX \rVert^2\) of all three methods. Fixed seed: the numbers are repeatable.
How to read Figure 3

Start at size 1: all inputs are similar, and all three methods are close; SparseGPT is a little better because it compensates (its changed weights carry a *). Slide the size of input 3 up. The weights on input 3 are small, so magnitude pruning still removes them in 5 of 6 rows, and its error grows to about 40%. Wanda and SparseGPT see the large input and keep those weights; their error falls. Switch to 2:4: the vertical line splits the inputs into two groups of 4, and every row keeps exactly 2 per group. 2:4 is harder than free 50% because good weights in the same group must compete.

What the papers measured

The Wanda paper compares all three methods on LLaMA and LLaMA-2 with the same calibration data (WikiText perplexity, lower is better). [Certain] Figure 5 in the next section plots all of them; here is the core of the table.

Model (dense)PatternMagnitudeSparseGPTWanda
LLaMA-7B (5.68)50%17.297.227.26
4:816.848.618.57
2:442.1311.0011.53
LLaMA-2-13B (4.57)50%6.375.635.56
4:86.766.606.55
2:48.338.328.27

Three lessons. First, at 50% unstructured both methods lose little, and Wanda is as good as SparseGPT. Second, the 2:4 pattern costs much more than free 50%: LLaMA-7B goes from 7.2 to about 11. The constraint "2 in every 4" forces the removal of some important weights. Third, on the hardest setting (2:4 on a 7B model) SparseGPT's compensation wins (11.00 against 11.53). Larger models lose less in every column. [Certain]

OWL: a different sparsity for each layer

Both SparseGPT and Wanda give every layer the same sparsity \(s\). At 50% that is fine. At 70% both break down: Wanda's LLaMA-7B perplexity is 85.77. [Certain] OWL ("Outlier Weighed Layerwise sparsity", Yin et al., 2023) notices that outlier features are not spread evenly over the depth of the model: some blocks have many more outliers than others. Its rule: blocks with more outliers keep more of their weights.

OWL first counts outliers in each layer \(l\), with the Wanda score \(A_{ij} = \lVert X_j\rVert_2 \, |W_{ij}|\):

\[ D^{l} = \frac{\#\{(i, j) : A^{l}_{ij} \gt M \cdot \bar A^{l}\}}{C_{in}\, C_{out}} \tag{114.7} \]
Read the symbols
\(D^l\)
the Layerwise Outlier Distribution (LOD): the fraction of weights in layer \(l\) whose score is "large"
\(\bar A^l\)
the mean score in layer \(l\)
\(M\)
the outlier threshold; the paper uses 5 for LLaMA-7B (values 3 to 10 were tried)
\(C_{in}, C_{out}\)
input and output size of the layer, so \(C_{in} C_{out}\) is its number of weights

Then it maps the outlier ratio to a sparsity for each transformer block (the paper found per-block better than per-linear-layer). Sparsity goes down linearly as \(D^l\) goes up, the spread between the most and least pruned block is \(2\lambda\), and the average stays at the target \(S\). [Certain] The official code does this as follows (we write it as a formula): [Likely]

\[ r_l = 2\lambda\, \frac{D^l - D_{min}}{D_{max} - D_{min}}, \qquad S_l = S - \big(r_l - \bar r\big) \tag{114.8} \]

Here \(\bar r\) is the mean of the \(r_l\), so the average of the \(S_l\) is exactly \(S\). The block with the most outliers gets \(S_l\) at the bottom of the range, the one with the fewest at the top. For LLaMA-7B the paper uses \(\lambda = 8\%\): at a target of 70%, the blocks range over about 16 points. [Certain] Then Wanda or SparseGPT prunes each block with its own \(S_l\). OWL costs almost nothing extra: the scores are the Wanda scores.

sparsity of the blockoutlier ratio (LOD), scaled
Figure 4. OWL allocation (formula 114.8) for 32 transformer blocks. The outlier profile is made up for illustration: many outliers in the first blocks, a second bump near the end. It is not measured on a real model.
How to read Figure 4

With \(\lambda = 0\) all blocks get the target sparsity: this is plain Wanda or SparseGPT. Raise \(\lambda\): the blocks with tall orange bars (many outliers) get shorter purple bars (lower sparsity), and the others are pruned more. The dashed line is the target; the readout shows that the average stays on it for any \(\lambda\). The spread between the lowest and highest block is always \(2\lambda\).

The effect at 70% sparsity is large (OWL paper, Table 3, WikiText perplexity): [Certain]

Model (dense)MagnitudeWandaOWL + WandaSparseGPTOWL + SparseGPT
LLaMA-7B (5.68)48419.1285.7724.5526.3019.49
LLaMA-13B (5.09)84539.4555.9017.1719.2414.55
LLaMA-30B977.7317.3710.7512.5610.28
magnitudeSparseGPTWanda
Figure 5. WikiText perplexity (log scale) from the Wanda paper (Table 2: 50% unstructured, 4:8 and 2:4) and from the OWL paper (Table 3: LLaMA-7B at 70% unstructured). The dashed line is the dense model.
How to read Figure 5

For each model, three groups of bars: 50%, 4:8 and 2:4. The grey magnitude bars are far above the dashed dense line, except on LLaMA-2-13B. SparseGPT and Wanda stay close to the line at 50% and move up as the pattern gets stricter. The OWL view uses its own colors (green = with OWL) and shows the 70% setting, where layer-wise sparsity matters most.

OWL + Wanda improves Wanda by 61.22 perplexity points on LLaMA-7B, with the same total number of zeros. Note that 70% is still far from the dense model: one-shot pruning at that level needs recovery training to be useful. [Likely] And the gain is a perplexity gain; it does not change the speed, because unstructured 70% sparsity still needs a sparse kernel (next section).

2:4 semi-structured sparsity: zeros that the GPU can skip

Unstructured sparsity puts the zeros anywhere. A GPU tensor core multiplies dense tiles of numbers; it cannot skip random zeros. To store random zeros we also need their positions, and those indices cost memory. Semi-structured (or \(n{:}m\)) sparsity is the middle ground: in every group of \(m\) consecutive weights along the input dimension, at most \(n\) are non-zero. NVIDIA Ampere and later GPUs support \(2{:}4\) in hardware: sparse tensor cores deliver twice the math throughput of dense ones for 2:4 matrices. [Certain]

The storage format is simple. In each group of 4, keep the 2 values and a 2-bit index (0 to 3) for each of them. With \(b\)-bit values:

\[ \text{bits per weight} = \frac{2b + 2 \cdot 2}{4} = \frac{b}{2} + 1 \tag{114.9} \]

For FP16 (\(b = 16\)) that is 9 bits per weight, 56.25% of dense. For INT8 it is 5 bits (62.5%). [Certain, arithmetic] At compute time, the indices select which entries of the input vector to read:

\[ y_i = \sum_{g=0}^{d_{in}/4 - 1} \; \sum_{k=1}^{2} v_{i,g,k}\; x_{4g + \text{idx}_{i,g,k}} \tag{114.10} \]
Read the symbols
\(g\)
the group of 4 inputs, \(4g\) to \(4g + 3\)
\(v_{i,g,k}\)
the \(k\)-th kept value of row \(i\) in group \(g\) (stored compressed)
\(\text{idx}_{i,g,k}\)
its 2-bit position inside the group (the "metadata")
\(y_i\)
the same result as the dense dot product of the masked row, with half the multiply-adds

Step through it in Figure 6.

Figure 6. One row of 16 FP16 weights made 2:4 sparse (here by magnitude), compressed into 8 values plus 8 two-bit indices, and multiplied with an input vector of 16 numbers.
How to read Figure 6

Use Next and Back. The groups are along the input dimension, the one that is summed. The 2-bit codes under the compressed values are the positions 0 to 3 inside each group. In the last step only the highlighted inputs are read. The sparse result equals the dense product of the masked row exactly; the difference to the unpruned row is the pruning error, which the pruning method (Wanda, SparseGPT) tries to keep small.

How much faster is it really?

"Twice the math" does not mean "twice as fast". There are two regimes, as in session 82:

  • Compute-bound (prefill, large batches): the upper limit is 2× for the matrix multiplications only. Attention, normalization and the rest do not get faster.
  • Memory-bound (decoding with a small batch): the time is set by bytes read. FP16 2:4 reads 9 bits instead of 16 per weight, so the upper limit is \(16 / 9 \approx 1.78\times\) for the weight reads.

Measured numbers are lower. In the Wanda paper, the matrix multiplications of LLaMA-65B at batch size 1 run 1.54× to 1.63× faster with 2:4 (for example 3.49 ms to 2.14 ms for the q/k/v/o projections), and the end-to-end latency of LLaMA-7B goes from 312 ms to 251 ms: 1.24×. [Certain] So 2:4 gives a real but modest speed-up, at a real accuracy cost (the 2:4 row of the table above). This trade-off is why 2:4 is often combined with quantization and with a short recovery fine-tuning. [Likely]

Why unstructured 50% saves no memory on a GPU

A general sparse format must store the position of every non-zero. In CSR (compressed sparse row), each non-zero has a value and a column index; with 32-bit indices and \(b\)-bit values, the bytes per weight at sparsity \(s\) are

\[ \text{CSR bytes per weight} = \Big(\frac{b}{8} + 4\Big)(1 - s), \qquad \text{smaller than dense only if } s \gt \frac{32}{b + 32} \tag{114.11} \]

For FP16 the break-even is \(32/48 = 66.7\%\) sparsity: at 50%, CSR uses 3 bytes per weight, more than the 2 bytes of dense. A bitmap (1 bit per position, then the values) is better: 1.125 bytes per weight at 50%, the same as 2:4. But a bitmap needs a special kernel to decode, and on GPUs such kernels (for example Flash-LLM) pay off mainly for memory-bound decoding at high sparsity. [Likely] On CPUs, sparse engines such as DeepSparse do turn unstructured sparsity into speed: the OWL paper measures 2.6× at 70% sparsity on LLaMA-2-7B-chat (213.83 ms to 81.89 ms per token, Intel Xeon, 36 cores). [Certain]

denseCSR, 32-bit indexbitmap2:4 (fixed at 50%)
Figure 7. Storage cost per weight (formulas 114.9 and 114.11) as a function of sparsity. Index and bitmap overheads only; real kernels may add padding or tile headers.
How to read Figure 7

Move the slider. CSR starts at 6 bytes per weight at 0% sparsity (FP16 value + 4-byte index) and crosses the dense line at 66.7%. The bitmap line is always close to the values alone plus 1/8 byte. The green square is 2:4: at its fixed 50%, it costs the same as a bitmap, but the GPU can use it directly in tensor cores. Switch to INT8: the index overhead is relatively larger, so CSR needs 80% sparsity to save memory.

Worked example: memory of a 7B model in 2:4

The linear layers of LLaMA-7B hold \(32 \times (4 \cdot 4096^2 + 3 \cdot 4096 \cdot 11008) \approx 6.48 \times 10^9\) weights. In FP16 that is 12.95 GB. In FP16 2:4 (9 bits per weight, formula 114.9) it is 7.29 GB: 5.7 GB less, 56.25% of dense. As CSR with 32-bit indices at the same 50%, it would be 19.4 GB, more than dense. Embeddings and the output layer are not pruned and stay as they were.

Pruning in PyTorch

The code builds one layer of 256 × 512 weights with 2,048 calibration tokens, 8 of whose input channels are outliers (20 times larger), and compares magnitude, Wanda and SparseGPT at 50% per row and at 2:4. The SparseGPT function follows Algorithm 1, the Wanda function follows formula (114.5). It runs on CPU in about 3 seconds; we ran it with PyTorch 2.15.

import torch
torch.manual_seed(0)
d_out, d_in, n_tok = 256, 512, 2048
W = torch.randn(d_out, d_in) / d_in ** 0.5
# calibration inputs with a few outlier features (as in real LLMs)
X = torch.randn(d_in, n_tok)
X[torch.randperm(d_in)[:8]] *= 20.0      # 8 outlier input channels at random places

def rel_err(W, Wp, X):
    return ((W @ X - Wp @ X).norm() ** 2 / (W @ X).norm() ** 2).item()

def nm_mask(score, n=2, m=4):
    # keep the n largest scores in every group of m consecutive inputs
    s = score.reshape(score.shape[0], -1, m)
    idx = s.topk(n, dim=-1).indices
    mask = torch.zeros_like(s, dtype=torch.bool).scatter_(-1, idx, True)
    return mask.reshape(score.shape)

def row_mask(score, sparsity):
    # per-output comparison group: prune the same fraction in every row
    k = int(score.shape[1] * sparsity)
    idx = score.argsort(dim=1)[:, :k]
    return ~torch.zeros_like(score, dtype=torch.bool).scatter_(1, idx, True)

def magnitude(W, X, sparsity=0.5, nm=False):
    s = W.abs()
    return W * (nm_mask(s) if nm else row_mask(s, sparsity))

def wanda(W, X, sparsity=0.5, nm=False):
    s = W.abs() * X.norm(dim=1)[None, :]        # formula (114.5)
    return W * (nm_mask(s) if nm else row_mask(s, sparsity))

def sparsegpt(W, X, sparsity=0.5, nm=False, block=128, damp=0.01):
    W = W.clone(); d = W.shape[1]
    H = 2 * X @ X.T                              # formula (114.2)
    H += damp * H.diagonal().mean() * torch.eye(d)
    Hinv = torch.linalg.cholesky(torch.cholesky_inverse(torch.linalg.cholesky(H)), upper=True)
    for b0 in range(0, d, block):
        b1 = min(b0 + block, d); W1 = W[:, b0:b1].clone(); E = torch.zeros_like(W1)
        Hb = Hinv[b0:b1, b0:b1]
        if not nm:                               # adaptive mask for this block of columns
            score = W1 ** 2 / Hb.diagonal()[None, :] ** 2
            mask = row_mask(score, sparsity)
        for j in range(b1 - b0):
            w = W1[:, j]; h = Hb[j, j]
            if nm and j % 4 == 0:                # choose 2 of the next 4 columns, per row
                sc = W1[:, j:j + 4] ** 2 / Hb.diagonal()[j:j + 4][None, :] ** 2
                m4 = nm_mask(sc)
            keep = m4[:, j % 4] if nm else mask[:, j]
            q = w * keep                         # pruned weights become 0
            e = (w - q) / h                      # formula (114.4), scaled error
            W1[:, j:] -= e[:, None] * Hb[j, j:][None, :]   # fix the remaining columns
            W1[:, j] = q; E[:, j] = e
        W[:, b0:b1] = W1
        W[:, b1:] -= E @ Hinv[b0:b1, b1:]        # lazy update of later blocks
    return W

for name, f in [("magnitude", magnitude), ("wanda", wanda), ("sparsegpt", sparsegpt)]:
    u = f(W, X); s = f(W, X, nm=True)
    print(f"{name:10s} 50%: {rel_err(W, u, X):.4f}   2:4: {rel_err(W, s, X):.4f}   zeros: {(s == 0).float().mean():.2f}")

s = sparsegpt(W, X, nm=True)
ok = ((s != 0).reshape(d_out, -1, 4).sum(-1) == 2).all().item()
print("every group of 4 has exactly 2 non-zeros:", ok)

# Output (PyTorch 2.15, CPU):
# magnitude  50%: 0.0669   2:4: 0.1305   zeros: 0.50
# wanda      50%: 0.0101   2:4: 0.0187   zeros: 0.50
# sparsegpt  50%: 0.0093   2:4: 0.0167   zeros: 0.50
# every group of 4 has exactly 2 non-zeros: True

Read the output like the paper tables: magnitude is about 7 times worse than the others because it removes weights on the outlier channels; Wanda and SparseGPT are close, and SparseGPT's compensation helps more at 2:4. Here the inputs are uncorrelated apart from scale, so Wanda is almost optimal; with correlated inputs the gap to SparseGPT grows.

To run a 2:4 model fast on a GPU, the weights must be converted to the compressed format and used with a sparse kernel. PyTorch has the function to_sparse_semi_structured in torch.sparse for this (CUDA, Ampere or newer), and NVIDIA cuSPARSELt and CUTLASS provide the kernels; serving engines such as vLLM can load 2:4 checkpoints. [Likely] We did not run these here (no GPU); check the current documentation of your library version.

Where this is used

SettingTypical useWhy
Serving on A100 / H100 class GPUs2:4 sparse LLM weights, often with FP8 or INT4, after a short recovery fine-tuning [Likely]Sparse tensor cores: up to 2× matmul math, about 44% less weight memory in FP16
CPU and edge inferenceUnstructured 50% to 80% sparsity with a sparse CPU engineCPUs can skip arbitrary zeros; 2.6× at 70% measured in the OWL paper
Fast compression experimentsWanda as a baseline that takes seconds; SparseGPT when accuracy matters more than pruning time0.54 s against 203.1 s for LLaMA-7B
High sparsity (60% to 70%)OWL layer-wise ratios on top of Wanda or SparseGPTSame zeros, much lower perplexity
VLMs (vision-language models)Prune the language model; keep the vision encoder and the projector dense or prune them lightly [Likely]The language model holds most weights and dominates decode time
Video-RAG and video QA on one GPUA 2:4 + quantized VLM answers questions over retrieved clipsFreed weight memory goes to the long KV cache of many frame tokens
Video classification and action recognition2:4 pruning of a video transformer backbone, calibrated on in-domain clipsThe input-aware score (114.5) uses real activation statistics of video tokens
Starting point for structured pruning and distillationWanda scores to rank channels or as the first step before training (session 115, session 122)Activation-aware importance is cheap and transfers to larger units

Common mistakes

  • Expecting speed from unstructured sparsity on a GPU. Random zeros are multiplied like any other number by dense kernels. Without a sparse format and kernel, 50% unstructured sparsity gives no speed and no memory gain, and CSR at 50% even uses more memory (formula 114.11).
  • Groups along the wrong dimension. The 2:4 groups must be 4 consecutive weights along the input (reduction) dimension, the one the dot product sums over. Groups along the output dimension do not match the hardware.
  • Ranking all weights of a layer together. For LLMs, compare weights inside each output row (Wanda's grouping). Per-layer ranking can empty whole rows.
  • Calibrating every layer on the dense model's inputs. Process the model block by block and feed each block the outputs of the already pruned blocks, so later layers see the inputs they will really get.
  • Computing the Wanda norm per sequence or per batch only. \(\lVert X_j\rVert_2\) must be accumulated over all calibration tokens (sum of squares, then the square root at the end).
  • Forgetting dampening in SparseGPT. Dead input channels make \(2XX^\top\) singular; the Cholesky then fails or the updates explode.
  • Judging only by WikiText perplexity. Pruned models can lose more on reasoning, long-context and knowledge-heavy tasks than perplexity suggests. Evaluate on the tasks you care about. [Likely]
  • Promising 2× speed for 2:4. The 2× is for the matrix math only; measured end-to-end gains are closer to 1.2× to 1.6×.

Interview questions

Why does magnitude pruning fail on LLMs while it works on CNNs?basic
LLMs have outlier features: a few input channels with values 10 to 100 times larger than the rest. A weight's effect on the output is roughly \(|w_{ij}| \cdot \lVert X_j \rVert\), not \(|w_{ij}|\). Magnitude pruning removes small weights even when they sit on huge inputs, so it destroys important outputs. On LLaMA-7B at 50%, perplexity goes from 5.68 to 17.29 with magnitude and to 7.26 with Wanda. CNNs are usually pruned with retraining and have fewer extreme activation outliers, so magnitude works there. [Likely for the CNN part]
What is the Wanda score, and how can you derive it from OBS?core
\(S_{ij} = |W_{ij}| \cdot \lVert X_j\rVert_2\). The OBS saliency is \(w^2 / (2[H^{-1}]_{jj})\) with \(H = 2XX^\top\). If \(H\) is diagonal, \([H^{-1}]_{jj} = 1/(2\lVert X_j \rVert^2)\) and the saliency is \(w^2 \lVert X_j \rVert^2 = S_{ij}^2\). With a diagonal \(H\), OBS also says the other weights should not move, which is why Wanda does no update. Wanda is the diagonal-Hessian special case of OBS.
Explain SparseGPT. How does it relate to GPTQ?core
SparseGPT solves the layer-wise problem \(\min \lVert WX - (M\odot \hat W)X\rVert^2\). It goes through the columns in a fixed order; at each column it sets the masked weights to 0 and spreads the error over the remaining columns with the OBS update, using the Cholesky factor \(U\) of \(H^{-1}\): \(e = (w_j - m_j w_j)/U_{jj}\), \(\mathbf{w}_{j:} \mathrel{-}= e\, U_{j,j:}\). The mask is chosen per block of 128 columns with the score \(w^2/U_{jj}^2\) on the updated weights. GPTQ is the same algorithm with "round to the grid" instead of "set to 0"; both share the fixed order, lazy block updates and the Cholesky trick, and the two can be done in one pass.
Why can SparseGPT use one sequence of inverse Hessians when every row has a different mask?advanced
Exact OBS removes weights in a different order per row, so each row has its own free set and its own inverse. SparseGPT instead treats every column in the same left-to-right order for all rows: at step \(j\), every row "fixes" weight \(j\), either to 0 (pruned) or to its current value (kept, error 0), and the free set is always the columns to the right of \(j\). Because \(H\) depends only on \(X\), the free set and the inverse are the same for all rows. The mask only changes the size of the error, not which inverse is used.
Why does Wanda compare weights per output row instead of per layer?core
The paper found per-output grouping clearly better for LLMs, for both Wanda and magnitude. Each output neuron then keeps the same fraction of its inputs, so no output loses almost all of its inputs. Different rows can have different weight scales; a global ranking would prune rows with small weights much more. It also matches \(n{:}m\) patterns, which are defined per row anyway.
What is 2:4 sparsity, and how is it stored?basic
In every group of 4 consecutive weights along the input dimension, at most 2 are non-zero, so the sparsity is 50%. The hardware stores the 2 values plus a 2-bit index for each: \(b/2 + 1\) bits per weight, 9 bits for FP16 (56.25% of dense). Sparse tensor cores (NVIDIA Ampere and later) use the indices to pick the matching inputs and do half the multiply-adds, which gives twice the math throughput.
A colleague says "2:4 makes our model 2× faster". What do you answer?core
The 2× is the peak math throughput of the matrix multiplications only. In compute-bound prefill, the other operators (attention, norms, activations) do not speed up. In memory-bound decode, the gain is limited by bytes read: 16 to 9 bits per weight, at most about 1.78× on the weight reads. The Wanda paper measured 1.54× to 1.63× on LLaMA-65B matmuls at batch size 1 and 1.24× end to end on LLaMA-7B. And 2:4 costs accuracy (LLaMA-7B perplexity about 11 instead of 5.68), so it is usually combined with recovery training.
Why does 50% unstructured sparsity save no memory on a GPU?core
Each non-zero needs its position. In CSR with 32-bit column indices and FP16 values, each kept weight costs 6 bytes, so at 50% sparsity the average is 3 bytes per weight against 2 for dense; break-even is at \(32/(16+32) = 66.7\%\). A bitmap reaches 1.125 bytes but needs a custom kernel. Dense GPU kernels also do not skip zeros, so there is no compute gain either. 2:4 fixes both with a tiny fixed-size index and hardware support.
What does OWL do and why does it help at high sparsity?advanced
OWL measures, for each layer, the fraction of weights whose Wanda score exceeds \(M\) times the layer mean (the Layerwise Outlier Distribution). Blocks with more outliers get a lower sparsity, others a higher one, with a spread of \(2\lambda\) and the same average as the target. Outliers carry much of the model's function, so protecting the layers that hold many of them keeps the model working. On LLaMA-7B at 70%, Wanda goes from 85.77 to 24.55 perplexity and SparseGPT from 26.30 to 19.49.
When would you choose SparseGPT over Wanda, and the other way round?core
At 50% unstructured on large models they are about equal, and Wanda is hundreds of times cheaper (5.6 s against 1353.4 s for the metric on LLaMA-65B), needs no matrix inverse and no extra memory for \(H\). SparseGPT is worth its cost on harder settings: 2:4 on smaller models (LLaMA-7B 11.00 against 11.53), very high sparsity, or joint pruning and quantization in one pass. If a recovery fine-tuning follows, the cheaper Wanda mask is often good enough. [Likely for the last point]
Can you combine 2:4 sparsity with quantization? What is the memory?advanced
Yes. The 2:4 format stores the kept values in any supported type plus 2-bit indices, so with \(b\)-bit values the cost is \(b/2 + 1\) bits per weight: 5 bits for INT8 and 3 bits for INT4, plus quantization scales. SparseGPT can prune and quantize in the same column pass; on OPT-175B, 50% + 4-bit gives 8.29 perplexity against 8.68 for 3-bit GPTQ. Hardware support for sparse INT8 and FP8 exists on recent NVIDIA GPUs. [Likely]

Exercises

  1. How many bits per weight does 2:4 need with INT4 values (ignore scales)? What fraction of dense INT4 is that?
  2. With FP16 values and 16-bit (instead of 32-bit) column indices, at what sparsity does CSR start to save memory?
  3. One row: \(\mathbf{w} = (0.3, -1.2, 0.5, 0.1)\), \(\lVert X_j\rVert = (10, 1, 2, 4)\), uncorrelated inputs. Which 2 weights do magnitude and Wanda keep under 2:4? Compute both errors \(\sum_{\text{pruned}} w_j^2 \lVert X_j\rVert^2\).
  4. OWL with target \(S = 70\%\), \(\lambda = 8\%\) and three blocks with LOD 9%, 5% and 1%. Compute the sparsity of each block with formula (114.8). Check the average.
  5. How many different 2:4 patterns does one group of 4 have? What is the minimum number of index bits per group, and how many does the hardware format use?
  6. A decode step reads FP16 weights and is completely memory-bound. What is the best possible speed-up from 2:4 for the weight reads? Compare with the measured 1.24× end to end and explain the gap.
Answerscheck
1. \((2 \cdot 4 + 2 \cdot 2)/4 = 3\) bits per weight, 75% of dense INT4: the index overhead is relatively larger at low bit widths. 2. Bytes per weight \((2 + 2)(1 - s)\) against 2 for dense: CSR saves memory only above \(s = 50\%\). 3. Scores \(|w|\cdot\lVert X\rVert = (3.0, 1.2, 1.0, 0.4)\). Wanda keeps weights 1 and 2 and prunes 3 and 4: error \(1.0^2 + 0.4^2 = 1.16\). Magnitude keeps weights 2 and 3 (\(|w| = 1.2, 0.5\)) and prunes 1 and 4: error \(3.0^2 + 0.4^2 = 9.16\), out of a total of 11.6. 4. \(r = 2 \cdot 0.08 \cdot (D - 1)/(9 - 1) = (0.16, 0.08, 0)\), \(\bar r = 0.08\), so \(S_l = 0.70 - (r_l - 0.08) = (62\%, 70\%, 78\%)\). The block with the most outliers is pruned least; the average is 70%. 5. \(\binom{4}{2} = 6\) patterns, so at least \(\log_2 6 \approx 2.58\) bits per group; the format uses 2 indices of 2 bits = 4 bits, which is simpler to decode. 6. \(16/9 \approx 1.78\times\). The measured 1.24× is lower because attention, the KV cache reads, norms, sampling and kernel launches are not made sparse, and the sparse kernels do not reach the full bandwidth limit.

Key takeaways

  • LLM pruning is solved layer by layer: keep \(WX\) close to \((M\odot \hat W)X\) on about 128 calibration sequences (114.1). The error is quadratic with \(H = 2XX^\top\) (114.2).
  • Magnitude pruning fails because of outlier features: a small weight on a large input can matter most.
  • SparseGPT = OBS (114.3) with GPTQ's fixed column order, Cholesky factor and lazy blocks (114.4), plus adaptive masks per 128 columns. 175B models in about 4 GPU hours.
  • Wanda = OBS with a diagonal Hessian: score \(|W_{ij}|\,\lVert X_j\rVert_2\) (114.5, 114.6), per-output ranking, no update, seconds instead of minutes. As good as SparseGPT at 50%; slightly worse on 2:4.
  • OWL gives blocks with many outliers a lower sparsity (114.7, 114.8); at 70% it cuts LLaMA-7B perplexity from 85.77 to 24.55 with Wanda.
  • 2:4 is the GPU-friendly pattern: \(b/2 + 1\) bits per weight (114.9), 2× peak matmul math, about 1.2× to 1.6× measured. Unstructured sparsity needs index storage (114.11) and special kernels.

Key formulas

\[ \varepsilon_m = \frac{w_m^2}{2[H^{-1}]_{mm}}, \qquad \delta = -\frac{w_m}{[H^{-1}]_{mm}}(H^{-1})_{m,:} \]
\[ S_{ij} = |W_{ij}|\,\lVert X_j\rVert_2, \qquad S_l = S - (r_l - \bar r) \]
\[ \text{2:4 bits/weight} = \frac{b}{2} + 1, \qquad \text{CSR bytes/weight} = \Big(\frac{b}{8} + 4\Big)(1 - s) \]

References for this session

  1. Frantar & Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
  2. Sun et al. (2023). A Simple and Effective Pruning Approach for Large Language Models (Wanda)
  3. Yin et al. (2023). Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
  4. Mishra et al. (2021). Accelerating Sparse Deep Neural Networks
  5. Frantar & Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
  6. Sun et al. (2023). A Simple and Effective Pruning Approach for Large Language Models (Wanda)
  7. Yin et al. (2023). Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
  8. Mishra et al. (2021). Accelerating Sparse Deep Neural Networks
  9. Hassibi & Stork (1993). Second order derivatives for network pruning: Optimal Brain Surgeon
  10. Frantar & Alistarh (2022). Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
  11. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  12. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models