LLM & VLM courseBlog
122Track 5: LLM inference and serving · session 45 of 65

prune with Wanda (2:4), recover with distillation

In this lab you prune a real small LLM, SmolLM2-135M, with Wanda to 50% sparsity, once unstructured and once in NVIDIA’s 2:4 pattern, and measure what it costs in perplexity. You count the bytes and time sparse kernels to see which zeros give real speed. Then you win back most of the lost quality with a short distillation run from the dense model, compare it with plain fine-tuning, and draw the trade-off curve.

Time: 4 to 6 hours (reading 1.5 hours, runs and analysis the rest)Level: Intermediate · hands-on labBefore this: Sessions 114 and 119, Python and PyTorch; a laptop CPU is enough for all steps except the GPU speed test, which needs an NVIDIA GPU of the Ampere generation or newer

What you will learn

  • How to implement Wanda block by block: input norms from forward hooks, the score |W_ij|·‖X_j‖₂, a per-row comparison, and the N:M masks 2:4 and 4:8
  • How much quality a small model loses at each sparsity, measured: perplexity from 10% to 70%, magnitude against Wanda, unstructured against 4:8 and 2:4
  • Why unstructured 50% sparsity gives no speed or memory gain on a GPU, how the 2:4 format stores b/2 + 1 bits per weight, and what speed-up to expect for decode and prefill
  • How to time dense and sparse kernels honestly, and how to run PyTorch’s 2:4 semi-structured kernels on a GPU
  • How to recover a pruned model with logit distillation from the dense teacher while keeping the mask fixed, and how it compares with plain fine-tuning on the same tokens
  • How to measure recovery as the share of the log-perplexity gap that is closed, and what the recovery run costs in FLOPs and memory
  • How to report the trade-off: perplexity, bytes, measured speed, recovery budget, and simple alternatives such as quantization

Why this matters

Session 114 showed how Wanda chooses which weights to remove: it multiplies each weight by the size of the input it sees, and removes the smallest products. In this lab you do it to a real model and measure what you get. You prune a small LLM to 50% sparsity in two ways (unstructured and 2:4), measure its perplexity, find out how much faster it really runs, and then win back most of the lost quality with a short distillation run: the dense model teaches the pruned one. At the end you draw the trade-off curve and write down which model you would ship.

Session 121 kept the full model and decided per token how much of it to run. This lab goes back to the static tools, which make the model smaller once for every input, and measures two of them end to end: pruning and distillation. The lab answers three questions that come up in real projects and in interviews. Does pruning keep enough quality? Does it give real speed, or only zeros? And how much training does it take to repair the damage? The answers are often surprising. Unstructured 50% sparsity gives no speed at all on a GPU. 2:4 sparsity gives speed, but it hurts quality much more. And small models break much faster than the big models in the papers.

This is how pruning is used in practice. NVIDIA’s Minitron models were made by pruning a 15B model to 8B and 4B and then retraining them with distillation from the 15B teacher; the paper reports up to 40 times fewer training tokens per model than training from scratch [Certain]. The Wanda paper itself recovers 2:4 LLaMA-7B with LoRA or full fine-tuning on C4 [Certain]. NVIDIA GPUs since the Ampere generation have sparse tensor cores that run 2:4 matrices at twice the peak math rate [Certain].

You will:

  1. Measure the dense model: perplexity on WikiText-2 (Step 1).
  2. Prune it with Wanda and with magnitude pruning, unstructured and 2:4, and see why the input norm matters (Figure 1, Step 2).
  3. Sweep the sparsity from 10% to 70% and plot perplexity (Figure 2, Step 3).
  4. Count the bytes and measure the speed: what sparse formats store, and what a sparse kernel gains (Figure 3, Step 4).
  5. Recover with distillation from the dense model, against plain fine-tuning on the same data (Figure 4, Step 5).
  6. Draw the trade-off and decide (Figure 5, Step 6).

Setup

What we ran. We ran every step except the GPU speed test on an Apple M5 Pro laptop with PyTorch 2.8 and transformers 4.57, in FP32. The Wanda sweep ran partly on the laptop GPU (MPS) and partly on the CPU. The two 2:4 distillation runs ran on the CPU and the 50% run on the laptop GPU; after 50 steps, the GPU run and an earlier CPU run of the same job agreed to within 0.01 in perplexity. All numbers on this page come from these runs, except where we quote a paper. The GPU code for 2:4 kernels in Step 4 follows the PyTorch documentation, but we could not run it: it needs an NVIDIA GPU of the Ampere generation or newer. You will measure that part.

pip install torch "transformers>=4.45" pyarrow

Model. HuggingFaceTB/SmolLM2-135M, the base model (not the instruct one). From its config.json: a Llama-style decoder with 30 layers, hidden size 576, MLP size 1536, 9 query heads and 3 key-value heads, a vocabulary of 49,152 tokens and tied input and output embeddings [Certain]. It was pretrained on 2 trillion tokens [Certain]. We counted 134,515,008 parameters. The seven linear layers of each block (q, k, v, o, gate, up, down) hold 3,538,944 weights, so the 30 blocks hold 106,168,320 weights: 78.9% of the model. These are the weights we prune. The embedding matrix (28,311,552 weights) is not pruned, as in the Wanda paper.

Data. Three disjoint pieces of text:

  • Calibration: 128 windows of 2048 tokens from C4 (the first 4,000 documents of the English validation shard 0). This is the same amount as in the Wanda paper, which used 128 sequences from the C4 training set [Certain].
  • Recovery training: C4 documents 4,000 to 40,000 of the same shard, about 10 million tokens. We use only 819,200 of them.
  • Evaluation: the WikiText-2 test set (304,978 tokens with this tokenizer), cut into 148 windows of 2048 tokens, and the last 1,576 C4 documents of the shard as a held-out C4 set. WikiText is never used for training, so the WikiText perplexity also tests whether the repair is general.

The table you will fill. Copy it now. The last section shows our numbers.

ModelLinear sparsityWikiText-2 pplHeld-out C4 pplBytes of linear weights (FP16)Measured speed-up (GPU)
A: dense0%212 MB1.0×
B: Wanda 50% unstructured50%212 MB (dense storage)
C: Wanda 2:450%119 MB
D: C + distillation50% (2:4)119 MBsame as C
E: C + fine-tuning on data (CE)50% (2:4)119 MBsame as C
F: B + distillation50%212 MBsame as B

Step 1: the dense baseline

We measure quality with perplexity (ppl): the exponential of the average next-token loss on a text the model did not train on. A perplexity of 15 means the model is, on average, as unsure as if it had to choose among 15 equally likely tokens. Lower is better.

\[ \text{ppl} = \exp\!\Big(-\frac{1}{N} \sum_{t=1}^{N} \log p_\theta\big(x_t \mid x_{\lt t}\big)\Big) \tag{122.1} \]
Read the symbols
\(x_t\)
the token at position \(t\) of the test text
\(p_\theta(x_t \mid x_{\lt t})\)
the probability the model gives to the true next token, after reading the tokens before it
\(N\)
the number of predicted tokens: here 148 windows of 2048 tokens, each predicting 2047 tokens

We use the same protocol as the Wanda code: non-overlapping windows of 2048 tokens, so the first tokens of each window have little context. Other protocols (a sliding window, a different length) give different numbers, so compare only numbers measured the same way [Certain].

# common.py, part 1: model, data, perplexity (formula 122.1)
import gzip, json, math, torch, torch.nn as nn
import pyarrow.parquet as pq
from transformers import AutoModelForCausalLM, AutoTokenizer
torch.manual_seed(0)
MID, dev = "HuggingFaceTB/SmolLM2-135M", "cpu"
tok = AutoTokenizer.from_pretrained(MID)
LIN = ["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj", "self_attn.o_proj",
       "mlp.gate_proj", "mlp.up_proj", "mlp.down_proj"]

def load():
    return AutoModelForCausalLM.from_pretrained(MID, dtype=torch.float32).to(dev).eval()

def get(mod, name):                         # get(layer, "mlp.up_proj")
    for p in name.split("."): mod = getattr(mod, p)
    return mod

docs = [json.loads(l)["text"] for l in gzip.open("c4-validation.00000-of-00008.json.gz", "rt")]
def c4_ids(lo, hi):                         # documents lo..hi as one token stream
    ids = []
    for t in docs[lo:hi]: ids += tok(t).input_ids + [tok.eos_token_id]
    return torch.tensor(ids)

def calib(n=128, L=2048, seed=0):           # 128 random windows, as in the Wanda paper
    s = c4_ids(0, 4000); g = torch.Generator().manual_seed(seed)
    st = torch.randint(0, s.numel() - L - 1, (n,), generator=g)
    return torch.stack([s[i:i + L] for i in st])

def wiki_test():                            # the WikiText-2 (raw) test split as a parquet file
    t = "\n\n".join(pq.read_table("wikitext2_test.parquet").column("text").to_pylist())
    return tok(t, return_tensors="pt").input_ids[0]

@torch.no_grad()
def ppl(model, ids, L=2048, bs=4):          # non-overlapping windows
    n = ids.numel() // L; x = ids[:n * L].view(n, L); nll = 0.0
    for i in range(0, n, bs):
        b = x[i:i + bs].to(dev); lg = model(b).logits[:, :-1].float()
        nll += nn.functional.cross_entropy(lg.reshape(-1, lg.size(-1)), b[:, 1:].reshape(-1),
                                           reduction="sum").item()
    return math.exp(nll / (n * (L - 1)))

Our result: the dense SmolLM2-135M has a WikiText-2 perplexity of 15.43. One evaluation pass took about 50 seconds on the laptop GPU and about 4 minutes on the CPU while other jobs were running. For comparison, the Wanda paper reports 5.68 for LLaMA-7B with the same protocol [Certain]: a 50 times larger model is much more certain about the next token.

Step 2: prune with Wanda

Pruning one linear layer is a small optimization problem. The layer computes \(Y = WX\). We want a 0/1 mask \(M\) that removes a fixed share of the weights and changes the output as little as possible on real inputs:

\[ \min_{M}\ \big\lVert WX - (M \odot W) X \big\rVert_F^2 \quad \text{such that a share } s \text{ of the entries of } M \text{ is } 0 \tag{122.2} \]

Here \(\odot\) is the element-wise product and \(X\) holds the calibration inputs of the layer, one column per token. SparseGPT solves this problem with second-order information and also updates the remaining weights (session 114). Wanda uses a cheaper answer. If we keep only the diagonal of \(XX^\top\), removing weight \(W_{ij}\) adds about \(W_{ij}^2 \lVert X_j \rVert_2^2\) to the error. So the weights with the smallest Wanda score are the cheapest to remove:

\[ S_{ij} = |W_{ij}| \cdot \lVert X_j \rVert_2, \qquad \lVert X_j \rVert_2 = \Big(\sum_{t} X_{jt}^2\Big)^{1/2} \tag{122.3} \]
Read the symbols
\(W_{ij}\)
the weight from input feature \(j\) to output feature \(i\) (row \(i\), column \(j\))
\(X_{jt}\)
the value of input feature \(j\) at calibration token \(t\)
\(\lVert X_j \rVert_2\)
the size of input feature \(j\) over all calibration tokens; LLMs have a few features that are much larger than the rest (outlier features)
\(S_{ij}\)
the Wanda score; magnitude pruning uses only \(|W_{ij}|\)

Two details matter. First, Wanda compares scores inside each output row: every row of \(W\) loses the same share of its weights. The Wanda paper found this per-output comparison works better for LLMs than comparing across the whole layer, but not for image classifiers [Certain]. Second, the mask is computed one transformer block at a time, and each block receives the outputs of the blocks that are already pruned. So later blocks see the inputs they will really get.

The 2:4 pattern adds a constraint: in every group of 4 consecutive weights of a row, exactly 2 stay. Wanda then removes the 2 smallest scores in each group:

\[ \sum_{k=0}^{3} M_{i,\,4g+k} = 2 \quad \text{for every row } i \text{ and every group } g \tag{122.4} \]

This is still 50% sparsity, but the zeros cannot go where they hurt least; they must be spread evenly. 4:8 (4 of every 8 stay) is a looser version of the same idea. Figure 1 shows the difference on a toy layer.

input norm \(\|X_j\|\) of each columnkept weight (darker = larger \(|W|\))pruned weight
Figure 1. One toy linear layer with 4 output rows and 8 inputs. Input 4 is an outlier feature whose size you set. The error is the share of the output energy that is lost, computed with the diagonal approximation behind formula (122.3) (exact when the input features are uncorrelated).
How to read Figure 1

Start with Wanda, 50% per row and an outlier of 8. Wanda keeps all 4 weights of the outlier column, although they are among the smallest weights (0.15, 0.25, 0.3, 0.1), and loses 7.5% of the output. Switch to magnitude: it removes all 4 of them and loses 61.0%, eight times more. Set the outlier to 1 (no outlier): both methods choose the same mask and lose 10.5%. With an outlier of 16, magnitude loses 85.7% and Wanda 2.8%. Now pick 2:4 (the dashed line splits each row into two groups of 4): with an outlier of 8, Wanda loses 8.3% instead of 7.5%. The constraint costs a little here, because some rows must give up a weight they would have kept. In a real model the cost is much larger, as Step 3 shows.

Here is the pruning code we ran, slightly shortened (our version kept the hidden states on the CPU between blocks). It adds the masks to common.py. It never calls model(...) on the full model during pruning: it keeps the hidden states of the 128 calibration windows in memory and runs one block at a time, so each block runs twice (once to collect the input norms with forward hooks, once more after pruning to produce the inputs of the next block).

Algorithm 1: sequential Wanda pruning
  1. input: model with blocks \(1..L\), calibration tokens (\(128 \times 2048\)), pattern: share \(s\) or N:M
  2. \(H \leftarrow\) embeddings of the calibration tokens // hidden states, \(128 \times 2048 \times 576\)
  3. for block \(l = 1..L\):
  4.   run block \(l\) on \(H\); for every linear layer, accumulate \(\sum_t X_{jt}^2\) per input feature \(j\)
  5.   for each linear layer: \(S \leftarrow |W| \cdot \sqrt{\sum_t X_{jt}^2}\) (formula 122.3)
  6.     share \(s\): in each row, zero the \(\lfloor s \cdot d_{in} \rfloor\) smallest \(S_{ij}\); N:M: in each group, zero the smallest (formula 122.4)
  7.   \(H \leftarrow\) block \(l\) run again on \(H\), now pruned // the next block sees pruned outputs
  8. return the pruned model and the masks (kept for Step 5)
# common.py, part 2: masks, magnitude and Wanda pruning
def mask_rows(score, s):                   # Wanda: the same share s in every output row
    k = int(score.shape[1] * s)
    idx = score.argsort(dim=1)[:, :k]
    m = torch.ones_like(score, dtype=torch.bool); m.scatter_(1, idx, False); return m

def mask_layer(score, s):                  # magnitude baseline: compare across the whole matrix
    k = int(score.numel() * s)
    thr = score.flatten().kthvalue(k).values.item() if k > 0 else -1
    return score > thr

def mask_nm(score, n, m):                  # "2:4": prune 2 of every 4 inputs, per row
    o, i = score.shape; s = score.reshape(o, i // m, m)
    idx = s.topk(m - n, dim=-1).indices    # keep the m - n largest
    return torch.zeros_like(s, dtype=torch.bool).scatter_(-1, idx, True).reshape(o, i)

def make_mask(score, pattern, per_row):
    if isinstance(pattern, str):
        n, m = map(int, pattern.split(":")); return mask_nm(score, n, m)
    return (mask_rows if per_row else mask_layer)(score, pattern)

@torch.no_grad()
def prune_magnitude(model, pattern):
    masks = {}
    for li, layer in enumerate(model.model.layers):
        for nm in LIN:
            lin = get(layer, nm); mk = make_mask(lin.weight.abs(), pattern, per_row=False)
            lin.weight.mul_(mk); masks[f"{li}.{nm}"] = mk
    return masks

@torch.no_grad()
def prune_wanda(model, pattern, cal, bs=8):    # Algorithm 1
    m = model.model
    h = m.embed_tokens(cal.to(dev))            # (128, 2048, 576) hidden states
    pos = torch.arange(cal.shape[1], device=dev)[None]
    masks = {}
    for li, layer in enumerate(m.layers):
        sq = {nm: torch.zeros(get(layer, nm).in_features, device=dev) for nm in LIN}
        def hook(nm):
            def f(mod, inp, out):
                x = inp[0].reshape(-1, inp[0].shape[-1]).float()
                sq[nm] += (x * x).sum(0)       # sum over tokens of X_jt^2
            return f
        hooks = [get(layer, nm).register_forward_hook(hook(nm)) for nm in LIN]
        for i in range(0, h.shape[0], bs):     # pass 1: collect input norms
            x = h[i:i + bs]; layer(x, position_ids=pos, position_embeddings=m.rotary_emb(x, pos))
        for k in hooks: k.remove()
        for nm in LIN:
            lin = get(layer, nm)
            score = lin.weight.abs() * sq[nm].sqrt()[None, :]     # formula (122.3)
            mk = make_mask(score, pattern, per_row=True)
            lin.weight.mul_(mk); masks[f"{li}.{nm}"] = mk
        for i in range(0, h.shape[0], bs):     # pass 2: outputs of the pruned block
            x = h[i:i + bs]; out = layer(x, position_ids=pos, position_embeddings=m.rotary_emb(x, pos))
            h[i:i + bs] = out[0] if isinstance(out, tuple) else out
    return masks

# usage
model = load(); te = wiki_test()
print("dense", ppl(model, te))
masks = prune_wanda(model, "2:4", calib())     # or 0.5 for unstructured 50%
print("wanda 2:4", ppl(model, te))

Calling a decoder layer directly needs the rotary position embeddings (position_embeddings); with no attention mask, the default SDPA attention applies a causal mask. We checked this: for the dense model in transformers 4.57, running the 30 blocks one by one and then the final norm gives exactly the same hidden states as model(...) (largest difference 0.0). Do the same check with your version before you trust the block-by-block path. Two practical notes from our runs. The hidden states of \(128 \times 2048\) tokens take 604 MB in FP32. And kthvalue is not implemented on Apple’s MPS backend in PyTorch 2.8, so move the scores to the CPU for the magnitude baseline.

Our results at 50% (WikiText-2): Wanda unstructured 34.95, Wanda 4:8 63.66, Wanda 2:4 125.67. Magnitude pruning at 50% gives 1,622.7, and magnitude 2:4 gives 2,547.5: the model is broken. Wanda’s calibration took 81 seconds per run on the laptop GPU (two block passes over 262,144 tokens; the score itself is almost free) and 22 minutes on a busy CPU. The Wanda paper reports 0.54 seconds to compute the scores for LLaMA-7B on a GPU, without the forward passes [Certain].

Step 3: the sparsity sweep

One number at 50% hides the shape of the damage. Prune at 10%, 20%, …, 70% with both methods, plus 4:8 and 2:4, and plot perplexity on a log scale. Each Wanda point is a full new pruning run from the dense model.

# sweep.py: perplexity vs sparsity (one JSON line per run)
import json
te, cal = wiki_test(), calib()
for p in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, "4:8", "2:4"]:
    for meth in ["magnitude", "wanda"]:
        model = load()
        masks = prune_magnitude(model, p) if meth == "magnitude" else prune_wanda(model, p, cal)
        print(json.dumps({"method": meth, "pattern": p, "wiki_ppl": ppl(model, te)}))
        if meth == "wanda" and p in (0.5, "2:4"):          # the starting points of Step 5
            torch.save(model.state_dict(), f"wanda_{p}.pt".replace(":", "of"))
            torch.save(masks, f"mask_{p}.pt".replace(":", "of"))
Wandamagnitudedense model
Figure 2. Real measurements: WikiText-2 perplexity of SmolLM2-135M after one-shot pruning of all linear layers, no retraining. Dots: unstructured sparsity. At 50%, a diamond marks 4:8 and a square marks 2:4 (shifted left and right a little so they do not overlap). The y axis is logarithmic and stops at 100,000.
How to read Figure 2

Move the slider from left to right. At 10% and 20%, Wanda is almost free: 15.49 and 15.93 against 15.43 dense. Magnitude is already worse at 20% (19.24). At 40%, Wanda gives 21.60 and magnitude 70.49. At 50% (the default), Wanda gives 34.95, 2.3 times the dense value; magnitude gives 1,622.7. Above 50%, both methods break down: at 60% Wanda gives 116.37 (7.5 times dense) and magnitude 21,332; at 70% Wanda gives 2,318.7 and magnitude 934,886, which is off the chart. Now look at the squares and diamonds at 50%: the N:M patterns cost much more than unstructured sparsity at the same 50%. Wanda 4:8 gives 63.66 and Wanda 2:4 gives 125.67, 8.1 times the dense value.

Compare this with the paper. For LLaMA-7B at 50%, Wanda goes from 5.68 to 7.26 (1.28 times), 4:8 to 8.57 and 2:4 to 11.53 (2.03 times) [Certain]. Our 135M model loses far more: 2.3 times at 50% unstructured and 8.1 times at 2:4. Small models have less redundancy, so each weight carries more of the function. The SparseGPT paper made the same observation: larger models are easier to prune [Certain]. So do not take the paper’s numbers as a forecast for your small model. Measure.

Note one more comparison in Figure 2: Wanda 2:4 at 50% (125.67) is worse than Wanda unstructured at 60% (116.37). For this model, the 2:4 rule costs more than removing another 10% of the weights freely. All magnitude results at 50% are broken (1,622.7 unstructured, 3,408.1 at 4:8 and 2,547.5 at 2:4); at that level the order among them means little.

The gap between unstructured and 2:4 also has a clear reason. Unstructured 50% can remove 3 or 4 weights from a group of 4 when they are unimportant and keep all 4 elsewhere. 2:4 forces exactly 2 out of every group. In Figure 1, this cost was small; in the real model, the scores inside many groups are very uneven, and the rule removes weights that matter.

Step 4: bytes and speed

Zeros only help if the hardware or the kernel can skip them. Two things can get faster. In decode (one new token per request), the time is mostly spent reading the weights from memory, so fewer bytes means more speed (session 82). In prefill and large batches, the time is spent on multiplications, so fewer multiplications means more speed.

Unstructured 50% on a GPU. The zeros are stored as zeros in a dense matrix, so the bytes and the math are the same as before: no speed-up. A sparse format such as CSR stores only the non-zero values plus a column index for each one. With FP16 values and 16-bit indices, that is \(2 \times 16 \times 0.5 = 16\) bits per original weight: no memory saving at 50%. With 32-bit indices it is 24 bits: more memory than dense. Sparse kernels for unstructured patterns are also much less efficient than dense tensor-core kernels at this sparsity [Likely].

2:4 on a GPU. Each group of 4 stores its 2 kept values and two 2-bit positions (the metadata). For values with \(b\) bits:

\[ \text{bits per weight}_{2:4} = \frac{2b + 2 \cdot 2}{4} = \frac{b}{2} + 1 \qquad (b = 16:\ 9 \text{ bits};\ b = 4:\ 3 \text{ bits}) \tag{122.5} \]

PyTorch documents the same numbers: a 2:4 FP16 or BF16 tensor takes 9/16 = 56.25% of the dense size, and INT8 takes 62.5% [Certain]. If decode were purely limited by weight reads, the best possible speed-up of the linear layers would be the ratio of bytes:

\[ \text{speed-up}_\text{decode} \le \frac{\text{bits}_\text{dense}}{\text{bits}_\text{sparse}} = \frac{16}{9} \approx 1.78, \qquad \text{speed-up}_\text{prefill} \le 2 \tag{122.6} \]

The 2 is the peak math rate of the sparse tensor cores against the dense ones [Certain]. Real numbers are lower, because attention, norms, the KV cache and the other operators do not get faster. The Wanda paper measured 2:4 matrix multiplications of LLaMA-65B at 1.54 to 1.63 times the dense speed (for example 3.49 ms to 2.14 ms for the q/k/v/o projections) with the CUTLASS kernels on an A6000 GPU, and an end-to-end latency of 251 ms instead of 312 ms for LLaMA-7B: 1.24 times [Certain].

Worked example: bytes of the linear layers

SmolLM2-135M has 106,168,320 linear weights. Dense FP16: \(106{,}168{,}320 \times 2 = 212.3\) MB. 2:4 FP16: \(\times 9/16 = 119.4\) MB, a saving of 92.9 MB. The embedding matrix and the norms (56.7 MB in FP16) stay dense, so the whole model goes from 269.0 MB to 176.1 MB: 1.53 times smaller, not 1.78 times.

LLaMA-7B has 32 blocks of \(4 \times 4096^2 + 3 \times 4096 \times 11008 = 202{,}375{,}168\) linear weights: 6,476,005,376 in total. Dense FP16: 12.95 GB. 2:4 FP16: 7.29 GB. 2:4 with 4-bit values (3 bits per weight): 2.43 GB, against 3.24 GB for dense INT4. Combined with quantization, 2:4 saves a smaller share (4/3 = 1.33 times), because the 2-bit metadata is a larger part of each weight.

On the GPU you will measure. The code below converts each linear weight of the pruned model to PyTorch’s 2:4 format and times the decode of a few tokens. We could not run it (no NVIDIA GPU). PyTorch calls this feature a prototype, supports it only on CUDA, and needs 2D FP16/BF16 matrices whose sizes are multiples of 64 [Certain]. SmolLM2’s matrices (576, 192 and 1536 wide) meet this.

# speed_24.py: 2:4 kernels on an NVIDIA GPU (Ampere or newer). NOT run by us.
import time, torch
from torch.sparse import to_sparse_semi_structured
model = load().half().cuda()               # with dev = "cuda" in common.py
model.load_state_dict(torch.load("wanda_2of4.pt"))   # the Wanda 2:4 model from Step 3
model.half()
x = torch.randint(0, 49152, (1, 1), device="cuda")

def bench(m, reps=200):                     # decode-like: one token, batch 1, no KV cache
    with torch.no_grad():
        for _ in range(20): m(x)
        torch.cuda.synchronize(); t = time.perf_counter()
        for _ in range(reps): m(x)
        torch.cuda.synchronize(); return (time.perf_counter() - t) / reps * 1e3

t_dense = bench(model)                       # the zeros are still stored densely here
for layer in model.model.layers:
    for nm in LIN:
        lin = get(layer, nm)
        lin.weight = torch.nn.Parameter(to_sparse_semi_structured(lin.weight))
t_24 = bench(model)
print(f"dense storage {t_dense:.3f} ms   2:4 kernels {t_24:.3f} ms   speed-up {t_dense / t_24:.2f}x")
# also time a prefill: x = torch.randint(0, 49152, (8, 512), device="cuda")

A 135M model is a hard case for this test. Its matrices are small, so kernel launch time and the other operators dominate, and the speed-up will be smaller than for a 7B model [Likely]. Time a large batch too (prefill, \(8 \times 512\) tokens): that is where the math speed of the sparse tensor cores shows.

What we could measure: CPU sparse kernels. On the CPU we timed one matrix multiplication with a dense FP32 matrix against the same matrix in PyTorch’s CSR format at 50% sparsity, once with random (unstructured) zeros and once with a 2:4 mask. Figure 3 shows the result next to the byte counts.

# bench.py: dense vs CSR at 50% on the CPU (FP32, 8 threads, best of 5)
import time, torch
torch.manual_seed(0); torch.set_num_threads(8)
def nm_mask(w):
    s = w.abs().reshape(w.shape[0], -1, 4); idx = s.topk(2, -1).indices
    return torch.zeros_like(s, dtype=torch.bool).scatter_(-1, idx, True).reshape(w.shape)
def tm(f, reps):
    for _ in range(3): f()
    best = 1e9
    for _ in range(5):
        t = time.perf_counter()
        for _ in range(reps): f()
        best = min(best, (time.perf_counter() - t) / reps)
    return best * 1e3
for name, (o, i) in {"SmolLM2 up_proj": (1536, 576), "LLaMA-7B up_proj": (11008, 4096)}.items():
    W = torch.randn(o, i); Wu = W * (torch.rand(o, i) > 0.5); W24 = W * nm_mask(W)
    cu, c24 = Wu.to_sparse_csr(), W24.to_sparse_csr()
    for T in [1, 16, 512]:
        X = torch.randn(i, T); reps = 50 if o * i * T < 1e9 else 5
        print(name, T, "dense %.4f ms" % tm(lambda: W @ X, reps),
              "csr50 %.4f ms" % tm(lambda: cu @ X, reps), "csr24 %.4f ms" % tm(lambda: c24 @ X, reps))
dense storageCSR (values + column indices)2:4 (values + 2-bit metadata)
Figure 3. Top: bits per linear weight and total size for each format, from formula (122.5); “\(\le\)” is the best decode speed-up from bytes alone (122.6). Bottom: our measured CPU time of one up-projection matrix multiplication (SmolLM2: \(1536 \times 576\); LLaMA-7B: \(11008 \times 4096\)) for a batch of 1, 16 or 512 tokens. The CPU numbers come from a busy laptop and can vary by tens of percent between runs.
How to read Figure 3

The top part does not change with the token buttons: memory is memory. Only 2:4 saves bytes at 50%; CSR with 16-bit indices is as big as dense, and with 32-bit indices it is bigger. Now look at the measured times. For SmolLM2 with 1 token, the dense matmul takes 0.0064 ms and CSR at 50% takes 2.85 ms: 445 times slower. With 512 tokens the gap shrinks to 31 times, and for the LLaMA-7B shape it is 24 to 155 times. The 2:4 mask makes no difference to CSR (2.86 ms): a general sparse format does not know about the pattern, so it cannot use it. This is why “50% sparse” alone says nothing about speed: it depends on the pattern, the format and the kernel.

What our CPU numbers show. On this laptop, CSR at 50% was 24 to 445 times slower than dense in all six cases. Two reasons. First, the dense matmul is very well tuned: it reached about 1.7 trillion FP32 operations per second at 512 tokens (Apple’s matrix unit through the Accelerate library [Likely]), and the small SmolLM2 matrix (3.5 MB) stays in the CPU cache. Second, PyTorch on this ARM laptop has no MKL, and its CSR kernel ran on one core: its time did not change when we went from 1 to 8 threads. On an x86 server with MKL the gap would be smaller, but published results agree that general sparse kernels need much more than 50% sparsity to beat dense ones [Likely]. So the lesson holds: zeros give speed only with a format and a kernel built for them. On a GPU that means 2:4 with sparse tensor cores, which you measure with speed_24.py.

Step 5: recover with distillation

The pruned model has lost quality, but most of its knowledge is still there. A short training run on the remaining weights can repair much of the damage. Two choices matter: what loss to use, and how to keep the pruned weights at zero.

The loss. Plain fine-tuning uses the next-token cross-entropy (CE) on text. Knowledge distillation (KD) uses the dense model as a teacher: at every position, the pruned student is trained to match the teacher’s full probability distribution over the next token (session 59, session 119). The teacher is the model before pruning, so its distribution is exactly the behavior we want back.

\[ \mathcal{L}_\text{KD} = \frac{1}{N} \sum_{t=1}^{N} \KL\big(p_T(\cdot \mid x_{\lt t}) \,\big\|\, p_S(\cdot \mid x_{\lt t})\big) = \frac{1}{N} \sum_{t=1}^{N} \sum_{v=1}^{V} p_T(v \mid x_{\lt t}) \log \frac{p_T(v \mid x_{\lt t})}{p_S(v \mid x_{\lt t})} \tag{122.7} \]
Read the symbols
\(p_T, p_S\)
the next-token distributions of the dense teacher and the pruned student (softmax at temperature 1)
\(V\)
the vocabulary size, 49,152
\(N\)
the number of token positions in the batch
\(\KL\)
the Kullback–Leibler divergence; it is 0 only when the student matches the teacher exactly

Why should this beat CE? A CE target is one token; a teacher target is 49,152 probabilities, so each position carries much more signal. And CE pulls the student toward the data, which is a different target from the teacher it was cut from. Minitron found that retraining a pruned model with the logit KL loss alone, at temperature 1, worked better than conventional training at the same compute [Certain]. We test it on our model.

Keeping the zeros. After every optimizer step, multiply each pruned weight matrix by its mask again:

\[ W \leftarrow M \odot \big(W - \eta\, u(W)\big) \tag{122.8} \]

Here \(u(W)\) is the AdamW update and \(\eta\) the learning rate. Without this line, the gradients would make the pruned weights non-zero again, and the sparsity (and with it any 2:4 speed) would be gone. The mask stays fixed: we only train the weights that survived, as in the Wanda paper [Certain]. The embedding matrix and the norms are trained too; they are dense anyway.

Measuring recovery. To compare runs that start from different points, we report the share of the damage that was repaired, in log-perplexity (which is the average loss):

\[ \text{gap closed} = \frac{\log \text{ppl}_\text{pruned} - \log \text{ppl}_\text{recovered}}{\log \text{ppl}_\text{pruned} - \log \text{ppl}_\text{dense}} \tag{122.9} \]

1 means the student is as good as the teacher; 0 means no repair.

Algorithm 2: recovery with distillation and a fixed mask
  1. input: pruned student \(S\) with masks \(M\), dense teacher \(T\) (frozen), training text, steps \(K\), learning rate \(\eta\)
  2. for step \(k = 1..K\):
  3.   sample 8 windows of 512 tokens // 4,096 tokens per step
  4.   student logits \(z_S\) (with gradient); teacher logits \(z_T\) (no gradient)
  5.   loss \(\leftarrow\) mean over positions of \(\KL(\softmax(z_T) \,\|\, \softmax(z_S))\) // (122.7); CE baseline: cross-entropy on the next token
  6.   backward; clip the gradient norm to 1; AdamW step with warm-up and cosine decay
  7.   for each pruned matrix: \(W \leftarrow M \odot W\) // (122.8)
  8.   every 50 steps: perplexity on WikiText-2 and held-out C4
# distill.py: recover a pruned model.  Usage: python distill.py 2of4 kd   (or: ce)
import sys, math, json, torch, torch.nn.functional as F
from common import *
P, MODE = sys.argv[1], sys.argv[2]
STEPS, LR, BS, L, WARM = 200, 1e-4, 8, 512, 20
torch.manual_seed(0)
te = wiki_test()[:2048 * 40]; c4v = c4_ids(44000, 45576)[:2048 * 20]   # eval subsets
train = c4_ids(4000, 40000)
student = load(); student.load_state_dict(torch.load(f"wanda_{P}.pt"))
masks = torch.load(f"mask_{P}.pt")
mk = [(get(student.model.layers[int(k.split(".")[0])], k.split(".", 1)[1]).weight, v)
      for k, v in masks.items()]
teacher = load().requires_grad_(False) if MODE == "kd" else None
student.train()
opt = torch.optim.AdamW(student.parameters(), lr=LR, weight_decay=0.0, betas=(0.9, 0.95))
lr_at = lambda s: LR * min(1, (s + 1) / WARM) * 0.5 * (1 + math.cos(math.pi * s / STEPS))
g = torch.Generator().manual_seed(1)
for step in range(1, STEPS + 1):
    st = torch.randint(0, train.numel() - L - 1, (BS,), generator=g)
    x = torch.stack([train[i:i + L] for i in st]).to(dev)
    lg = student(x).logits[:, :-1].float()
    if MODE == "kd":                                    # formula (122.7)
        with torch.no_grad(): tl = teacher(x).logits[:, :-1].float()
        V = lg.size(-1)
        loss = F.kl_div(F.log_softmax(lg, -1).reshape(-1, V), F.log_softmax(tl, -1).reshape(-1, V),
                        log_target=True, reduction="batchmean")
    else:                                               # the CE baseline
        loss = F.cross_entropy(lg.reshape(-1, lg.size(-1)), x[:, 1:].reshape(-1))
    for gp in opt.param_groups: gp["lr"] = lr_at(step - 1)
    opt.zero_grad(); loss.backward()
    torch.nn.utils.clip_grad_norm_(student.parameters(), 1.0); opt.step()
    with torch.no_grad():
        for w, m in mk: w.mul_(m)                       # formula (122.8): zeros stay zero
    if step % 50 == 0:
        student.eval()
        print(json.dumps({"step": step, "loss": round(loss.item(), 4),
                          "wiki": ppl(student, te), "c4": ppl(student, c4v)}))
        student.train()
torch.save(student.state_dict(), f"student_{P}_{MODE}.pt")

Our script also logged the CE loss in the KD runs, the KL divergence, and the time per step; this shortened version prints less. F.kl_div with reduction="batchmean" divides by the first dimension, which here is the number of token positions, so the loss is the average KL per token, as in (122.7). We ran three runs of 200 steps (819,200 training tokens each): 2:4 with KD, 2:4 with CE, and 50% unstructured with KD.

Worked example: what the recovery run costs

One KD step on 4,096 tokens costs about \(6N\) FLOPs per token for the student (forward and backward) plus \(2N\) for the teacher’s forward, with \(N = 134{,}515{,}008\): \(8 \times 134{,}515{,}008 \times 4{,}096 = 4.41 \times 10^{12}\) FLOPs. 200 steps: \(8.8 \times 10^{14}\) FLOPs. Pretraining on 2 trillion tokens cost about \(6N \times 2 \times 10^{12} = 1.6 \times 10^{21}\) FLOPs. So the recovery costs about \(5.5 \times 10^{-7}\) of the pretraining: one part in 1.8 million. A CE step skips the teacher and costs \(6N\) per token, 25% less.

Memory: the teacher adds one more copy of the weights (538 MB in FP32 here, 269 MB in BF16). The logits are the hidden cost: \(8 \times 511 \times 49{,}152\) FP32 values are 804 MB for the student, the same again for the teacher, plus the softmax outputs and gradients. For a large vocabulary, compute the KL in chunks of positions, or keep only the teacher’s top-k probabilities [Likely].

2:4 + KD2:4 + CE (dashed)50% unstructured + KD
Figure 4. Real logs of distill.py on the laptop: perplexity every 50 steps, on the first 40 windows of WikiText-2 (81,920 tokens) and on 20 windows of held-out C4 (40,960 tokens). These subsets are smaller than the full test set, so the values differ a little from the full-set numbers in the last section. The y axis is logarithmic.
How to read Figure 4

Start with 2:4 + KD on WikiText-2. Perplexity falls from 118.02 to 42.93 in the first 50 steps: most of the repair happens at once. Then it falls slowly to 38.76 at step 200, which closes 53% of the log gap to the dense model on this subset (14.49). Switch to 2:4 + CE: the same tokens and steps end at 41.93 (49%). KD after only 100 steps (39.93) is already better than CE after 200. Switch to held-out C4: both runs close more of the gap there (62% for KD, 60% for CE), because the training text is also C4. Last, 50% + KD starts much lower (32.59) and ends at 23.82, 39% of its gap.

Our recovery results on the full WikiText-2 test set. The 2:4 model went from 125.67 to 41.24 with KD, which closes 53% of the log gap, and to 44.78 with CE (49%). With the same 819,200 tokens, KD is better by 9% in perplexity. It is also better at the same compute: 100 KD steps cost \(100 \times 8N\) FLOPs for each token position of a batch, less than 200 CE steps (\(200 \times 6N\)), and it was already better on the subset (39.93 against 41.93). The 50% unstructured model went from 34.95 to 25.24 with KD (40%). But read the numbers against each other. The repaired 2:4 model (41.24) is still worse than the unstructured model before any repair (34.95), and 2.7 times the dense perplexity. 200 steps are a short first look; the Wanda paper used up to 3 days on 4 GPUs for full fine-tuning [Certain].

Step 6: the trade-off, and the decision

Put everything in one place: quality against the cost of serving. Figure 5 plots perplexity against bits per linear weight, the quantity that sets decode speed (formula 122.6). Unstructured 50% sits at 16 bits, like the dense model, because on a GPU it is stored densely. 2:4 sits at 9 bits.

Figure 5. Full WikiText-2 perplexity (log scale) against bits per linear weight in FP16 on a GPU. Dashed lines join a pruned model to its recovered version. “Our run” uses our measurements; “Wanda paper” uses Table 6 of Sun et al. (2023), where LLaMA-7B was pruned with Wanda and then fine-tuned on C4 with LoRA (a budget of 12 hours on one GPU) or with full fine-tuning (a budget of 3 days on 4 GPUs) [Certain].
How to read Figure 5

In “Our run”, the dense model sits at 16 bits and 15.43. The 50% unstructured model sits at the same 16 bits, because a GPU stores it densely, with a worse perplexity (34.95, 25.24 after KD): on a GPU without sparse kernels it costs the same as dense and is only worse. The 2:4 models sit at 9 bits: 125.67 after pruning, 41.24 after KD and 44.78 after CE. Now switch to “Wanda paper”. The picture is the same in shape but very different in size: for LLaMA-7B, 2:4 goes from 11.53 to 8.24 with LoRA and 7.02 with full fine-tuning, only 1.24 times the dense 5.68. A large model with a long recovery run ends close to dense; our small model with 200 steps ends 2.7 times worse.

Our filled table (full WikiText-2 test set; held-out C4 on the 40,960-token subset):

ModelLinear sparsityWikiText-2 pplHeld-out C4 pplBytes of linear weights (FP16)Measured speed-up (GPU)
A: dense0%15.4322.49212 MB1.0×
B: Wanda 50% unstructured50%34.9546.11212 MB (dense storage)1.0× (dense kernels)
C: Wanda 2:450%125.67160.75119 MBnot measured (no NVIDIA GPU)
D: C + distillation50% (2:4)41.2447.25119 MBsame as C
E: C + fine-tuning on data (CE)50% (2:4)44.7849.81119 MBsame as C
F: B + distillation50%25.2432.05212 MBsame as B

Our decision for SmolLM2-135M. We would not ship any of the pruned models. The best 2:4 model (41.24) is 2.7 times worse than dense, for at most 1.78 times faster decode of the linear layers. Model F (25.24) is the best pruned model, but it gives no speed and no memory gain on a GPU. For a model this small, a longer recovery run is the first thing to try; if the gap stays large, a 4-bit quantized dense model is a simpler way to cut memory [Likely]. For a 7B model the paper’s numbers point the other way: 2:4 plus a few GPU-days of recovery comes within 24% of dense, and then the 2:4 speed-up can be worth it.

A decision memo, half a page, makes the result reviewable:

  1. Goal. Example: “a model with less than 120 MB of linear weights and at most 20% higher perplexity than dense.”
  2. Results. The table, plus the measured GPU speed-up of Step 4 for decode and for prefill.
  3. Decision. Which model, with its recovery budget. If the 2:4 speed-up on your GPU is below about 1.2 times end to end, compare with simply using a smaller dense model or with 4-bit quantization (session 112), which may give more speed for less quality loss [Likely].
  4. Checks. Perplexity is not everything: run a few zero-shot tasks (session 40) and read some generated text. Pruned models can keep a good perplexity and still lose specific skills [Likely].

Where this is used

SettingHow pruning and recovery are usedWhat to watch
Model families from one big model (Minitron, Llama 3.2 1B and 3B)Prune a trained model (mostly structured: heads, neurons, layers), then retrain with logit distillation from the big modelStructured pruning gives speed on any hardware; see session 115
2:4 sparse checkpoints for GPU servingPrune to 2:4 with Wanda or SparseGPT, recover, and serve with sparse kernels; often combined with INT8 or FP8 [Likely]Real speed-up is 1.2 to 1.6 times, not 2; check your engine supports the format
Memory-limited edge devicesUnstructured sparsity plus compression; useful only with a sparse-aware runtime [Likely]No gain without a kernel that skips zeros; often 4-bit quantization is simpler
Video understanding with a VLM (video retrieval, action recognition, key-moment detection)Prune the language model of the VLM to 2:4 to speed up the prefill of many frame tokens, where the math speed of sparse tensor cores helps most [Speculation]Calibrate on real video prompts: frame tokens have different outlier features than text [Likely]
Video-RAG and video captioning at scaleA pruned and distilled captioner runs over millions of clips; the dense model is the teacherMeasure caption quality, not only perplexity; check rare actions and objects
Research on sparsityWanda as a fast baseline: seconds per model instead of minutes for SparseGPT [Certain]Compare at the same protocol (calibration data, sequence length, sparsity pattern)

Common mistakes

  1. Expecting speed from unstructured sparsity on a GPU. The zeros are still stored and multiplied. Only 2:4 (or structured pruning) gives speed with standard GPU kernels; CSR at 50% is slower than dense (Figure 3).
  2. Comparing scores across the whole matrix with Wanda. Wanda compares inside each output row. A global or per-layer comparison is the magnitude-pruning habit and works worse for LLMs.
  3. Calibrating with unpruned inputs. If every block is scored with the inputs of the dense model, later blocks are pruned for inputs they will never see. Run the blocks in order and feed each one the outputs of the pruned blocks before it.
  4. Forgetting the mask during recovery. Without \(W \leftarrow M \odot W\) after each step, the zeros fill up and the 2:4 pattern is lost. Count zeros after training.
  5. Recovering on the evaluation data. Training on WikiText and testing on WikiText measures memorization of the domain. Recover on general text (C4 here) and test on something else.
  6. Trusting a paper’s numbers for a small model. LLaMA-7B loses 2 times in perplexity at 2:4; our 135M model lost 8.1 times. Small models are much harder to prune.
  7. Promising 2× from 2:4. 2 times is peak matrix math. Decode gains at most 1.78 times on weight reads, and measured end-to-end gains are about 1.2 to 1.6 times.
  8. Reading 1% perplexity differences as real. The device alone changed our 2:4 result by 1.5%. Fix the device and seed, and look at differences much larger than that.

Interview questions

You pruned 50% of the weights of an LLM and it is not faster. Why?basic
Unstructured zeros are stored inside a dense matrix, and dense GPU kernels multiply them like any other number, so memory traffic and math are unchanged. Storing the matrix in a sparse format does not help at 50%: CSR with FP16 values and 16-bit indices is exactly as large as dense, and sparse kernels at this sparsity are slower than dense tensor-core kernels. On our CPU, CSR at 50% was 445 times slower than dense for one token. To get speed, use 2:4 sparsity with sparse tensor-core kernels, or structured pruning, which makes the matrices smaller.
What is the Wanda score, and where does it come from?basic
\(S_{ij} = |W_{ij}| \cdot \lVert X_j \rVert_2\): the weight size times the norm of its input feature over the calibration tokens. It comes from the layer-wise reconstruction error \(\lVert WX - (M \odot W)X \rVert^2\). Removing one weight adds \(W_{ij}^2 (XX^\top)_{jj} = W_{ij}^2 \lVert X_j \rVert^2\) if you keep only the diagonal of \(XX^\top\), so the cheapest weights to remove are those with the smallest \(S_{ij}\). It is the OBS saliency with a diagonal Hessian and no weight update. It matters because LLMs have a few input features that are much larger than the others; small weights on those features carry a lot of output (Figure 1).
Why does Wanda compare weights inside each output row?core
Each output feature is computed by its own row. If you compare across the whole matrix, some rows can lose most of their weights while others lose almost none, and the rows that are hit hard produce outputs that are badly wrong. Comparing inside each row removes the same share from every output, which keeps every output roughly intact. The Wanda paper found the per-output comparison clearly better for LLMs, while for image classifiers (ConvNeXt-B, DeiT-B) the per-layer comparison was slightly better, so this seems to be a property of LLMs [Certain]. A side benefit: 2:4 is also a per-row constraint, so the two fit together.
What is 2:4 sparsity, how is it stored, and what speed-up can you expect?core
In every group of 4 consecutive weights of a row, 2 are zero. The format stores the 2 kept values and a 2-bit position for each: \(b/2 + 1\) bits per weight, 9 bits for FP16 (56.25% of dense). Sparse tensor cores (NVIDIA Ampere and newer) skip the zeros and reach twice the peak math rate. Expect at most 2 times in compute-bound matmuls and at most 1.78 times in memory-bound decode, less end to end: the Wanda paper measured 1.54 to 1.63 times on LLaMA-65B matmuls and 1.24 times end to end on LLaMA-7B, because attention, norms and the KV cache are not sparse.
Why does 2:4 hurt quality more than unstructured 50% at the same sparsity?core
Unstructured pruning can put the zeros where they cost least: it may remove 4 weights from one group and none from another. 2:4 forces exactly 2 zeros into every group of 4, so in groups where 3 or 4 weights matter, it must remove important weights. On SmolLM2-135M, Wanda gave a WikiText-2 perplexity of 34.95 at 50% unstructured, 63.66 at 4:8 and 125.67 at 2:4, against 15.43 dense. On LLaMA-7B the paper reports 7.26, 8.57 and 11.53 against 5.68. The looser 4:8 pattern sits in between.
Why recover with distillation instead of fine-tuning on the data?core
The dense model is the behavior we want back, and it gives a full distribution over 49,152 tokens at every position instead of one correct token, so each step carries far more signal. CE on data pulls the student toward the data distribution, which is a different, noisier target. Minitron found the logit KL loss alone at temperature 1 better than conventional training after pruning, at the same compute. In our lab, with the same 819,200 tokens, KD reached 41.24 on WikiText-2 against 44.78 for CE, and KD after 100 steps was already better than CE after 200. The cost is a teacher forward pass per step (about a third more compute than CE) and memory for the teacher and its logits.
How do you keep pruned weights at zero during recovery training?basic
Keep the masks from pruning and apply \(W \leftarrow M \odot W\) after every optimizer step (or mask the gradients and make sure the optimizer adds nothing to masked entries; weight decay and momentum can otherwise move them). The mask is fixed: we train only the surviving weights. At the end, check that the sparsity is still exactly 50% and that every group of 4 has 2 non-zeros. With LoRA, merge the adapter and then apply the mask again, because the low-rank update is dense [Likely].
Your 135M model loses 8 times in perplexity at 2:4, but the paper’s 7B loses 2 times. Is your code wrong?advanced
Not necessarily. Larger models are easier to prune: they have more redundant weights, and the SparseGPT paper shows the gap to dense shrinks as the model grows. A 135M model has little to spare. Check the code anyway: the dense perplexity through the block-by-block path must equal the normal forward; the masks must have exactly 2 non-zeros per group; the scores must use the input norms of each layer (q, k and v share an input, gate and up share an input, o and down have their own); and calibration must feed pruned outputs forward. Then compare magnitude against Wanda: if Wanda is not far better than magnitude (125.67 against 2,547.5 for us), something is wrong.
How do you report a pruning result honestly?advanced
Give the full trade-off, not one number: perplexity (with the exact protocol: dataset, window length, overlapping or not) for dense, pruned and recovered models; at least one downstream task; the pattern (unstructured, 4:8, 2:4) and which layers were pruned; the memory of the stored model; and the measured speed-up on named hardware for decode and prefill, end to end, not only for one matmul. Report the recovery budget in tokens and GPU hours, and compare with simple alternatives at the same memory: a smaller dense model, or quantization.
Can you combine 2:4 sparsity with quantization? What do you gain?advanced
Yes. The values of a 2:4 matrix can be INT8, FP8 or INT4, and the hardware supports sparse INT8 and FP8 tensor-core math [Likely]. Bits per weight follow \(b/2 + 1\): 3 bits for 4-bit values, against 4 for dense INT4, so the gain over quantization alone is 4/3 = 1.33 times in bytes, not 2, because the metadata is a fixed 1 bit per weight. Quality losses add up, so prune first, recover, then quantize (or use a method that does both), and measure each step.

Exercises

  1. A row of a layer has weights \((0.8, -0.1, 0.4, 0.3)\) and input norms \((1, 10, 1, 2)\). Which 2 weights does Wanda remove under 2:4, and which does magnitude pruning remove? Compare the errors \(\sum W^2 \lVert X \rVert^2\) of the removed weights.
  2. Using formula (122.5), how many bits per weight does 2:4 need with FP8 values? How many megabytes are the linear layers of SmolLM2-135M in that format?
  3. In Figure 2, at which sparsity does magnitude pruning first become more than 4 times worse than Wanda? Explain why the gap grows with sparsity.
  4. A 2:4 model is pruned from ppl 15.43 to 125.67 and recovered to 30.0. What share of the gap is closed by formula (122.9)? What share would a perplexity of 20.0 mean?
  5. Decode of a model takes 20 ms per token; 70% of it is reading linear weights, the rest is attention, KV cache and other work. With 2:4 FP16 and perfect kernels, what is the best possible end-to-end speed-up?
  6. Our KD run used 819,200 tokens. How many tokens did the Wanda paper’s LoRA run see, if it ran for 12 hours at 5,000 tokens per second? What does this say about our 200-step result?
Answerscheck
1. Wanda scores \(|W| \cdot \lVert X \rVert = (0.8, 1.0, 0.4, 0.6)\): in the one group of 4, Wanda removes weights 3 and 4 (scores 0.4 and 0.6), error \(0.16 + 0.09 \times 4 = 0.52\). Magnitude removes the two smallest \(|W|\), weights 2 and 4: error \(0.01 \times 100 + 0.36 = 1.36\). The total is 2.16, so Wanda loses 24% of the output energy and magnitude 63%: the weight −0.1 on the large input is the second most important one. 2. \(b = 8\): \(8/2 + 1 = 5\) bits per weight; \(106{,}168{,}320 \times 5 / 8 = 66.4\) MB, against 212.3 MB in FP16. 3. At 50%: magnitude 1,622.7 against Wanda 34.95, 46 times worse. At 40% the ratio is 3.3 (70.49 against 21.60), at 30% 1.7. Magnitude pruning ignores the input size, so it removes small weights on the outlier features first; the more you prune, the more of these important weights it removes, and the error grows much faster than for Wanda. 4. \((\ln 125.67 - \ln 30)/(\ln 125.67 - \ln 15.43) = 1.432/2.097 = 0.68\): 68% of the gap. A perplexity of 20.0 would mean 88%. 5. The weight part becomes \(0.7 \times 9/16 = 0.394\) of the old time; plus the other 0.3: 0.694 of 20 ms = 13.9 ms, a speed-up of 1.44 times at best. This is why measured end-to-end gains are far below 2 times. 6. \(12 \times 3600 \times 5000 = 216\) million tokens, 264 times our 819,200. Our run is a very short first look: its result is a lower bound on what recovery can do. The curves in Figure 4 flatten at the end because the cosine schedule brings the learning rate to zero, not because the repair is complete.

Key takeaways

  • Wanda scores each weight by \(|W_{ij}| \cdot \lVert X_j \rVert_2\) (122.3), compares inside each output row and needs no weight update; on SmolLM2-135M at 50% it gave 34.95 against 1,622.7 for magnitude pruning (dense 15.43).
  • 2:4 (122.4) costs much more quality than unstructured 50%: 125.67 for our model. Small models break much faster than the 7B models in the papers.
  • Unstructured 50% gives no speed or memory gain on a GPU. 2:4 stores \(b/2 + 1\) bits per weight (122.5) and gives at most 1.78× in decode and 2× in math (122.6); measured end-to-end gains are about 1.24×.
  • Recover with a fixed mask (122.8) and distillation from the dense model (122.7). On our 2:4 model, 200 KD steps (819,200 tokens) went from 125.67 to 41.24 (53% of the log gap), better than CE (44.78), but the model is still 2.7 times worse than dense.
  • Report the whole trade-off: perplexity before and after recovery, bytes, measured speed on named hardware, the recovery budget, and simple alternatives such as quantization.

Key formulas

\[ S_{ij} = |W_{ij}|\,\lVert X_j \rVert_2, \qquad \sum_{k=0}^{3} M_{i,4g+k} = 2, \qquad \text{bits}_{2:4} = \frac{b}{2} + 1, \qquad \text{speed-up}_\text{decode} \le \frac{16}{9} \]
\[ \mathcal{L}_\text{KD} = \frac{1}{N}\sum_t \KL\big(p_T \,\|\, p_S\big), \qquad W \leftarrow M \odot \big(W - \eta\, u(W)\big), \qquad \text{gap closed} = \frac{\log \text{ppl}_\text{p} - \log \text{ppl}_\text{r}}{\log \text{ppl}_\text{p} - \log \text{ppl}_\text{d}} \]

References for this session

  1. Sun et al. (2023). A Simple and Effective Pruning Approach for Large Language Models (Wanda)
  2. Mishra et al. (2021). Accelerating Sparse Deep Neural Networks
  3. Muralidharan et al. (2024). Compact Language Models via Pruning and Knowledge Distillation (Minitron)
  4. Frantar & Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
  5. Allal et al. (2025). SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
  6. Raffel et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5, C4)
  7. PyTorch (2025). torch.sparse: Sparse Semi-Structured Tensors
  8. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models