Dynamic inference: early exit and layer skipping
Some inputs and some tokens are easy, so a model does not always need all of its layers. Early exit stops after a few layers when the model is already sure, and layer skipping jumps over layers. This session goes from early exit for BERT classifiers (DeeBERT, FastBERT, PABEE) to early exit for LLM generation (CALM, SkipDecode), explains why skipped layers leave holes in the KV cache, and ends with LayerSkip, which uses its own early layers as a draft model for speculative decoding.
What you will learn
- How early exit works, and how to compute its speed-up from the average depth and the cost of checking an exit
- Sample-level exit for classification: DeeBERT's entropy rule, FastBERT's normalized entropy and self-distillation, PABEE's patience counter
- Token-level exit for generation with CALM: softmax response, hidden-state saturation and a small classifier as confidence, plus the decaying threshold
- The KV-cache problem: why skipped layers leave holes, and four fixes (recompute, state propagation, SkipDecode, verification)
- SkipDecode: batch-wide, position-based, decreasing exit points that skip lower layers instead of top layers
- LayerSkip: layer dropout and a shared-head early-exit loss in training, then self-speculative decoding with one model and one KV cache
- What the papers measured, and a simple model to estimate the speed-up of self-speculation
Why this matters
A 32-layer language model spends exactly 32 layers on every token. It spends them on "the" after "on" and on the last digit of a hard sum. Many tokens are easy: a small part of the network already knows the answer. Dynamic inference lets the amount of compute depend on the input. The two main tools are early exit (stop after a few layers when the model is already sure) and layer skipping (jump over some layers).
All the methods in the last sessions were static: pruning (session 115), quantization and distillation (session 119) make the model smaller once, and then every input pays the same smaller price. Dynamic methods keep the full model and decide per input, or even per token, how much of it to run. The efficient-inference survey of Zhou et al. puts them under model-level optimization, next to mixture-of-experts (session 95): MoE chooses which experts to run (width), early exit chooses how many layers to run (depth). [Certain]
The story has three steps. First, early exit for classification (DeeBERT, FastBERT, PABEE, 2020): one decision per input, no cache, simple. Second, early exit for generation (CALM, SkipDecode): one decision per token, and a new problem appears. A token that exits at layer 3 never computes the keys and values of layers 4 to 32, but later tokens want to attend to them (session 80). Third, LayerSkip (2024) trains a model so that its early layers are good drafters, and then uses them for self-speculative decoding: draft with the first layers, check with the rest, and get exactly the full model's output.
The idea: put exits inside the network
Take a network with \(L\) layers. After some layers \(l\), add an exit: a small head that turns the hidden state \(\mathbf{h}_l\) into a prediction (class probabilities, or next-token probabilities). At inference, after each layer we look at the exit's prediction and ask a simple question: "is this good enough?" If yes, we stop and return it. If no, we run the next layer.
If a fraction \(q_l\) of the inputs exits at layer \(l\), the average number of layers is \(\E[\ell] = \sum_l l\, q_l\). Each exit that we check also costs something: a head and a decision. Write that cost as a fraction \(c\) of one layer. Then the speed-up over the full model is
- Read the symbols
- \(L\)
- number of layers of the full model
- \(q_l\)
- fraction of inputs (or tokens) that stop at layer \(l\); the \(q_l\) add up to 1
- \(\E[\ell]\)
- average number of layers that are run
- \(c\)
- cost of checking one exit, as a fraction of the cost of one layer (here we assume an exit is checked after every layer)
A 12-layer classifier. Half of the inputs exit at layer 3, 30% at layer 6, and 20% need all 12 layers. Then \(\E[\ell] = 0.5 \cdot 3 + 0.3 \cdot 6 + 0.2 \cdot 12 = 5.7\) layers, and the speed-up is \(12 / 5.7 = 2.11\times\) if exits are free. If each exit check costs 2% of a layer, the cost is \(5.7 \times 1.02 = 5.81\) layers and the speed-up is \(2.06\times\).
For a small classifier head, \(c\) is tiny. For an LLM it is not. The exit head of an LLM is the LM head, a \(d \times V\) matrix. In Llama 2 7B (\(d = 4096\), \(V = 32{,}000\)) it has 131M weights, while one layer has 202M: \(c \approx 0.65\). In Llama 3 8B (\(V = 128{,}256\)) the head has 525M weights against 218M per layer: one full softmax at an exit costs 2.4 layers. [Certain] (Computed from the config files on Hugging Face.) This is the first reason why LLM early exit needs care.
Two design questions follow, and every method in this session is an answer to them. How do we train the exits, so that early layers give useful predictions? And what rule decides that a prediction is good enough?
Sample-level early exit for classification
In 2020, three papers made BERT-style classifiers faster with early exit. Here one input (a sentence, or a pair of sentences) gets one label. The whole input goes through the layers together, so "exit" means "stop the whole input". There is no generation and no cache, so nothing is missing later. This is the easy case.
DeeBERT: stop when the entropy is low
DeeBERT (Xin et al.) adds a small classifier, called an off-ramp, after every transformer layer. Each off-ramp outputs class probabilities \(\mathbf{p}_l\). The rule: stop at the first layer where the entropy of \(\mathbf{p}_l\) is below a threshold \(S\). Entropy measures how spread out a distribution is: 0 when one class has all the probability, \(\log N\) when all \(N\) classes are equal.
A larger \(S\) means more inputs exit early: faster, but less accurate. Training has two stages. First, fine-tune the backbone and the last classifier as usual. Then freeze them and train the other off-ramps on the sum of their losses. So the final layer is exactly as good as a normal fine-tuned model. DeeBERT reports up to about 40% less inference time with little loss in quality, on BERT-base and RoBERTa-base and six GLUE tasks. [Certain]
FastBERT: self-distillation and normalized entropy
FastBERT (Liu et al.) uses the same structure. It calls the exits student classifiers and the last classifier the teacher. Its two changes: the uncertainty is the entropy divided by its maximum \(\log N\), so it is between 0 and 1 for any number of classes, and the students are trained to copy the teacher (self-distillation, as in session 119) instead of the labels. Because they only need the teacher's outputs, the students can train on unlabeled data.
The threshold on \(u_l\) is called the "speed": a higher speed means more early exits. FastBERT reports speed-ups from about 1 to 12 times over BERT on twelve English and Chinese datasets, depending on the threshold. [Certain]
PABEE: stop when the layers stop changing their mind
A confident wrong answer has low entropy too. Neural networks are often over-confident, so an early layer can be sure and wrong. PABEE (Zhou et al., "BERT Loses Patience") does not trust any single exit. It counts how many layers in a row give the same predicted class, and stops when this count reaches the patience \(t\):
For regression, "same" means \(|y_l - y_{l-1}| \lt \tau\). If the count never reaches \(t\), the last layer answers. Patience \(t\) means the earliest possible exit is at layer \(t + 1\). The exits are trained together with a weighted loss that gives deeper layers more weight:
PABEE also argues that early exit can improve accuracy. Deep models can "overthink": a correct prediction at a middle layer turns wrong at a later layer. On ALBERT-base, PABEE reports a 1.57× speed-up together with a higher GLUE score than the full model. [Certain]
Two classes. An exit outputs \(\mathbf{p} = (0.9, 0.1)\). Its entropy is \(-(0.9 \ln 0.9 + 0.1 \ln 0.1) = 0.325\) nats. With DeeBERT and \(S = 0.2\), this input does not exit. FastBERT's normalized value is \(0.325 / \ln 2 = 0.469\). A sharper \(\mathbf{p} = (0.97, 0.03)\) has entropy \(0.135\), below \(S = 0.2\), so DeeBERT exits.
PABEE with \(t = 3\): suppose layers 1 to 6 predict A, B, B, A, A, A. The counts are 0, 0, 1, 0, 1, 2, so no exit yet; if layer 7 also says A, the count is 3 and the input exits at layer 7 with answer A. Note that PABEE never looks at the probabilities, only at the labels.
Figure 1 runs the entropy rule and the patience rule on 400 simulated inputs. Each input has a hidden difficulty. Easy inputs become confident after a few layers; hard inputs stay uncertain, and some are wrong even at layer 12. The data is a toy simulation, not a real model, but it shows the trade-off curve that all these papers report.
How to read Figure 1
Start with the entropy rule at \(S = 0.2\): many inputs exit early, the average depth falls to about 6 layers, and the accuracy drops only a little below the full model's 88.25%. Move \(S\) toward 0.69 (that is \(\ln 2\), the largest possible entropy for two classes): almost everything exits at layer 1, and the accuracy falls hard. Then switch to patience. With \(t = 1\) the rule is too easy to satisfy; at \(t = 7\) or more it matches the full model, because only inputs whose prediction is stable for many layers exit. Every point on these curves is one possible product: you choose the threshold that gives the accuracy you need.
- input: input \(x\), layers \(f_1, \dots, f_L\), exit heads \(g_1, \dots, g_L\), rule (entropy threshold \(S\) or patience \(t\))
- \(\mathbf{h} \leftarrow \text{embed}(x)\); \(\text{cnt} \leftarrow 0\); \(\text{prev} \leftarrow \text{none}\)
- for \(l = 1\) to \(L\):
- \(\mathbf{h} \leftarrow f_l(\mathbf{h})\); \(\mathbf{p} \leftarrow \softmax(g_l(\mathbf{h}))\)
- if rule is entropy and \(H(\mathbf{p}) \lt S\): return \(\argmax \mathbf{p}\) // formula (121.2)
- if rule is patience: \(\text{cnt} \leftarrow \text{cnt} + 1\) if \(\argmax \mathbf{p} = \text{prev}\) else \(0\); \(\text{prev} \leftarrow \argmax \mathbf{p}\)
- if \(\text{cnt} = t\): return \(\text{prev}\) // formula (121.4)
- return \(\argmax \mathbf{p}\) // the last layer always answers
Token-level early exit for generation: CALM
For a decoder that generates text, the natural unit is one token. Each token gets its own exit decision. The idea is the same, but three things become harder: the confidence measure is over a vocabulary of tens of thousands of words, one wrong early token can change everything that follows, and the KV cache has holes (next section). CALM, "Confident Adaptive Language Modeling" (Schuster et al., 2022), handles all three.
Three confidence measures
Let \(\mathbf{d}^i_t\) be the hidden state of token \(t\) after decoder layer \(i\). CALM exits when a confidence \(c^i_t\) is at least a threshold \(\lambda\). It tests three choices of \(c\):
- Read the symbols
- \(p_{(1)}, p_{(2)}\)
- the largest and second-largest probabilities of \(\softmax(W \mathbf{d}^i_t)\), the shared LM head applied at layer \(i\)
- \(\cos(\cdot,\cdot)\)
- cosine similarity of the hidden state with the one from the layer before; near 1 means "this layer changed almost nothing"
- \(\mathbf{w}, b\)
- a tiny linear classifier (only \(d + 1\) weights), trained after the model is frozen to predict "the exit at this layer gives the same token as the last layer"
The softmax response is the most accurate signal, but it needs the full LM head at every layer it checks, which we saw is expensive. (CALM notes that the next layer can start in parallel, so the extra FLOPs do not all add latency.) The saturation measure is free but weak; it was often below the static baseline that uses the same number of layers for every token. The classifier is very cheap and sometimes gives better speed than the softmax. [Certain]
Training uses a weighted average of the losses of all layers, \(\mathcal{L} = \sum_i \omega_i \mathcal{L}_i\) with \(\omega_i = i / \sum_j j\), the same shape as formula (121.5). [Certain]
A threshold that decays over the sequence
CALM measured that an error early in a generated sequence hurts the final output more than an error late in the sequence: early mistakes are copied forward. So it makes exiting harder at the start and easier later, with a decaying threshold:
Here \(N\) is the maximum output length and \(\tau\) a temperature. With \(\tau = 0\) the threshold is the constant \(0.9\lambda + 0.1\); a larger \(\tau\) makes it fall faster toward \(0.9\lambda\). The single number \(\lambda\) is then calibrated on held-out data with a statistical method (Learn then Test), so that the early-exit output stays close to the full model's output with a chosen tolerance \(\delta\) and high probability. [Certain]
\(\lambda = 0.9\) and \(\tau = 4\). At \(t = 0\): \(0.81 + 0.1 \cdot 1 = 0.91\). At \(t = N/2\): \(0.81 + 0.1\, e^{-2} = 0.824\). At \(t = N\): \(0.81 + 0.1\, e^{-4} = 0.812\). So the first token needs a top-1 minus top-2 gap of 0.91 to exit, the last token only 0.812.
Figure 2 runs this rule, layer by layer, on a toy decoder with \(L = 8\) layers and a vocabulary of 8 words. The toy has a real residual stream: each token starts with a random hidden vector \(\mathbf{d}^0_t\), every layer adds an update, and the LM head turns \(\mathbf{d}^i_t\) into 8 logits (in the toy, the head simply reads one number per word). Easy tokens get a strong push toward their word in the first layers. Hard tokens are first pushed toward a wrong word, which fades with depth. The full model (all 8 layers) writes "the cat sat on the mat . the dog slept". Everything you see, the softmax, the gap \(p_{(1)} - p_{(2)}\), the cosine and the threshold, is computed in your browser from these vectors.
- \(\lambda' \leftarrow \operatorname{clip}(0.9\lambda + 0.1\,e^{-\tau t/N})\) // formula (121.7)
- for \(i = 1\) to \(L\):
- \(\mathbf{d}^i \leftarrow \text{layer}_i(\mathbf{d}^{i-1})\); write \(\mathbf{k}^i_t, \mathbf{v}^i_t\) to the cache
- \(c \leftarrow p_{(1)} - p_{(2)}\) of \(\softmax(W\mathbf{d}^i)\), or \(\cos(\mathbf{d}^i, \mathbf{d}^{i-1})\) // formula (121.6)
- if \(c \ge \lambda'\) or \(i = L\): break
- emit \(\hat y_t = \argmax \softmax(W \mathbf{d}^i)\)
- for \(j = i+1\) to \(L\): \(\mathbf{k}^j_t = W^j_K \mathbf{d}^i,\ \mathbf{v}^j_t = W^j_V \mathbf{d}^i\) // formula (121.8)
How to read Figure 2
Press Play or Step. Each step is one line of the algorithm: run a layer and check the exit, or emit a token. The box next to the algorithm shows formula (121.6) and formula (121.7) with the real numbers. Token \(t = 0\) ("the") is easy: at layer 2, \(c = 0.954 - 0.015 = 0.939 \ge 0.910\), so it exits after 2 layers. Token \(t = 1\) ("cat") starts with the wrong word: layer 1 says "dog" with 0.713, and the gap only passes \(\lambda' = 0.877\) at layer 7. With \(\lambda = 0.9\) and \(\tau = 4\) the exit layers are \((2, 7, 5, 4, 3, 5, 4, 7, 1, 8)\): 46 of 80 layer runs, 42.5% saved, 1.74× fewer layers. But token \(t = 8\) exits at layer 1 with "cat", confident and wrong: \(c = 0.923 - 0.018 = 0.905 \ge \lambda' = 0.814\), while the full model says "dog". Set \(\tau = 0\): the threshold stays at 0.910, token 8 does not exit early, no token is wrong, and the cost rises to 60 runs (25% saved). Switch to saturation: the cosine is near 1 after one or two layers for almost every token, so it saves 75% but gets 3 tokens wrong. This is CALM's finding in small: the saturation signal is cheap but weak.
How to read Figure 3
Set \(\tau = 0\): the orange line is flat and every position faces the same bar. Raise \(\tau\): the line falls to the right, so later tokens exit earlier and the average depth goes down, while the first tokens are still protected. Lower \(\lambda\) to see more aggressive exiting. In the real paper, the calibration step chooses \(\lambda\) for you from the quality you ask for.
What CALM measured
CALM was tested on an 8-layer T5 1.1 encoder-decoder for summarization (CNN/DM), translation (WMT EN-FR) and question answering (SQuAD). [Certain] With the softmax measure it reports up to about 3× speed-up while provably keeping the output quality within the chosen tolerance. [Certain] An oracle that exits at the first layer whose token equals the last layer's token used only 1.53 decoder layers on average on CNN/DM, with ROUGE-L 38.24 against 38.32 for the full model. [Certain] So the potential is large; the hard part is a confidence measure that is close to the oracle.
The KV-cache problem
Here is the real difficulty of token-level exit. In a decoder, token \(t\) at layer \(i\) attends to the keys and values of all earlier tokens at the same layer \(i\). With a KV cache, those keys and values were saved when the earlier tokens were processed. But suppose token 2 exited at layer 3. Its keys and values at layers 4 to 8 were never computed. When token 5 reaches layer 6, the cache has a hole where token 2 should be.
There are four ways to deal with the holes:
- Recompute later. When a later token needs layer \(i\) of an earlier token, run the missing layers for that earlier token then. This is exact, but the saved work comes back: each earlier token ends up running as deep as the deepest later token.
- Copy the state (CALM's state propagation). When token \(s\) exits at layer \(e_s\), use its last hidden state \(\mathbf{d}^{e_s}_s\) as if it were the hidden state of every skipped layer, and compute keys and values from it with each skipped layer's own projections:
\[ \mathbf{k}^{i}_s = W^{i}_K\, \mathbf{d}^{e_s}_s, \qquad \mathbf{v}^{i}_s = W^{i}_V\, \mathbf{d}^{e_s}_s \qquad \text{for all } i \gt e_s \tag{121.8} \]This costs only the K and V projections (no attention, no MLP), and all of them can run in parallel because they share one input. CALM tested it with the oracle: ROUGE-L 38.24 against 38.31 for an oracle that never skips, so copying the hidden state barely hurts. Copying the projected keys and values of the exit layer instead dropped ROUGE-L to 23.02. [Certain] So the projections of the skipped layer matter.
- Never create holes (SkipDecode). Choose exit points so that a later token never needs a layer that an earlier token skipped (next section).
- Fill the holes in a later pass (LayerSkip). Treat the early-exit tokens as drafts, and let a full-depth verification pass compute the missing layers for all of them at once (section after next).
Eight tokens, eight layers, exit layers \(e = (8, 3, 5, 2, 6, 8, 4, 3)\). The tokens compute \(39\) of the \(64\) (token, layer) cells. Now count, for each token \(t\) and each layer \(i \le e_t\), the earlier tokens \(s\) with \(e_s \lt i\): those are attention reads that hit a hole. There are 30 such missing reads. Recompute fixes them with 16 extra layer runs: each token must run up to the deepest exit of any later token, \(\sum_s \max(0, \max_{t \gt s} e_t - e_s) = 16\). State copying instead fills all \(64 - 39 = 25\) skipped cells with cheap K/V projections. Figure 4 shows the same grid.
- for token \(t = 1\) to \(T\) (one decoding step each):
- run its layers \(R_t\): \(1..e_t\) (early exit) or \(\{1\} \cup\) the top \(e_t - 1\) (SkipDecode, formula (121.10))
- for each \(i \in R_t\): attention reads \(\mathbf{k}^i_s, \mathbf{v}^i_s\) of every \(s \lt t\)
- if \(\mathbf{k}^i_s\) is missing: a hole (no fix), or run layer \(i\) for token \(s\) now (recompute)
- after the exit: \(\mathbf{k}^j_t = W^j_K \mathbf{d}^{e_t}_t\) for \(j \gt e_t\) (copy state, formula (121.8))
How to read Figure 4
Press Play. In "Early exit, no fix", tokens 1 to 3 are fine until token 3 (5 layers) reaches layers 4 and 5, where token 2 (exit at 3) has no keys: 2 holes. Token 5 (6 layers) hits 8 holes, and token 6, which runs all 8 layers, hits 16: the red cells in the columns of tokens 2, 3, 4 and 5 that stopped low. After all 8 tokens there are 30 missing reads, the number of the worked example. "Recompute" removes every hole, but the orange cells appear exactly when a later token needs them: 2 after token 3, 8 after token 5, 16 after token 6, so the total grows from 39 to 55 of 64. "Copy state" adds the green cells right after each exit (25 in total), with only two small matrix products each. "SkipDecode" uses the exit schedule (8, 7, 6, 5, 5, 4, 3, 2) of formula (121.10) and spends each token's layers at the top of the stack, after one warm-up layer. Every token computes a subset of the layers of every earlier token, so the count of holes stays at 0 with no extra work.
- input: token \(y_t\), KV cache for all layers, threshold \(\lambda'(\lambda, t)\) from formula (121.7)
- \(\mathbf{d} \leftarrow \text{embed}(y_t)\)
- for \(i = 1\) to \(L\):
- \(\mathbf{d} \leftarrow \text{layer}_i(\mathbf{d}, \text{cache}_i)\); append \(\mathbf{k}^i_t, \mathbf{v}^i_t\) to \(\text{cache}_i\)
- \(c \leftarrow\) confidence of \(\mathbf{d}\) at layer \(i\) // formula (121.6)
- if \(c \ge \lambda'\) or \(i = L\): break
- for every skipped layer \(j = i+1\) to \(L\): append \(W^j_K \mathbf{d},\ W^j_V \mathbf{d}\) to \(\text{cache}_j\) // state propagation, formula (121.8)
- return \(y_{t+1} = \argmax \softmax(W_{\text{head}}\, \mathbf{d})\)
SkipDecode: early exit that works with batching and caching
CALM still has two practical problems. In a batch, tokens of different requests exit at different layers, but the GPU runs one layer for the whole batch at once. So the batch is done only when its deepest token is done:
For a batch of 4 tokens with exits \((2, 3, 8, 4)\), the step runs 8 layers and only \(17 / 32 = 53\%\) of that work is needed. And the state copying of formula (121.8) still adds work for every skipped layer.
SkipDecode (Del Corro et al., 2023) removes both problems with one decision: the exit point is not chosen per token by a confidence. It is a fixed function of the position in the sequence, the same for every request in the batch, and it never increases:
- Read the symbols
- \(P\)
- prompt length; prompt tokens run all \(L\) layers (they are processed in parallel in the prefill)
- \(N\)
- maximum sequence length
- \(e_{\max}, e_{\min}\)
- number of layers for the first and for the last generated token; they fix the compute budget in advance
Why decreasing? The paper measured that the loss per position falls along the sequence: later tokens have more context and are easier. [Certain] And because exits never increase, every earlier token has done at least as much work as the current one. The second idea: SkipDecode skips layers instead of stopping early. Each token runs one warm-up layer at the bottom and then the top layers of the stack. So any layer that the current token runs is a layer that all earlier tokens also ran, and their keys and values are in the cache. No holes, no copying, no recompute. With batch-wide exit points, there is no waiting for the slowest request either. [Certain]
SkipDecode was tested on OPT 1.3B and 6.7B (24 and 32 layers) on E2E, Reddit-TLDR and CNN-DM, with 2× to 5× speed-ups and small loss in quality; one warm-up layer worked best in all settings. [Certain] The price is flexibility: a hard token late in a sequence still gets few layers, because the rule ignores the token's actual difficulty.
LayerSkip: train for early exit, then self-speculate
All methods so far share a weakness: when an early exit is wrong, the error goes into the output. LayerSkip (Elhoushi et al., Meta, 2024) changes the goal. Early exit is used only to make a draft, and the full model checks the draft, as in speculative decoding (session 128). The output is then exactly what the full model would produce (with greedy decoding), and the early layers only decide the speed.
Training recipe
A normal LLM's middle layers are bad at predicting the next token through the final LM head: the head was trained only on the last layer. LayerSkip fixes this with two training changes, and no new heads.
Layer dropout. During training, skip each layer \(l\) for a random part of the batch. Early layers are dropped rarely, late layers often, so the model learns not to depend on its last layers:
\(D(l)\) grows from 0 at layer 0 to 1 at the last layer \(L - 1\). \(S(t)\) scales over training steps: 1 for continued pretraining and fine-tuning, and a similar exponential curriculum when training from scratch. For Llama 2 7B (32 layers) with \(p_{\max} = 0.1\): layer 0 is never dropped, layer 15 with probability 0.040, layer 31 with probability 0.1. [Certain]
Early-exit loss. Apply the shared LM head \(g\) to the output of every layer and add the losses with normalized weights:
The scale \(e(l)\) grows roughly with \(l^2\) (\(e(l) = e_{\text{scale}} \sum_{i=0}^{l} i\), with a large weight for the last layer), so late layers count most. The binary curriculum \(C(t, l)\) turns on the exit loss for only some layers at each step (for example every \(R\)-th layer, rotating), because computing the loss of all layers at every step slowed training and hurt the last layer. [Certain] For Llama 2 7B continued pretraining, the paper used \(p_{\max} = 0.1\), \(e_{\text{scale}} = 0.2\) and \(R = 8\). [Certain]
Self-speculative decoding
After this training, the first \(E\) layers plus the shared head form a small, fast draft model that lives inside the big model. Decoding repeats two steps:
- Self-drafting. Generate \(d\) tokens autoregressively with only the first \(E\) layers (early exit at a fixed layer \(E\)).
- Self-verification. Run the remaining layers \(E+1, \dots, L\) on all \(d\) draft tokens in one parallel pass. Keep the draft tokens up to the first one where the full model disagrees, then add the full model's own token at that point.
The KV-cache problem disappears. The first \(E\) layers are the same in both steps, so their keys and values from drafting are already the right ones: there is one KV cache. The hidden state at layer \(E\) of each draft token is saved too (the "exit query cache"; together the KVQ cache), so verification starts directly at layer \(E + 1\). And the verification pass computes the missing layers \(E+1\) to \(L\) for all draft tokens, so after each cycle the cache is complete again. Compared with normal speculative decoding there is no second model in memory. [Certain]
How fast is it? Use the standard model from session 128: each draft token is accepted with probability \(\alpha\), independently. One cycle then yields on average \((1 - \alpha^{d+1})/(1 - \alpha)\) tokens. In the memory-bound decode phase (session 82), the cost of a pass is roughly the number of layers whose weights are read, whatever the number of tokens in the pass. So the \(d\) drafting steps cost \(d \cdot E / L\) of a full forward pass. The verification pass runs layers \(E+1..L\) for the draft positions and all layers for the last draft token (that gives the extra token), so it reads every layer once and costs about one full pass. This is the formula of session 128 with draft cost \(c = E/L\):
- Read the symbols
- \(\alpha\)
- probability that one draft token equals the full model's token (assumed independent for each token)
- \(d\)
- number of draft tokens per cycle
- \(E\)
- exit layer used for drafting; \(L\) is the total number of layers
- \((1 - \alpha^{d+1})/(1 - \alpha)\)
- expected tokens per cycle: the accepted drafts plus one token from the verifier
- \(1 + dE/L\)
- cost of one cycle in units of one full forward pass (the LM head and other overheads are ignored)
\(L = 32\), draft with \(E = 8\) layers, \(d = 4\) draft tokens, \(\alpha = 0.75\). Tokens per cycle: \((1 - 0.75^5)/(1 - 0.75) = 3.05\). Cost: \(1 + 4 \cdot 8/32 = 2.0\) full passes. Speed-up \(\approx 3.05 / 2.0 = 1.53\times\). With \(d = 6\) the tokens rise to 3.47 but the cost to 2.5, so the speed-up falls to \(1.39\times\). The best choice here is \(d = 3\), with \(1.56\times\): more drafts are only worth it if \(\alpha\) is high.
How to read Figure 5
The trade-off is between \(E\) and \(\alpha\). A small \(E\) makes drafts cheap, but in a real model it also lowers \(\alpha\) (shallow layers guess worse). Here \(\alpha\) is a separate slider, so you can ask "how good must my 8-layer draft be?". Try \(E = 8\), \(\alpha = 0.5\): the best \(d\) is 1 and the gain is only 1.2×. Raise \(\alpha\) to 0.9: the best \(d\) grows to 6 and the speed-up reaches about 2.1×. Move \(E\) to 24: drafting is expensive and even with \(\alpha = 0.9\) the best is about 1.09×. Bars below the grey line mean the method is slower than normal decoding.
- input: prefix \(x\), exit layer \(E\), draft length \(d\), shared KVQ cache
- while not finished:
- for \(j = 1\) to \(d\): run layers \(1..E\) on the last token, apply the shared head, append \(\hat y_j = \argmax\) // draft; writes K, V of layers 1..E and the layer-E state
- run layers \(E+1..L\) on the draft positions in one pass, starting from the saved layer-E states (the last draft token also runs layers \(1..E\)) // verify; fills K, V of layers E+1..L
- \(y_j \leftarrow\) the full model's token at each draft position \(j\)
- \(m \leftarrow\) number of leading positions with \(\hat y_j = y_j\)
- accept \(\hat y_1, \dots, \hat y_m\), then append \(y_{m+1}\) (the full model's correction) // output = full-model greedy output
- drop the cache entries of the rejected draft tokens
Figure 6 runs Algorithm 3 on the toy decoder of Figure 2. The draft is the toy's own layer \(E\) read by the shared head; the verifier is the same model with all 8 layers. Because the verifier decides, the output is always the full model's sentence. Only the cost changes.
- while not finished:
- for \(j = 1\) to \(k = \min(d, \text{tokens left})\): run layers \(1..E\), \(\hat y_j = \argmax \softmax(W\mathbf{d}^E)\)
- run layers \(E+1..L\) on the \(k\) drafts in one pass: full-model tokens \(y_1, \dots, y_{k+1}\)
- \(m \leftarrow\) number of leading positions with \(\hat y_j = y_j\)
- accept \(\hat y_1..\hat y_m\), append \(y_{m+1}\); drop the cache of rejected drafts
- cost \(\mathrel{+}= 1 + kE/L\) full passes // denominator of formula (121.13)
How to read Figure 6
Press Play. With \(E = 3\) and \(d = 4\), the first cycle drafts "the cat sat on" with 3 layers each, the verification pass agrees with all four and adds a fifth token, "the": 5 tokens for \(1 + 4 \cdot 3/8 = 2.5\) passes. The second cycle drafts "mat . the cat"; the verifier rejects "cat" and writes "dog": 4 tokens for 2.5 passes. The last cycle drafts "dog" for the last position, the verifier writes "slept". In total 10 tokens cost 6.375 passes, a measured speed-up of 1.57×, and the sentence is exactly the full model's. 7 of 9 drafts were accepted (\(\hat\alpha = 0.778\)), and formula (121.13) with this \(\hat\alpha\) predicts only 1.29×: here the accepted drafts come in runs, not independently, which is the warning of the note below. Now move \(E\) to 6 with \(d = 3\): the drafts are as good, but each costs 6/8 of a pass, and the total of 10.75 passes is slower than normal decoding (0.93×). \(E = 1\) with \(d = 2\) gives 1.63×, the best in this toy, even though the 1-layer draft is right only 5 times in 9.
What LayerSkip measured
The paper trained several Llama models with the recipe (continued pretraining, pretraining from scratch, and fine-tuning) and reports speed-ups of up to 2.16× on CNN/DM summarization, 1.82× on coding and 2.0× on TOPv2 semantic parsing. [Certain] One row in its tables shows why verification matters. For Llama 2 7B after continued pretraining, plain early exit at layer 8 on one-shot CNN/DM gave ROUGE-2 0.012, against 0.079 for the full model; self-speculation from the same layer 8 (12 draft tokens) gave 0.078 at 1.86× speed. [Certain] So the early layers are much too weak to answer alone, but good enough to guess, and the verifier keeps the quality.
Putting it into code
The code below trains a tiny 6-layer decoder on a toy language (each token is the sum of the two before it, modulo 16) with a LayerSkip-style recipe: layer dropout (121.11), one shared exit head, and an early-exit loss weighted toward late layers (121.5). Then it measures how often each exit agrees with the last layer, runs CALM-style softmax-response exit, and runs self-speculative decoding with a 1-layer draft, checking that the output equals normal greedy decoding. For clarity it does not use a KV cache; every pass recomputes the prefix. It is deterministic on CPU and runs in under a minute.
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) V, D, L, T = 16, 64, 6, 20 # vocab, width, layers, sequence length class Block(nn.Module): def __init__(s): super().__init__() s.n1, s.n2 = nn.LayerNorm(D), nn.LayerNorm(D) s.qkv, s.o = nn.Linear(D, 3 * D), nn.Linear(D, D) s.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D)) def forward(s, h): q, k, v = s.qkv(s.n1(h)).chunk(3, dim=-1) a = F.scaled_dot_product_attention(q, k, v, is_causal=True) # one head, causal h = h + s.o(a) return h + s.mlp(s.n2(h)) class TinyLM(nn.Module): def __init__(s): super().__init__() s.emb, s.pos = nn.Embedding(V, D), nn.Embedding(T, D) s.blocks = nn.ModuleList(Block() for _ in range(L)) s.norm, s.head = nn.LayerNorm(D), nn.Linear(D, V) # ONE shared exit head def forward(s, x, upto=L, drop=None): h = s.emb(x) + s.pos(torch.arange(x.shape[1])) outs = [] for l in range(upto): if drop is None or not drop[l]: # layer dropout skips block l h = s.blocks[l](h) outs.append(s.head(s.norm(h))) # logits of exit l+1 return outs def batch(n): # toy language: x[i+1] = x[i] + x[i-1] (mod V), two random start tokens x = torch.randint(0, V, (n, T)) for i in range(2, T): x[:, i] = (x[:, i - 1] + x[:, i - 2]) % V return x model = TinyLM(); opt = torch.optim.AdamW(model.parameters(), lr=3e-3) p_max = 0.1 p_l = [p_max * (math.exp(l * math.log(2) / (L - 1)) - 1) for l in range(L)] # formula (121.11) w = torch.arange(1, L + 1, dtype=torch.float); w = w / w.sum() # later exits weigh more, formula (121.5) for it in range(400): x = batch(64) drop = [bool(torch.rand(()) < p) for p in p_l] outs = model(x[:, :-1], drop=drop) loss = sum(w[l] * F.cross_entropy(o.reshape(-1, V), x[:, 1:].reshape(-1)) for l, o in enumerate(outs)) opt.zero_grad(); loss.backward(); opt.step() # 1) how often does exit l agree with the last layer? (CALM's oracle question) model.eval(); x = batch(512) with torch.no_grad(): outs = model(x[:, :-1]) final = outs[-1].argmax(-1) print("agree with last layer:", [round((o.argmax(-1) == final).float().mean().item(), 3) for o in outs]) # 2) token-level exit with the softmax response, formula (121.6): top-1 minus top-2 lam = 0.9 probs = torch.stack([o.softmax(-1) for o in outs]) # (L, n, T-1, V) top2 = probs.topk(2, dim=-1).values; conf = top2[..., 0] - top2[..., 1] ok = conf >= lam; ok[-1] = True # the last layer always exits exit_l = ok.float().argmax(0) # first confident layer (0-based) pred = torch.gather(probs.argmax(-1), 0, exit_l[None])[0] print("avg layers:", round((exit_l + 1).float().mean().item(), 2), "of", L, "| same token as full model:", round((pred == final).float().mean().item(), 3)) # 3) self-speculative greedy decoding: draft with E layers, verify with all L @torch.no_grad() def greedy(prefix, n, upto=L): x = prefix.clone() for _ in range(n): x = torch.cat([x, model(x, upto)[-1][:, -1].argmax(-1, keepdim=True)], 1) return x @torch.no_grad() def self_spec(prefix, n, E=2, d=4): x, drafted, accepted, passes = prefix.clone(), 0, 0, 0 while x.shape[1] < prefix.shape[1] + n: k = min(d, prefix.shape[1] + n - x.shape[1]) draft = greedy(x, k, upto=E) # cheap: E layers per token full = model(draft, L)[-1][:, x.shape[1] - 1:].argmax(-1) # ONE pass of the full model passes += 1; prop = draft[:, x.shape[1]:] m = 0 while m < k and prop[0, m] == full[0, m]: m += 1 drafted += k; accepted += m x = torch.cat([x, prop[:, :m], full[:, m:m + 1]], 1)[:, :prefix.shape[1] + n] return x, accepted / drafted, passes prefix = batch(1)[:, :3] ref = greedy(prefix, 16) out, acc, passes = self_spec(prefix, 16, E=1) print("identical to full greedy:", torch.equal(out, ref), "| acceptance:", round(acc, 2), "| full passes:", passes, "for 16 tokens")
On our run (PyTorch 2.8, CPU) it printed: agreement with the last layer [0.854, 0.945, 0.961, 0.988, 0.996, 1.0] for exits 1 to 6; with \(\lambda = 0.9\), an average of 1.88 layers out of 6 while choosing the same token as the full model 99.9% of the time; and for self-speculation, identical to full greedy: True with 81% of the draft tokens accepted. The exact numbers can change with other PyTorch versions [Likely]; the True must not.
Where this is used
| Setting | Method that fits | Why |
|---|---|---|
| Text classification at high volume (spam, moderation, intent, sentiment) | DeeBERT / FastBERT / PABEE exits on an encoder | One decision per input, no cache; most inputs are easy |
| Phones and edge devices with a latency budget | Early exit with a threshold chosen per device | One trained model serves several speed–accuracy points (FastBERT's "speed" knob) |
| Video classification and action recognition | Exits on the per-frame or per-clip backbone; stop when the label is stable over several exits (patience) | Many frames or clips are easy (static scene, clear action), so average compute falls [Likely] |
| Key-moment detection on live video streams | A cheap early exit runs on every frame; only uncertain frames go deeper | Most frames contain nothing interesting; the deep layers are saved for candidates [Likely] |
| Video retrieval and fingerprinting | Early-exit embeddings for a first, coarse search; full-depth embeddings to re-rank the top results | Same idea as draft-and-verify: cheap guess, exact check [Speculation] |
| LLM generation at batch size 1 (local assistants) | CALM-style exit, or LayerSkip self-speculation | Decoding is memory-bound; fewer layers per token means fewer weights to read |
| LLM serving with large batches | SkipDecode (batch-wide, position-based exits) or self-speculation | Per-token exits break batching and the KV cache; these two keep both |
| Summarization and long outputs | LayerSkip self-speculation (CNN/DM: up to 2.16×) | Long outputs with many predictable tokens give high acceptance |
LayerSkip checkpoints for several Llama models were released, and self-speculation with an early-exit draft is available in the Hugging Face transformers generation API. [Likely] Static layer removal (deleting layers that change the hidden state little) is the "same depth for every token" version of these ideas, and belongs with session 115.
Common mistakes
- Adding exits to a model that was not trained for them. The LM head of a normal LLM gives poor predictions from middle layers. Early exit needs an exit loss (121.5, 121.12) or separately trained exit heads, or the speed-up comes with a large quality loss.
- Ignoring the cost of the exit check. For an LLM, one softmax over the vocabulary can cost more than a whole layer (2.4 layers for Llama 3 8B). Check exits only at a few layers, use a cheap classifier, or compute the head only for the top candidates.
- Leaving holes in the KV cache. If a token exits and nothing fills its skipped layers, later tokens attend to missing or stale keys and values. Use state propagation (121.8), position-based skipping, or a verification pass.
- Copying the exit layer's K and V into the skipped layers. CALM measured that this breaks quality (ROUGE-L 23.02 instead of 38.24). Copy the hidden state and apply each layer's own \(W_K, W_V\).
- Measuring speed-up at batch size 1 and deploying with big batches. With per-token exits, a batch waits for its slowest member (121.9); the speed-up can vanish.
- Trusting a single confident exit. Over-confident early layers make wrong early exits. Patience (PABEE), calibration (CALM) or verification (LayerSkip) protect against this.
- Using one threshold for all positions in generation. Early errors in a sequence propagate; a decaying threshold (121.7) or a decreasing schedule (121.10) spends more compute where it matters.
- Too many draft tokens in self-speculation. Each extra draft costs \(E/L\) of a pass; with a low acceptance rate the speed-up falls, and can drop below 1× (Figure 5).
Interview questions
What is early exit, and what two choices define an early-exit method?basic
Compare DeeBERT, FastBERT and PABEE.core
Why is early exit easy for BERT classification but hard for LLM generation?core
Explain the KV-cache problem of token-level early exit and the ways to solve it.core
What confidence measures does CALM use, and what are their trade-offs?core
Why does CALM use a decaying threshold, and how is the threshold chosen?advanced
How does SkipDecode stay compatible with batching and the KV cache?advanced
Describe the LayerSkip training recipe.core
How does self-speculative decoding differ from normal speculative decoding?core
A 32-layer model drafts with 8 layers, 4 tokens per cycle, acceptance 0.75. Estimate the speed-up. What if acceptance is 0.5?advanced
Can early exit improve accuracy? When?advanced
Exercises
- A 24-layer model has exits after every layer. 40% of inputs exit at layer 4, 40% at layer 12 and 20% at layer 24. Each exit check costs 5% of a layer. Compute the speed-up with formula (121.1).
- Three classes, an exit outputs \(\mathbf{p} = (0.7, 0.2, 0.1)\). Compute the entropy and FastBERT's normalized uncertainty \(u\).
- PABEE with patience \(t = 2\). The layer predictions are A, A, B, B, B, A, ... At which layer does the input exit, and with which label?
- In Figure 4, change the exits to \(e = (4, 4, 4, 4, 4, 4, 4, 4)\) (static depth 4). How many missing reads are there with no fix? Why?
- SkipDecode with \(L = 32\), prompt length \(P = 100\), \(N = 200\), \(e_{\max} = 24\), \(e_{\min} = 8\). How many layers do positions 100, 150 and 200 use?
- For LayerSkip with \(L = 16\) and \(p_{\max} = 0.2\), compute the dropout probability of layers 0, 8 and 15 (with \(S(t) = 1\)).
- Self-speculation with \(L = 32\), \(E = 16\), \(d = 3\), \(\alpha = 0.8\). Compute the speed-up with formula (121.13). Is it better than \(E = 8\), \(d = 3\), \(\alpha = 0.6\)?
Answerscheck
Key takeaways
- Dynamic inference spends compute per input: early exit stops after a few layers when the model is sure; the speed-up is \(L / (\E[\ell](1+c))\) (121.1), and the exit cost \(c\) is large for LLM heads.
- For classification, exits are easy: DeeBERT uses entropy (121.2), FastBERT normalized entropy and self-distillation (121.3), PABEE a patience counter (121.4) that is robust to over-confident layers.
- For generation, CALM exits per token with a softmax-response, saturation or classifier confidence (121.6), a decaying threshold (121.7) and calibration; up to about 3× on T5.
- Token-level exit leaves holes in the KV cache. Fix them by recomputing, by copying the hidden state through each layer's own projections (121.8), or avoid them with SkipDecode's batch-wide, decreasing, top-layer schedule (121.10).
- LayerSkip trains with layer dropout (121.11) and a shared-head early-exit loss (121.12), then drafts with the first \(E\) layers and verifies with the rest: one model, one KV cache, the full model's greedy output, up to 2.16× on CNN/DM.
Key formulas
References for this session
- Xin et al. (2020). DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
- Liu et al. (2020). FastBERT: a Self-distilling BERT with Adaptive Inference Time
- Zhou et al. (2020). BERT Loses Patience: Fast and Robust Inference with Early Exit (PABEE)
- Schuster et al. (2022). Confident Adaptive Language Modeling
- Del Corro et al. (2023). SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference
- Elhoushi et al. (2024). LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
- Xin et al. (2020). DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
- Liu et al. (2020). FastBERT: a Self-distilling BERT with Adaptive Inference Time
- Zhou et al. (2020). BERT Loses Patience: Fast and Robust Inference with Early Exit (PABEE)
- Schuster et al. (2022). Confident Adaptive Language Modeling
- Del Corro et al. (2023). SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference
- Elhoushi et al. (2024). LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
- Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models