LLM & VLM courseBlog
121Track 5: LLM inference and serving · session 44 of 65

Dynamic inference: early exit and layer skipping

Some inputs and some tokens are easy, so a model does not always need all of its layers. Early exit stops after a few layers when the model is already sure, and layer skipping jumps over layers. This session goes from early exit for BERT classifiers (DeeBERT, FastBERT, PABEE) to early exit for LLM generation (CALM, SkipDecode), explains why skipped layers leave holes in the KV cache, and ends with LayerSkip, which uses its own early layers as a draft model for speculative decoding.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: The KV cache (session 80), Why inference is slow (session 82), basic softmax and entropy; Speculative decoding 1 (session 128) helps for the last part

What you will learn

  • How early exit works, and how to compute its speed-up from the average depth and the cost of checking an exit
  • Sample-level exit for classification: DeeBERT's entropy rule, FastBERT's normalized entropy and self-distillation, PABEE's patience counter
  • Token-level exit for generation with CALM: softmax response, hidden-state saturation and a small classifier as confidence, plus the decaying threshold
  • The KV-cache problem: why skipped layers leave holes, and four fixes (recompute, state propagation, SkipDecode, verification)
  • SkipDecode: batch-wide, position-based, decreasing exit points that skip lower layers instead of top layers
  • LayerSkip: layer dropout and a shared-head early-exit loss in training, then self-speculative decoding with one model and one KV cache
  • What the papers measured, and a simple model to estimate the speed-up of self-speculation

Why this matters

A 32-layer language model spends exactly 32 layers on every token. It spends them on "the" after "on" and on the last digit of a hard sum. Many tokens are easy: a small part of the network already knows the answer. Dynamic inference lets the amount of compute depend on the input. The two main tools are early exit (stop after a few layers when the model is already sure) and layer skipping (jump over some layers).

All the methods in the last sessions were static: pruning (session 115), quantization and distillation (session 119) make the model smaller once, and then every input pays the same smaller price. Dynamic methods keep the full model and decide per input, or even per token, how much of it to run. The efficient-inference survey of Zhou et al. puts them under model-level optimization, next to mixture-of-experts (session 95): MoE chooses which experts to run (width), early exit chooses how many layers to run (depth). [Certain]

The story has three steps. First, early exit for classification (DeeBERT, FastBERT, PABEE, 2020): one decision per input, no cache, simple. Second, early exit for generation (CALM, SkipDecode): one decision per token, and a new problem appears. A token that exits at layer 3 never computes the keys and values of layers 4 to 32, but later tokens want to attend to them (session 80). Third, LayerSkip (2024) trains a model so that its early layers are good drafters, and then uses them for self-speculative decoding: draft with the first layers, check with the rest, and get exactly the full model's output.

The idea: put exits inside the network

Take a network with \(L\) layers. After some layers \(l\), add an exit: a small head that turns the hidden state \(\mathbf{h}_l\) into a prediction (class probabilities, or next-token probabilities). At inference, after each layer we look at the exit's prediction and ask a simple question: "is this good enough?" If yes, we stop and return it. If no, we run the next layer.

If a fraction \(q_l\) of the inputs exits at layer \(l\), the average number of layers is \(\E[\ell] = \sum_l l\, q_l\). Each exit that we check also costs something: a head and a decision. Write that cost as a fraction \(c\) of one layer. Then the speed-up over the full model is

\[ \text{speed-up} = \frac{L}{\E[\ell]\,(1 + c)}, \qquad \E[\ell] = \sum_{l=1}^{L} l\, q_l \tag{121.1} \]
Read the symbols
\(L\)
number of layers of the full model
\(q_l\)
fraction of inputs (or tokens) that stop at layer \(l\); the \(q_l\) add up to 1
\(\E[\ell]\)
average number of layers that are run
\(c\)
cost of checking one exit, as a fraction of the cost of one layer (here we assume an exit is checked after every layer)
Worked example: average depth and speed-up

A 12-layer classifier. Half of the inputs exit at layer 3, 30% at layer 6, and 20% need all 12 layers. Then \(\E[\ell] = 0.5 \cdot 3 + 0.3 \cdot 6 + 0.2 \cdot 12 = 5.7\) layers, and the speed-up is \(12 / 5.7 = 2.11\times\) if exits are free. If each exit check costs 2% of a layer, the cost is \(5.7 \times 1.02 = 5.81\) layers and the speed-up is \(2.06\times\).

For a small classifier head, \(c\) is tiny. For an LLM it is not. The exit head of an LLM is the LM head, a \(d \times V\) matrix. In Llama 2 7B (\(d = 4096\), \(V = 32{,}000\)) it has 131M weights, while one layer has 202M: \(c \approx 0.65\). In Llama 3 8B (\(V = 128{,}256\)) the head has 525M weights against 218M per layer: one full softmax at an exit costs 2.4 layers. [Certain] (Computed from the config files on Hugging Face.) This is the first reason why LLM early exit needs care.

Two design questions follow, and every method in this session is an answer to them. How do we train the exits, so that early layers give useful predictions? And what rule decides that a prediction is good enough?

Sample-level early exit for classification

In 2020, three papers made BERT-style classifiers faster with early exit. Here one input (a sentence, or a pair of sentences) gets one label. The whole input goes through the layers together, so "exit" means "stop the whole input". There is no generation and no cache, so nothing is missing later. This is the easy case.

DeeBERT: stop when the entropy is low

DeeBERT (Xin et al.) adds a small classifier, called an off-ramp, after every transformer layer. Each off-ramp outputs class probabilities \(\mathbf{p}_l\). The rule: stop at the first layer where the entropy of \(\mathbf{p}_l\) is below a threshold \(S\). Entropy measures how spread out a distribution is: 0 when one class has all the probability, \(\log N\) when all \(N\) classes are equal.

\[ H(\mathbf{p}_l) = -\sum_{i=1}^{N} p_{l,i} \log p_{l,i}, \qquad \text{exit at the first } l \text{ with } H(\mathbf{p}_l) \lt S \tag{121.2} \]

A larger \(S\) means more inputs exit early: faster, but less accurate. Training has two stages. First, fine-tune the backbone and the last classifier as usual. Then freeze them and train the other off-ramps on the sum of their losses. So the final layer is exactly as good as a normal fine-tuned model. DeeBERT reports up to about 40% less inference time with little loss in quality, on BERT-base and RoBERTa-base and six GLUE tasks. [Certain]

FastBERT: self-distillation and normalized entropy

FastBERT (Liu et al.) uses the same structure. It calls the exits student classifiers and the last classifier the teacher. Its two changes: the uncertainty is the entropy divided by its maximum \(\log N\), so it is between 0 and 1 for any number of classes, and the students are trained to copy the teacher (self-distillation, as in session 119) instead of the labels. Because they only need the teacher's outputs, the students can train on unlabeled data.

\[ u_l = \frac{H(\mathbf{p}_l)}{\log N} \in [0, 1], \qquad \mathcal{L}_{\text{FastBERT}} = \sum_{l=1}^{L-1} \KL\!\left(\mathbf{p}_L \,\Vert\, \mathbf{p}_l\right) \tag{121.3} \]

The threshold on \(u_l\) is called the "speed": a higher speed means more early exits. FastBERT reports speed-ups from about 1 to 12 times over BERT on twelve English and Chinese datasets, depending on the threshold. [Certain]

PABEE: stop when the layers stop changing their mind

A confident wrong answer has low entropy too. Neural networks are often over-confident, so an early layer can be sure and wrong. PABEE (Zhou et al., "BERT Loses Patience") does not trust any single exit. It counts how many layers in a row give the same predicted class, and stops when this count reaches the patience \(t\):

\[ \text{cnt}_l = \begin{cases} \text{cnt}_{l-1} + 1 & \text{if } \argmax \mathbf{p}_l = \argmax \mathbf{p}_{l-1} \\ 0 & \text{otherwise} \end{cases}, \qquad \text{exit when } \text{cnt}_l = t \tag{121.4} \]

For regression, "same" means \(|y_l - y_{l-1}| \lt \tau\). If the count never reaches \(t\), the last layer answers. Patience \(t\) means the earliest possible exit is at layer \(t + 1\). The exits are trained together with a weighted loss that gives deeper layers more weight:

\[ \mathcal{L} = \frac{\sum_{j=1}^{L} j \cdot \mathcal{L}_j}{\sum_{j=1}^{L} j} \tag{121.5} \]

PABEE also argues that early exit can improve accuracy. Deep models can "overthink": a correct prediction at a middle layer turns wrong at a later layer. On ALBERT-base, PABEE reports a 1.57× speed-up together with a higher GLUE score than the full model. [Certain]

Worked example: the three rules on one input

Two classes. An exit outputs \(\mathbf{p} = (0.9, 0.1)\). Its entropy is \(-(0.9 \ln 0.9 + 0.1 \ln 0.1) = 0.325\) nats. With DeeBERT and \(S = 0.2\), this input does not exit. FastBERT's normalized value is \(0.325 / \ln 2 = 0.469\). A sharper \(\mathbf{p} = (0.97, 0.03)\) has entropy \(0.135\), below \(S = 0.2\), so DeeBERT exits.

PABEE with \(t = 3\): suppose layers 1 to 6 predict A, B, B, A, A, A. The counts are 0, 0, 1, 0, 1, 2, so no exit yet; if layer 7 also says A, the count is 3 and the input exits at layer 7 with answer A. Note that PABEE never looks at the probabilities, only at the labels.

Figure 1 runs the entropy rule and the patience rule on 400 simulated inputs. Each input has a hidden difficulty. Easy inputs become confident after a few layers; hard inputs stay uncertain, and some are wrong even at layer 12. The data is a toy simulation, not a real model, but it shows the trade-off curve that all these papers report.

inputs that exit at this layer, correctexit here, wrong
Figure 1. Sample-level early exit on 400 simulated inputs and a 12-layer classifier with two classes. Bars show where the inputs exit. The readout gives the average depth, the speed-up from formula (121.1) with free exits, and the accuracy compared to always running all 12 layers.
How to read Figure 1

Start with the entropy rule at \(S = 0.2\): many inputs exit early, the average depth falls to about 6 layers, and the accuracy drops only a little below the full model's 88.25%. Move \(S\) toward 0.69 (that is \(\ln 2\), the largest possible entropy for two classes): almost everything exits at layer 1, and the accuracy falls hard. Then switch to patience. With \(t = 1\) the rule is too easy to satisfy; at \(t = 7\) or more it matches the full model, because only inputs whose prediction is stable for many layers exit. Every point on these curves is one possible product: you choose the threshold that gives the accuracy you need.

Algorithm 1: sample-level early exit (inference)
  1. input: input \(x\), layers \(f_1, \dots, f_L\), exit heads \(g_1, \dots, g_L\), rule (entropy threshold \(S\) or patience \(t\))
  2. \(\mathbf{h} \leftarrow \text{embed}(x)\); \(\text{cnt} \leftarrow 0\); \(\text{prev} \leftarrow \text{none}\)
  3. for \(l = 1\) to \(L\):
  4.   \(\mathbf{h} \leftarrow f_l(\mathbf{h})\); \(\mathbf{p} \leftarrow \softmax(g_l(\mathbf{h}))\)
  5.   if rule is entropy and \(H(\mathbf{p}) \lt S\): return \(\argmax \mathbf{p}\) // formula (121.2)
  6.   if rule is patience: \(\text{cnt} \leftarrow \text{cnt} + 1\) if \(\argmax \mathbf{p} = \text{prev}\) else \(0\); \(\text{prev} \leftarrow \argmax \mathbf{p}\)
  7.     if \(\text{cnt} = t\): return \(\text{prev}\) // formula (121.4)
  8. return \(\argmax \mathbf{p}\) // the last layer always answers

Token-level early exit for generation: CALM

For a decoder that generates text, the natural unit is one token. Each token gets its own exit decision. The idea is the same, but three things become harder: the confidence measure is over a vocabulary of tens of thousands of words, one wrong early token can change everything that follows, and the KV cache has holes (next section). CALM, "Confident Adaptive Language Modeling" (Schuster et al., 2022), handles all three.

Three confidence measures

Let \(\mathbf{d}^i_t\) be the hidden state of token \(t\) after decoder layer \(i\). CALM exits when a confidence \(c^i_t\) is at least a threshold \(\lambda\). It tests three choices of \(c\):

\[ c^i_t = \underbrace{p_{(1)} - p_{(2)}}_{\text{softmax response}}, \qquad c^i_t = \underbrace{\cos\!\left(\mathbf{d}^i_t, \mathbf{d}^{i-1}_t\right)}_{\text{hidden-state saturation}}, \qquad c^i_t = \underbrace{\sigma\!\left(\mathbf{w}^\top \mathbf{d}^i_t + b\right)}_{\text{early-exit classifier}} \tag{121.6} \]
Read the symbols
\(p_{(1)}, p_{(2)}\)
the largest and second-largest probabilities of \(\softmax(W \mathbf{d}^i_t)\), the shared LM head applied at layer \(i\)
\(\cos(\cdot,\cdot)\)
cosine similarity of the hidden state with the one from the layer before; near 1 means "this layer changed almost nothing"
\(\mathbf{w}, b\)
a tiny linear classifier (only \(d + 1\) weights), trained after the model is frozen to predict "the exit at this layer gives the same token as the last layer"

The softmax response is the most accurate signal, but it needs the full LM head at every layer it checks, which we saw is expensive. (CALM notes that the next layer can start in parallel, so the extra FLOPs do not all add latency.) The saturation measure is free but weak; it was often below the static baseline that uses the same number of layers for every token. The classifier is very cheap and sometimes gives better speed than the softmax. [Certain]

Training uses a weighted average of the losses of all layers, \(\mathcal{L} = \sum_i \omega_i \mathcal{L}_i\) with \(\omega_i = i / \sum_j j\), the same shape as formula (121.5). [Certain]

A threshold that decays over the sequence

CALM measured that an error early in a generated sequence hurts the final output more than an error late in the sequence: early mistakes are copied forward. So it makes exiting harder at the start and easier later, with a decaying threshold:

\[ \lambda'(\lambda, t) = \operatorname{clip}_{[0,1]}\!\left( \frac{9}{10}\lambda + \frac{1}{10} e^{-\tau t / N} \right) \tag{121.7} \]

Here \(N\) is the maximum output length and \(\tau\) a temperature. With \(\tau = 0\) the threshold is the constant \(0.9\lambda + 0.1\); a larger \(\tau\) makes it fall faster toward \(0.9\lambda\). The single number \(\lambda\) is then calibrated on held-out data with a statistical method (Learn then Test), so that the early-exit output stays close to the full model's output with a chosen tolerance \(\delta\) and high probability. [Certain]

Worked example: the decaying threshold

\(\lambda = 0.9\) and \(\tau = 4\). At \(t = 0\): \(0.81 + 0.1 \cdot 1 = 0.91\). At \(t = N/2\): \(0.81 + 0.1\, e^{-2} = 0.824\). At \(t = N\): \(0.81 + 0.1\, e^{-4} = 0.812\). So the first token needs a top-1 minus top-2 gap of 0.91 to exit, the last token only 0.812.

Figure 2 runs this rule, layer by layer, on a toy decoder with \(L = 8\) layers and a vocabulary of 8 words. The toy has a real residual stream: each token starts with a random hidden vector \(\mathbf{d}^0_t\), every layer adds an update, and the LM head turns \(\mathbf{d}^i_t\) into 8 logits (in the toy, the head simply reads one number per word). Easy tokens get a strong push toward their word in the first layers. Hard tokens are first pushed toward a wrong word, which fades with depth. The full model (all 8 layers) writes "the cat sat on the mat . the dog slept". Everything you see, the softmax, the gap \(p_{(1)} - p_{(2)}\), the cosine and the threshold, is computed in your browser from these vectors.

layer run, confidence below \(\lambda'\)exit layerskipped, K and V from copied statecurrent step
  1. \(\lambda' \leftarrow \operatorname{clip}(0.9\lambda + 0.1\,e^{-\tau t/N})\) // formula (121.7)
  2. for \(i = 1\) to \(L\):
  3. \(\mathbf{d}^i \leftarrow \text{layer}_i(\mathbf{d}^{i-1})\); write \(\mathbf{k}^i_t, \mathbf{v}^i_t\) to the cache
  4. \(c \leftarrow p_{(1)} - p_{(2)}\) of \(\softmax(W\mathbf{d}^i)\), or \(\cos(\mathbf{d}^i, \mathbf{d}^{i-1})\) // formula (121.6)
  5. if \(c \ge \lambda'\) or \(i = L\): break
  6. emit \(\hat y_t = \argmax \softmax(W \mathbf{d}^i)\)
  7. for \(j = i+1\) to \(L\): \(\mathbf{k}^j_t = W^j_K \mathbf{d}^i,\ \mathbf{v}^j_t = W^j_V \mathbf{d}^i\) // formula (121.8)
Figure 2. CALM-style token-level early exit, animated on a toy 8-layer decoder that generates 10 tokens. Left: columns are tokens \(t = 0, \dots, 9\), rows are layers (layer 1 at the bottom); each cell that runs shows its confidence \(c^i_t\). Below the grid: the threshold \(\lambda'(t)\) of formula (121.7), the token that the exit emits, and the token of the full model. Right: the softmax of the shared LM head at the current cell, and the confidence against the threshold. The readout counts the layer runs, the compute saved and the tokens that differ from the full model.
How to read Figure 2

Press Play or Step. Each step is one line of the algorithm: run a layer and check the exit, or emit a token. The box next to the algorithm shows formula (121.6) and formula (121.7) with the real numbers. Token \(t = 0\) ("the") is easy: at layer 2, \(c = 0.954 - 0.015 = 0.939 \ge 0.910\), so it exits after 2 layers. Token \(t = 1\) ("cat") starts with the wrong word: layer 1 says "dog" with 0.713, and the gap only passes \(\lambda' = 0.877\) at layer 7. With \(\lambda = 0.9\) and \(\tau = 4\) the exit layers are \((2, 7, 5, 4, 3, 5, 4, 7, 1, 8)\): 46 of 80 layer runs, 42.5% saved, 1.74× fewer layers. But token \(t = 8\) exits at layer 1 with "cat", confident and wrong: \(c = 0.923 - 0.018 = 0.905 \ge \lambda' = 0.814\), while the full model says "dog". Set \(\tau = 0\): the threshold stays at 0.910, token 8 does not exit early, no token is wrong, and the cost rises to 60 runs (25% saved). Switch to saturation: the cosine is near 1 after one or two layers for almost every token, so it saves 75% but gets 3 tokens wrong. This is CALM's finding in small: the saturation signal is cheap but weak.

exit layer of each token (of 8)threshold \(\lambda'(t)\), right axis
Figure 3. The decaying threshold on 40 simulated tokens and an 8-layer decoder (the size of the T5 decoder used in the CALM paper). Each token has a confidence that grows with depth; a token exits at the first layer where its softmax response reaches \(\lambda'(t)\) from formula (121.7). The confidences are simulated and fixed; only the rule changes.
How to read Figure 3

Set \(\tau = 0\): the orange line is flat and every position faces the same bar. Raise \(\tau\): the line falls to the right, so later tokens exit earlier and the average depth goes down, while the first tokens are still protected. Lower \(\lambda\) to see more aggressive exiting. In the real paper, the calibration step chooses \(\lambda\) for you from the quality you ask for.

What CALM measured

CALM was tested on an 8-layer T5 1.1 encoder-decoder for summarization (CNN/DM), translation (WMT EN-FR) and question answering (SQuAD). [Certain] With the softmax measure it reports up to about 3× speed-up while provably keeping the output quality within the chosen tolerance. [Certain] An oracle that exits at the first layer whose token equals the last layer's token used only 1.53 decoder layers on average on CNN/DM, with ROUGE-L 38.24 against 38.32 for the full model. [Certain] So the potential is large; the hard part is a confidence measure that is close to the oracle.

The KV-cache problem

Here is the real difficulty of token-level exit. In a decoder, token \(t\) at layer \(i\) attends to the keys and values of all earlier tokens at the same layer \(i\). With a KV cache, those keys and values were saved when the earlier tokens were processed. But suppose token 2 exited at layer 3. Its keys and values at layers 4 to 8 were never computed. When token 5 reaches layer 6, the cache has a hole where token 2 should be.

There are four ways to deal with the holes:

  1. Recompute later. When a later token needs layer \(i\) of an earlier token, run the missing layers for that earlier token then. This is exact, but the saved work comes back: each earlier token ends up running as deep as the deepest later token.
  2. Copy the state (CALM's state propagation). When token \(s\) exits at layer \(e_s\), use its last hidden state \(\mathbf{d}^{e_s}_s\) as if it were the hidden state of every skipped layer, and compute keys and values from it with each skipped layer's own projections:
    \[ \mathbf{k}^{i}_s = W^{i}_K\, \mathbf{d}^{e_s}_s, \qquad \mathbf{v}^{i}_s = W^{i}_V\, \mathbf{d}^{e_s}_s \qquad \text{for all } i \gt e_s \tag{121.8} \]
    This costs only the K and V projections (no attention, no MLP), and all of them can run in parallel because they share one input. CALM tested it with the oracle: ROUGE-L 38.24 against 38.31 for an oracle that never skips, so copying the hidden state barely hurts. Copying the projected keys and values of the exit layer instead dropped ROUGE-L to 23.02. [Certain] So the projections of the skipped layer matter.
  3. Never create holes (SkipDecode). Choose exit points so that a later token never needs a layer that an earlier token skipped (next section).
  4. Fill the holes in a later pass (LayerSkip). Treat the early-exit tokens as drafts, and let a full-depth verification pass compute the missing layers for all of them at once (section after next).
Worked example: counting the holes

Eight tokens, eight layers, exit layers \(e = (8, 3, 5, 2, 6, 8, 4, 3)\). The tokens compute \(39\) of the \(64\) (token, layer) cells. Now count, for each token \(t\) and each layer \(i \le e_t\), the earlier tokens \(s\) with \(e_s \lt i\): those are attention reads that hit a hole. There are 30 such missing reads. Recompute fixes them with 16 extra layer runs: each token must run up to the deepest exit of any later token, \(\sum_s \max(0, \max_{t \gt s} e_t - e_s) = 16\). State copying instead fills all \(64 - 39 = 25\) skipped cells with cheap K/V projections. Figure 4 shows the same grid.

computed: K and V in cacheextra compute (recompute)K, V projected from copied stateskipped: no K, Vhole that the current token needs
  1. for token \(t = 1\) to \(T\) (one decoding step each):
  2. run its layers \(R_t\): \(1..e_t\) (early exit) or \(\{1\} \cup\) the top \(e_t - 1\) (SkipDecode, formula (121.10))
  3. for each \(i \in R_t\): attention reads \(\mathbf{k}^i_s, \mathbf{v}^i_s\) of every \(s \lt t\)
  4. if \(\mathbf{k}^i_s\) is missing: a hole (no fix), or run layer \(i\) for token \(s\) now (recompute)
  5. after the exit: \(\mathbf{k}^j_t = W^j_K \mathbf{d}^{e_t}_t\) for \(j \gt e_t\) (copy state, formula (121.8))
Figure 4. The KV-cache problem on 8 tokens and 8 layers. Columns are tokens in order, rows are layers (layer 1 at the bottom). The animation generates the tokens one by one (or click a column to jump to that token). At each step the current token's attention reads at each of its layers are checked against the cache of the earlier tokens, and missing entries turn red. The readout counts computed cells, missing reads and extra work so far for the chosen strategy.
How to read Figure 4

Press Play. In "Early exit, no fix", tokens 1 to 3 are fine until token 3 (5 layers) reaches layers 4 and 5, where token 2 (exit at 3) has no keys: 2 holes. Token 5 (6 layers) hits 8 holes, and token 6, which runs all 8 layers, hits 16: the red cells in the columns of tokens 2, 3, 4 and 5 that stopped low. After all 8 tokens there are 30 missing reads, the number of the worked example. "Recompute" removes every hole, but the orange cells appear exactly when a later token needs them: 2 after token 3, 8 after token 5, 16 after token 6, so the total grows from 39 to 55 of 64. "Copy state" adds the green cells right after each exit (25 in total), with only two small matrix products each. "SkipDecode" uses the exit schedule (8, 7, 6, 5, 5, 4, 3, 2) of formula (121.10) and spends each token's layers at the top of the stack, after one warm-up layer. Every token computes a subset of the layers of every earlier token, so the count of holes stays at 0 with no extra work.

Algorithm 2: CALM token-level exit with state propagation (one decoding step)
  1. input: token \(y_t\), KV cache for all layers, threshold \(\lambda'(\lambda, t)\) from formula (121.7)
  2. \(\mathbf{d} \leftarrow \text{embed}(y_t)\)
  3. for \(i = 1\) to \(L\):
  4.   \(\mathbf{d} \leftarrow \text{layer}_i(\mathbf{d}, \text{cache}_i)\); append \(\mathbf{k}^i_t, \mathbf{v}^i_t\) to \(\text{cache}_i\)
  5.   \(c \leftarrow\) confidence of \(\mathbf{d}\) at layer \(i\) // formula (121.6)
  6.   if \(c \ge \lambda'\) or \(i = L\): break
  7. for every skipped layer \(j = i+1\) to \(L\): append \(W^j_K \mathbf{d},\ W^j_V \mathbf{d}\) to \(\text{cache}_j\) // state propagation, formula (121.8)
  8. return \(y_{t+1} = \argmax \softmax(W_{\text{head}}\, \mathbf{d})\)

SkipDecode: early exit that works with batching and caching

CALM still has two practical problems. In a batch, tokens of different requests exit at different layers, but the GPU runs one layer for the whole batch at once. So the batch is done only when its deepest token is done:

\[ \text{layers run at one step} = \max_{b \in \text{batch}} e_b, \qquad \text{useful fraction} = \frac{\sum_b e_b}{B \cdot \max_b e_b} \tag{121.9} \]

For a batch of 4 tokens with exits \((2, 3, 8, 4)\), the step runs 8 layers and only \(17 / 32 = 53\%\) of that work is needed. And the state copying of formula (121.8) still adds work for every skipped layer.

SkipDecode (Del Corro et al., 2023) removes both problems with one decision: the exit point is not chosen per token by a confidence. It is a fixed function of the position in the sequence, the same for every request in the batch, and it never increases:

\[ e_i = \begin{cases} L & i \lt P \\ \operatorname{round}\big((1 - t_i)\, e_{\max} + t_i\, e_{\min}\big) & i \ge P \end{cases}, \qquad t_i = \frac{i - P}{N - P} \tag{121.10} \]
Read the symbols
\(P\)
prompt length; prompt tokens run all \(L\) layers (they are processed in parallel in the prefill)
\(N\)
maximum sequence length
\(e_{\max}, e_{\min}\)
number of layers for the first and for the last generated token; they fix the compute budget in advance

Why decreasing? The paper measured that the loss per position falls along the sequence: later tokens have more context and are easier. [Certain] And because exits never increase, every earlier token has done at least as much work as the current one. The second idea: SkipDecode skips layers instead of stopping early. Each token runs one warm-up layer at the bottom and then the top layers of the stack. So any layer that the current token runs is a layer that all earlier tokens also ran, and their keys and values are in the cache. No holes, no copying, no recompute. With batch-wide exit points, there is no waiting for the slowest request either. [Certain]

SkipDecode was tested on OPT 1.3B and 6.7B (24 and 32 layers) on E2E, Reddit-TLDR and CNN-DM, with 2× to 5× speed-ups and small loss in quality; one warm-up layer worked best in all settings. [Certain] The price is flexibility: a hard token late in a sequence still gets few layers, because the rule ignores the token's actual difficulty.

LayerSkip: train for early exit, then self-speculate

All methods so far share a weakness: when an early exit is wrong, the error goes into the output. LayerSkip (Elhoushi et al., Meta, 2024) changes the goal. Early exit is used only to make a draft, and the full model checks the draft, as in speculative decoding (session 128). The output is then exactly what the full model would produce (with greedy decoding), and the early layers only decide the speed.

Training recipe

A normal LLM's middle layers are bad at predicting the next token through the final LM head: the head was trained only on the last layer. LayerSkip fixes this with two training changes, and no new heads.

Layer dropout. During training, skip each layer \(l\) for a random part of the batch. Early layers are dropped rarely, late layers often, so the model learns not to depend on its last layers:

\[ p_{l,t} = S(t)\, D(l)\, p_{\max}, \qquad D(l) = e^{\frac{l \ln 2}{L - 1}} - 1 \tag{121.11} \]

\(D(l)\) grows from 0 at layer 0 to 1 at the last layer \(L - 1\). \(S(t)\) scales over training steps: 1 for continued pretraining and fine-tuning, and a similar exponential curriculum when training from scratch. For Llama 2 7B (32 layers) with \(p_{\max} = 0.1\): layer 0 is never dropped, layer 15 with probability 0.040, layer 31 with probability 0.1. [Certain]

Early-exit loss. Apply the shared LM head \(g\) to the output of every layer and add the losses with normalized weights:

\[ J = \sum_{l=0}^{L-1} \tilde e(t, l)\; \mathrm{CE}\big(g(\mathbf{x}_{l+1}), Y\big), \qquad \tilde e(t, l) = \frac{C(t, l)\, e(l)}{\sum_{i} C(t, i)\, e(i)} \tag{121.12} \]

The scale \(e(l)\) grows roughly with \(l^2\) (\(e(l) = e_{\text{scale}} \sum_{i=0}^{l} i\), with a large weight for the last layer), so late layers count most. The binary curriculum \(C(t, l)\) turns on the exit loss for only some layers at each step (for example every \(R\)-th layer, rotating), because computing the loss of all layers at every step slowed training and hurt the last layer. [Certain] For Llama 2 7B continued pretraining, the paper used \(p_{\max} = 0.1\), \(e_{\text{scale}} = 0.2\) and \(R = 8\). [Certain]

Self-speculative decoding

After this training, the first \(E\) layers plus the shared head form a small, fast draft model that lives inside the big model. Decoding repeats two steps:

  1. Self-drafting. Generate \(d\) tokens autoregressively with only the first \(E\) layers (early exit at a fixed layer \(E\)).
  2. Self-verification. Run the remaining layers \(E+1, \dots, L\) on all \(d\) draft tokens in one parallel pass. Keep the draft tokens up to the first one where the full model disagrees, then add the full model's own token at that point.

The KV-cache problem disappears. The first \(E\) layers are the same in both steps, so their keys and values from drafting are already the right ones: there is one KV cache. The hidden state at layer \(E\) of each draft token is saved too (the "exit query cache"; together the KVQ cache), so verification starts directly at layer \(E + 1\). And the verification pass computes the missing layers \(E+1\) to \(L\) for all draft tokens, so after each cycle the cache is complete again. Compared with normal speculative decoding there is no second model in memory. [Certain]

How fast is it? Use the standard model from session 128: each draft token is accepted with probability \(\alpha\), independently. One cycle then yields on average \((1 - \alpha^{d+1})/(1 - \alpha)\) tokens. In the memory-bound decode phase (session 82), the cost of a pass is roughly the number of layers whose weights are read, whatever the number of tokens in the pass. So the \(d\) drafting steps cost \(d \cdot E / L\) of a full forward pass. The verification pass runs layers \(E+1..L\) for the draft positions and all layers for the last draft token (that gives the extra token), so it reads every layer once and costs about one full pass. This is the formula of session 128 with draft cost \(c = E/L\):

\[ \text{speed-up} \approx \frac{1 - \alpha^{d+1}}{(1 - \alpha)\left(1 + d\,E / L\right)} \tag{121.13} \]
Read the symbols
\(\alpha\)
probability that one draft token equals the full model's token (assumed independent for each token)
\(d\)
number of draft tokens per cycle
\(E\)
exit layer used for drafting; \(L\) is the total number of layers
\((1 - \alpha^{d+1})/(1 - \alpha)\)
expected tokens per cycle: the accepted drafts plus one token from the verifier
\(1 + dE/L\)
cost of one cycle in units of one full forward pass (the LM head and other overheads are ignored)
Worked example: self-speculation on a 32-layer model

\(L = 32\), draft with \(E = 8\) layers, \(d = 4\) draft tokens, \(\alpha = 0.75\). Tokens per cycle: \((1 - 0.75^5)/(1 - 0.75) = 3.05\). Cost: \(1 + 4 \cdot 8/32 = 2.0\) full passes. Speed-up \(\approx 3.05 / 2.0 = 1.53\times\). With \(d = 6\) the tokens rise to 3.47 but the cost to 2.5, so the speed-up falls to \(1.39\times\). The best choice here is \(d = 3\), with \(1.56\times\): more drafts are only worth it if \(\alpha\) is high.

speed-up for each \(d\) (current \(E\) and \(\alpha\))selected \(d\)1× line (no gain)
Figure 5. The speed-up model of formula (121.13) for a 32-layer model, such as Llama 2 7B. Bars show the speed-up for every number of draft tokens \(d\) from 1 to 12, for the chosen exit layer and acceptance rate.
How to read Figure 5

The trade-off is between \(E\) and \(\alpha\). A small \(E\) makes drafts cheap, but in a real model it also lowers \(\alpha\) (shallow layers guess worse). Here \(\alpha\) is a separate slider, so you can ask "how good must my 8-layer draft be?". Try \(E = 8\), \(\alpha = 0.5\): the best \(d\) is 1 and the gain is only 1.2×. Raise \(\alpha\) to 0.9: the best \(d\) grows to 6 and the speed-up reaches about 2.1×. Move \(E\) to 24: drafting is expensive and even with \(\alpha = 0.9\) the best is about 1.09×. Bars below the grey line mean the method is slower than normal decoding.

Algorithm 3: LayerSkip self-speculative decoding (greedy)
  1. input: prefix \(x\), exit layer \(E\), draft length \(d\), shared KVQ cache
  2. while not finished:
  3.   for \(j = 1\) to \(d\): run layers \(1..E\) on the last token, apply the shared head, append \(\hat y_j = \argmax\) // draft; writes K, V of layers 1..E and the layer-E state
  4.   run layers \(E+1..L\) on the draft positions in one pass, starting from the saved layer-E states (the last draft token also runs layers \(1..E\)) // verify; fills K, V of layers E+1..L
  5.   \(y_j \leftarrow\) the full model's token at each draft position \(j\)
  6.   \(m \leftarrow\) number of leading positions with \(\hat y_j = y_j\)
  7.   accept \(\hat y_1, \dots, \hat y_m\), then append \(y_{m+1}\) (the full model's correction) // output = full-model greedy output
  8.   drop the cache entries of the rejected draft tokens

Figure 6 runs Algorithm 3 on the toy decoder of Figure 2. The draft is the toy's own layer \(E\) read by the shared head; the verifier is the same model with all 8 layers. Because the verifier decides, the output is always the full model's sentence. Only the cost changes.

layers \(1..E\): draft (one step per token)layers \(E+1..L\): verify (one pass)accepted draftrejected drafttoken from the verifier
  1. while not finished:
  2. for \(j = 1\) to \(k = \min(d, \text{tokens left})\): run layers \(1..E\), \(\hat y_j = \argmax \softmax(W\mathbf{d}^E)\)
  3. run layers \(E+1..L\) on the \(k\) drafts in one pass: full-model tokens \(y_1, \dots, y_{k+1}\)
  4. \(m \leftarrow\) number of leading positions with \(\hat y_j = y_j\)
  5. accept \(\hat y_1..\hat y_m\), append \(y_{m+1}\); drop the cache of rejected drafts
  6. cost \(\mathrel{+}= 1 + kE/L\) full passes // denominator of formula (121.13)
Figure 6. LayerSkip self-speculative decoding, animated on the toy 8-layer decoder of Figure 2 (10 tokens). Top: the output so far. Middle: the draft tokens of the current cycle, each with its layer stack (blue: drafted with layers \(1..E\); orange: filled by the verification pass), the draft word above and the full model's word below. Bottom: the cost so far in full forward passes, against normal decoding (one pass per token).
How to read Figure 6

Press Play. With \(E = 3\) and \(d = 4\), the first cycle drafts "the cat sat on" with 3 layers each, the verification pass agrees with all four and adds a fifth token, "the": 5 tokens for \(1 + 4 \cdot 3/8 = 2.5\) passes. The second cycle drafts "mat . the cat"; the verifier rejects "cat" and writes "dog": 4 tokens for 2.5 passes. The last cycle drafts "dog" for the last position, the verifier writes "slept". In total 10 tokens cost 6.375 passes, a measured speed-up of 1.57×, and the sentence is exactly the full model's. 7 of 9 drafts were accepted (\(\hat\alpha = 0.778\)), and formula (121.13) with this \(\hat\alpha\) predicts only 1.29×: here the accepted drafts come in runs, not independently, which is the warning of the note below. Now move \(E\) to 6 with \(d = 3\): the drafts are as good, but each costs 6/8 of a pass, and the total of 10.75 passes is slower than normal decoding (0.93×). \(E = 1\) with \(d = 2\) gives 1.63×, the best in this toy, even though the 1-layer draft is right only 5 times in 9.

What LayerSkip measured

The paper trained several Llama models with the recipe (continued pretraining, pretraining from scratch, and fine-tuning) and reports speed-ups of up to 2.16× on CNN/DM summarization, 1.82× on coding and 2.0× on TOPv2 semantic parsing. [Certain] One row in its tables shows why verification matters. For Llama 2 7B after continued pretraining, plain early exit at layer 8 on one-shot CNN/DM gave ROUGE-2 0.012, against 0.079 for the full model; self-speculation from the same layer 8 (12 draft tokens) gave 0.078 at 1.86× speed. [Certain] So the early layers are much too weak to answer alone, but good enough to guess, and the verifier keeps the quality.

Putting it into code

The code below trains a tiny 6-layer decoder on a toy language (each token is the sum of the two before it, modulo 16) with a LayerSkip-style recipe: layer dropout (121.11), one shared exit head, and an early-exit loss weighted toward late layers (121.5). Then it measures how often each exit agrees with the last layer, runs CALM-style softmax-response exit, and runs self-speculative decoding with a 1-layer draft, checking that the output equals normal greedy decoding. For clarity it does not use a KV cache; every pass recomputes the prefix. It is deterministic on CPU and runs in under a minute.

import math, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
V, D, L, T = 16, 64, 6, 20          # vocab, width, layers, sequence length

class Block(nn.Module):
    def __init__(s):
        super().__init__()
        s.n1, s.n2 = nn.LayerNorm(D), nn.LayerNorm(D)
        s.qkv, s.o = nn.Linear(D, 3 * D), nn.Linear(D, D)
        s.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))
    def forward(s, h):
        q, k, v = s.qkv(s.n1(h)).chunk(3, dim=-1)
        a = F.scaled_dot_product_attention(q, k, v, is_causal=True)   # one head, causal
        h = h + s.o(a)
        return h + s.mlp(s.n2(h))

class TinyLM(nn.Module):
    def __init__(s):
        super().__init__()
        s.emb, s.pos = nn.Embedding(V, D), nn.Embedding(T, D)
        s.blocks = nn.ModuleList(Block() for _ in range(L))
        s.norm, s.head = nn.LayerNorm(D), nn.Linear(D, V)          # ONE shared exit head
    def forward(s, x, upto=L, drop=None):
        h = s.emb(x) + s.pos(torch.arange(x.shape[1]))
        outs = []
        for l in range(upto):
            if drop is None or not drop[l]:                         # layer dropout skips block l
                h = s.blocks[l](h)
            outs.append(s.head(s.norm(h)))                          # logits of exit l+1
        return outs

def batch(n):   # toy language: x[i+1] = x[i] + x[i-1] (mod V), two random start tokens
    x = torch.randint(0, V, (n, T))
    for i in range(2, T):
        x[:, i] = (x[:, i - 1] + x[:, i - 2]) % V
    return x

model = TinyLM(); opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
p_max = 0.1
p_l = [p_max * (math.exp(l * math.log(2) / (L - 1)) - 1) for l in range(L)]   # formula (121.11)
w = torch.arange(1, L + 1, dtype=torch.float); w = w / w.sum()                # later exits weigh more, formula (121.5)
for it in range(400):
    x = batch(64)
    drop = [bool(torch.rand(()) < p) for p in p_l]
    outs = model(x[:, :-1], drop=drop)
    loss = sum(w[l] * F.cross_entropy(o.reshape(-1, V), x[:, 1:].reshape(-1)) for l, o in enumerate(outs))
    opt.zero_grad(); loss.backward(); opt.step()

# 1) how often does exit l agree with the last layer? (CALM's oracle question)
model.eval(); x = batch(512)
with torch.no_grad():
    outs = model(x[:, :-1])
final = outs[-1].argmax(-1)
print("agree with last layer:", [round((o.argmax(-1) == final).float().mean().item(), 3) for o in outs])

# 2) token-level exit with the softmax response, formula (121.6): top-1 minus top-2
lam = 0.9
probs = torch.stack([o.softmax(-1) for o in outs])               # (L, n, T-1, V)
top2 = probs.topk(2, dim=-1).values; conf = top2[..., 0] - top2[..., 1]
ok = conf >= lam; ok[-1] = True                                   # the last layer always exits
exit_l = ok.float().argmax(0)                                     # first confident layer (0-based)
pred = torch.gather(probs.argmax(-1), 0, exit_l[None])[0]
print("avg layers:", round((exit_l + 1).float().mean().item(), 2), "of", L,
      "| same token as full model:", round((pred == final).float().mean().item(), 3))

# 3) self-speculative greedy decoding: draft with E layers, verify with all L
@torch.no_grad()
def greedy(prefix, n, upto=L):
    x = prefix.clone()
    for _ in range(n):
        x = torch.cat([x, model(x, upto)[-1][:, -1].argmax(-1, keepdim=True)], 1)
    return x

@torch.no_grad()
def self_spec(prefix, n, E=2, d=4):
    x, drafted, accepted, passes = prefix.clone(), 0, 0, 0
    while x.shape[1] < prefix.shape[1] + n:
        k = min(d, prefix.shape[1] + n - x.shape[1])
        draft = greedy(x, k, upto=E)                              # cheap: E layers per token
        full = model(draft, L)[-1][:, x.shape[1] - 1:].argmax(-1) # ONE pass of the full model
        passes += 1; prop = draft[:, x.shape[1]:]
        m = 0
        while m < k and prop[0, m] == full[0, m]:
            m += 1
        drafted += k; accepted += m
        x = torch.cat([x, prop[:, :m], full[:, m:m + 1]], 1)[:, :prefix.shape[1] + n]
    return x, accepted / drafted, passes

prefix = batch(1)[:, :3]
ref = greedy(prefix, 16)
out, acc, passes = self_spec(prefix, 16, E=1)
print("identical to full greedy:", torch.equal(out, ref), "| acceptance:", round(acc, 2), "| full passes:", passes, "for 16 tokens")

On our run (PyTorch 2.8, CPU) it printed: agreement with the last layer [0.854, 0.945, 0.961, 0.988, 0.996, 1.0] for exits 1 to 6; with \(\lambda = 0.9\), an average of 1.88 layers out of 6 while choosing the same token as the full model 99.9% of the time; and for self-speculation, identical to full greedy: True with 81% of the draft tokens accepted. The exact numbers can change with other PyTorch versions [Likely]; the True must not.

Where this is used

SettingMethod that fitsWhy
Text classification at high volume (spam, moderation, intent, sentiment)DeeBERT / FastBERT / PABEE exits on an encoderOne decision per input, no cache; most inputs are easy
Phones and edge devices with a latency budgetEarly exit with a threshold chosen per deviceOne trained model serves several speed–accuracy points (FastBERT's "speed" knob)
Video classification and action recognitionExits on the per-frame or per-clip backbone; stop when the label is stable over several exits (patience)Many frames or clips are easy (static scene, clear action), so average compute falls [Likely]
Key-moment detection on live video streamsA cheap early exit runs on every frame; only uncertain frames go deeperMost frames contain nothing interesting; the deep layers are saved for candidates [Likely]
Video retrieval and fingerprintingEarly-exit embeddings for a first, coarse search; full-depth embeddings to re-rank the top resultsSame idea as draft-and-verify: cheap guess, exact check [Speculation]
LLM generation at batch size 1 (local assistants)CALM-style exit, or LayerSkip self-speculationDecoding is memory-bound; fewer layers per token means fewer weights to read
LLM serving with large batchesSkipDecode (batch-wide, position-based exits) or self-speculationPer-token exits break batching and the KV cache; these two keep both
Summarization and long outputsLayerSkip self-speculation (CNN/DM: up to 2.16×)Long outputs with many predictable tokens give high acceptance

LayerSkip checkpoints for several Llama models were released, and self-speculation with an early-exit draft is available in the Hugging Face transformers generation API. [Likely] Static layer removal (deleting layers that change the hidden state little) is the "same depth for every token" version of these ideas, and belongs with session 115.

Common mistakes

  • Adding exits to a model that was not trained for them. The LM head of a normal LLM gives poor predictions from middle layers. Early exit needs an exit loss (121.5, 121.12) or separately trained exit heads, or the speed-up comes with a large quality loss.
  • Ignoring the cost of the exit check. For an LLM, one softmax over the vocabulary can cost more than a whole layer (2.4 layers for Llama 3 8B). Check exits only at a few layers, use a cheap classifier, or compute the head only for the top candidates.
  • Leaving holes in the KV cache. If a token exits and nothing fills its skipped layers, later tokens attend to missing or stale keys and values. Use state propagation (121.8), position-based skipping, or a verification pass.
  • Copying the exit layer's K and V into the skipped layers. CALM measured that this breaks quality (ROUGE-L 23.02 instead of 38.24). Copy the hidden state and apply each layer's own \(W_K, W_V\).
  • Measuring speed-up at batch size 1 and deploying with big batches. With per-token exits, a batch waits for its slowest member (121.9); the speed-up can vanish.
  • Trusting a single confident exit. Over-confident early layers make wrong early exits. Patience (PABEE), calibration (CALM) or verification (LayerSkip) protect against this.
  • Using one threshold for all positions in generation. Early errors in a sequence propagate; a decaying threshold (121.7) or a decreasing schedule (121.10) spends more compute where it matters.
  • Too many draft tokens in self-speculation. Each extra draft costs \(E/L\) of a pass; with a low acceptance rate the speed-up falls, and can drop below 1× (Figure 5).

Interview questions

What is early exit, and what two choices define an early-exit method?basic
Early exit adds prediction heads after intermediate layers and stops the forward pass at the first layer whose prediction is judged good enough. A method is defined by (1) how the exits are trained, for example DeeBERT's two-stage training, FastBERT's self-distillation, or a weighted loss over all layers, and (2) the exit rule, for example entropy below a threshold, a patience counter, or a softmax-response or classifier confidence above a calibrated threshold. The speed-up is \(L / (\E[\ell](1 + c))\), so a cheap check (small \(c\)) matters as much as a small average depth.
Compare DeeBERT, FastBERT and PABEE.core
All three put a classifier after each BERT layer. DeeBERT exits when the entropy of the off-ramp's distribution is below \(S\); it trains the backbone and final classifier first, then freezes them and trains the off-ramps. FastBERT normalizes the entropy by \(\log N\) and trains the student exits by self-distillation (KL to the final classifier), which can use unlabeled data. PABEE ignores confidence and exits when \(t\) consecutive layers predict the same class; it trains all exits jointly with a depth-weighted loss and can even beat the full model by avoiding "overthinking" (1.57× faster with a better score on ALBERT-base).
Why is early exit easy for BERT classification but hard for LLM generation?core
In classification there is one decision per input, the exit head is small, and nothing after the decision depends on the skipped layers. In generation, there is a decision per token; the exit head is the full LM head (expensive for big vocabularies); an early error is copied forward by autoregression; and later tokens attend to the keys and values of earlier tokens at every layer, which do not exist for layers an earlier token skipped. Batching adds another problem: a batch is only as fast as its deepest token.
Explain the KV-cache problem of token-level early exit and the ways to solve it.core
Token \(t\) at layer \(i\) needs \(\mathbf{k}^i_s, \mathbf{v}^i_s\) for all \(s \lt t\). If token \(s\) exited below layer \(i\), those were never computed. Fixes: (1) recompute the missing layers when needed, exact but it gives back most of the saving; (2) CALM's state propagation, which computes \(W^i_K \mathbf{d}^{e_s}_s\) and \(W^i_V \mathbf{d}^{e_s}_s\) from the exit hidden state, cheap and almost lossless in their oracle test; (3) SkipDecode, where exits are monotonically decreasing over positions and run the top layers, so no hole is ever needed; (4) LayerSkip, where a full-depth verification pass fills the missing layers for all draft tokens.
What confidence measures does CALM use, and what are their trade-offs?core
Softmax response: the gap between the top-2 probabilities of the LM head at that layer. The most accurate, but it needs a full vocabulary projection per checked layer. Hidden-state saturation: cosine similarity between consecutive hidden states. Free, no parameters, but weak, often below a static baseline. Early-exit classifier: a linear layer with \(d+1\) weights trained to predict whether the exit matches the last layer. Very cheap, and sometimes faster overall than softmax even though it uses more layers.
Why does CALM use a decaying threshold, and how is the threshold chosen?advanced
CALM found that errors early in the generated sequence hurt the final output more, because later tokens condition on them. The threshold \(\lambda'(\lambda, t) = \text{clip}(0.9\lambda + 0.1e^{-\tau t/N})\) is strict for early positions and looser later. The base \(\lambda\) is calibrated on held-out data with Learn then Test, a multiple-testing procedure that picks the most aggressive \(\lambda\) for which the risk (textual distance to the full model's output, or loss of quality against references) stays below \(\delta\) with high probability.
How does SkipDecode stay compatible with batching and the KV cache?advanced
It sets one exit point per sequence position for the whole batch, so no request waits for another. The exit points decrease linearly from \(e_{\max}\) to \(e_{\min}\) over the positions, so every earlier token has done at least as much work. And it skips the lower-middle layers instead of the top ones (one warm-up layer, then the top layers), so the layers a later token runs are always a subset of what earlier tokens ran: their keys and values are in the cache. The cost: the schedule ignores the actual difficulty of a token.
Describe the LayerSkip training recipe.core
Two parts. Layer dropout: during training, layer \(l\) is skipped with probability \(S(t) D(l) p_{\max}\), where \(D(l) = e^{l\ln2/(L-1)} - 1\) rises from 0 at the first layer to 1 at the last, so late layers are dropped most. Early-exit loss: the one shared LM head is applied to every layer's output and the cross-entropies are summed with normalized weights that grow about quadratically with depth; a curriculum enables only some layers' exit losses per step (rotational or gradual), because all exits at every step slowed training and hurt the last layer. No extra heads are added.
How does self-speculative decoding differ from normal speculative decoding?core
The draft model is the first \(E\) layers of the same model with the shared head, not a separate small model. So there is no second set of weights in memory, the draft and the target share one KV cache for layers \(1..E\), and verification starts from the saved layer-\(E\) states (KVQ cache), running only layers \(E+1..L\). As with normal speculative decoding, greedy verification makes the output identical to the full model's greedy output; the early layers only change the speed.
A 32-layer model drafts with 8 layers, 4 tokens per cycle, acceptance 0.75. Estimate the speed-up. What if acceptance is 0.5?advanced
Tokens per cycle \((1 - 0.75^5)/0.25 = 3.05\); cost \(1 + 4 \cdot 8/32 = 2.0\) passes (four 8-layer draft steps plus one verification pass that reads all layers); speed-up \(\approx 1.53\times\). With \(\alpha = 0.5\): tokens \((1 - 0.5^5)/0.5 = 1.94\), same cost, speed-up \(\approx 0.97\times\), slower than normal decoding. With low acceptance, fewer drafts are better: \(d = 1\) gives \(1.5 / 1.25 = 1.2\times\). This assumes the memory-bound regime and independent acceptances.
Can early exit improve accuracy? When?advanced
Yes, sometimes. PABEE shows "overthinking": an intermediate layer can be right while a later layer is wrong, and requiring agreement across several exits also acts like a small ensemble. On ALBERT-base, PABEE was both faster and more accurate than the full model. For generation with verification (LayerSkip), quality can at best match the full model with greedy decoding; without verification, early exit usually costs some quality, controlled by the threshold.

Exercises

  1. A 24-layer model has exits after every layer. 40% of inputs exit at layer 4, 40% at layer 12 and 20% at layer 24. Each exit check costs 5% of a layer. Compute the speed-up with formula (121.1).
  2. Three classes, an exit outputs \(\mathbf{p} = (0.7, 0.2, 0.1)\). Compute the entropy and FastBERT's normalized uncertainty \(u\).
  3. PABEE with patience \(t = 2\). The layer predictions are A, A, B, B, B, A, ... At which layer does the input exit, and with which label?
  4. In Figure 4, change the exits to \(e = (4, 4, 4, 4, 4, 4, 4, 4)\) (static depth 4). How many missing reads are there with no fix? Why?
  5. SkipDecode with \(L = 32\), prompt length \(P = 100\), \(N = 200\), \(e_{\max} = 24\), \(e_{\min} = 8\). How many layers do positions 100, 150 and 200 use?
  6. For LayerSkip with \(L = 16\) and \(p_{\max} = 0.2\), compute the dropout probability of layers 0, 8 and 15 (with \(S(t) = 1\)).
  7. Self-speculation with \(L = 32\), \(E = 16\), \(d = 3\), \(\alpha = 0.8\). Compute the speed-up with formula (121.13). Is it better than \(E = 8\), \(d = 3\), \(\alpha = 0.6\)?
Answerscheck
1. \(\E[\ell] = 0.4 \cdot 4 + 0.4 \cdot 12 + 0.2 \cdot 24 = 11.2\); cost \(11.2 \times 1.05 = 11.76\); speed-up \(24 / 11.76 = 2.04\times\). 2. \(H = -(0.7\ln0.7 + 0.2\ln0.2 + 0.1\ln0.1) = 0.802\) nats; \(u = 0.802/\ln 3 = 0.730\). 3. Counts: layer 1: 0, layer 2: 1, layer 3: 0, layer 4: 1, layer 5: 2. It exits at layer 5 with label B. 4. Zero. Every token runs layers 1 to 4, and every later token only needs layers 1 to 4 of earlier tokens, which exist. A static depth is just a smaller model; holes come from different depths, and only when a later token goes deeper than an earlier one. 5. \(t = 0, 0.5, 1\): 24, 16 and 8 layers. 6. \(D(l) = e^{l \ln 2/15} - 1\): layer 0: 0; layer 8: \(2^{8/15} - 1 = 0.447\), so \(p = 0.089\); layer 15: \(p = 0.2\). 7. Tokens \((1 - 0.8^4)/0.2 = 2.952\); cost \(1 + 3 \cdot 16/32 = 2.5\); speed-up \(1.18\times\). The other: tokens \((1 - 0.6^4)/0.4 = 2.176\); cost \(1 + 3 \cdot 8/32 = 1.75\); speed-up \(1.24\times\). The shallow draft wins: it guesses worse, but its guesses are half the price.

Key takeaways

  • Dynamic inference spends compute per input: early exit stops after a few layers when the model is sure; the speed-up is \(L / (\E[\ell](1+c))\) (121.1), and the exit cost \(c\) is large for LLM heads.
  • For classification, exits are easy: DeeBERT uses entropy (121.2), FastBERT normalized entropy and self-distillation (121.3), PABEE a patience counter (121.4) that is robust to over-confident layers.
  • For generation, CALM exits per token with a softmax-response, saturation or classifier confidence (121.6), a decaying threshold (121.7) and calibration; up to about 3× on T5.
  • Token-level exit leaves holes in the KV cache. Fix them by recomputing, by copying the hidden state through each layer's own projections (121.8), or avoid them with SkipDecode's batch-wide, decreasing, top-layer schedule (121.10).
  • LayerSkip trains with layer dropout (121.11) and a shared-head early-exit loss (121.12), then drafts with the first \(E\) layers and verifies with the rest: one model, one KV cache, the full model's greedy output, up to 2.16× on CNN/DM.

Key formulas

\[ \text{speed-up} = \frac{L}{\E[\ell](1+c)}, \qquad H(\mathbf{p}) \lt S, \qquad \mathbf{k}^i_s = W^i_K \mathbf{d}^{e_s}_s \]
\[ \lambda' = 0.9\lambda + 0.1e^{-\tau t/N}, \qquad \text{speed-up}_{\text{self-spec}} \approx \frac{1-\alpha^{d+1}}{(1-\alpha)(1 + dE/L)} \]

References for this session

  1. Xin et al. (2020). DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
  2. Liu et al. (2020). FastBERT: a Self-distilling BERT with Adaptive Inference Time
  3. Zhou et al. (2020). BERT Loses Patience: Fast and Robust Inference with Early Exit (PABEE)
  4. Schuster et al. (2022). Confident Adaptive Language Modeling
  5. Del Corro et al. (2023). SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference
  6. Elhoushi et al. (2024). LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
  7. Xin et al. (2020). DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
  8. Liu et al. (2020). FastBERT: a Self-distilling BERT with Adaptive Inference Time
  9. Zhou et al. (2020). BERT Loses Patience: Fast and Robust Inference with Early Exit (PABEE)
  10. Schuster et al. (2022). Confident Adaptive Language Modeling
  11. Del Corro et al. (2023). SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference
  12. Elhoushi et al. (2024). LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
  13. Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
  14. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models