LLM & VLM courseBlog
128Track 5: LLM inference and serving · session 51 of 65

Speculative decoding 1: the math

In the decode phase, a large model reads all of its weights to make one token, and the GPU waits on memory most of the time. Speculative decoding lets a small draft model guess several tokens, and the large model checks all the guesses in one pass. This session derives the acceptance rule that keeps the large model's exact output distribution, the expected number of tokens per step, the speed-up formula, and the cases where speculation makes things slower.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Decoding strategies (session 79), the KV cache (session 80), Why inference is slow and the roofline (session 82)

What you will learn

  • Why one forward pass on 5 tokens costs about the same as on 1 token in the decode phase, and how speculative decoding uses this
  • The draft-then-verify loop: the draft proposes γ tokens, the target scores all of them in one pass
  • The acceptance rule min(1, p/q) and the residual distribution, and a short proof that the output has exactly the target distribution
  • The acceptance rate α = Σ min(p, q) = 1 − total variation distance, and what greedy decoding and temperature do to it
  • Expected tokens per step (1 − α^(γ+1))/(1 − α), the wall-time speed-up, the best γ, and the rule α > c
  • Why speculation stops paying at large batch sizes (the compute-bound regime) and with weak drafts, with a roofline model of an H100
  • What the original papers measured: T5-XXL with a T5-small draft (2.3× to 3.4×) and Chinchilla 70B with a 4B draft (1.92× to 2.46×)

Why this matters

A 7B model on an H100 needs about 4 ms to make one token at batch size 1. It needs almost the same 4 ms to score 5 tokens at once. Speculative decoding turns that fact into a 2 to 3 times faster generation, with exactly the same output distribution as the big model.

In session 82 we saw that the decode phase is memory-bound. Each new token needs one full read of the weights, and the GPU’s arithmetic units sit idle most of the time. The previous sessions made each step cheaper: faster kernels for the matrix-vector products (session 126), fewer kernel launches (session 127). This session attacks a different cost: the number of sequential steps.

The idea is simple. A small, fast draft model guesses the next few tokens. The large target model (the model whose output we want) then checks all the guesses in one forward pass. If the guesses are good, one weight read of the big model produces several tokens. Two independent papers proposed this in late 2022 and early 2023: Leviathan, Kalman and Matias at Google (“speculative decoding”) and Chen et al. at DeepMind (“speculative sampling”). [Certain] Both use the same acceptance rule. That rule is the heart of this session, because it makes the method lossless: the tokens come from the target model’s distribution, not from an approximation of it.

Speculative decoding is in every major serving engine (vLLM, SGLang, TensorRT-LLM, llama.cpp) and in Hugging Face transformers as “assisted generation”. [Likely] The next session, session 129, is about better drafters (Medusa, EAGLE, Lookahead, tree verification). All of them reuse the math from this page.

The opportunity: checking is cheaper than writing

Recall the decode cost model from session 82. A dense model with \(N\) parameters stored in BF16 (2 bytes each) must read \(2N\) bytes of weights per forward pass, and does about \(2N\) FLOPs per token in that pass. If the pass processes \(n\) tokens (from one sequence or many), the weights are read once and used \(n\) times. On a GPU with memory bandwidth \(BW\) and peak compute \(P\), the pass takes about

\[ t(n) \approx \max\!\left( \frac{2N}{BW},\ \frac{2N\,n}{P} \right) \tag{128.1} \]
Read the symbols
\(N\)
number of model parameters
\(n\)
number of tokens processed in the same forward pass
\(BW\)
memory bandwidth in bytes per second (3.35 TB/s for an H100 SXM)
\(P\)
peak dense BF16 compute in FLOP per second (989 TFLOP/s for an H100 SXM)

This ignores the KV cache reads and kernel overheads, but it captures the key point. The first term (memory) does not depend on \(n\). The second term (compute) grows with \(n\). The two are equal at \(n^* = P/BW = 989/3.35 \approx 295\) tokens: the ridge point of the roofline. Below 295 tokens per pass, the pass is memory-bound and its time is flat. For a 7B model: \(t(1) = 14 \times 10^9 / 3.35 \times 10^{12} = 4.18\) ms, and \(t(5) = 4.18\) ms too.

So in the decode phase, scoring 5 tokens costs about the same as scoring 1. But we cannot simply score 5 future tokens, because we do not know them yet: token 3 depends on token 2. Speculative decoding fills this gap with guesses. The guesses come from a model that is 10 to 100 times cheaper, and the big model only verifies them. Verification is parallel: one pass over the context plus the \(\gamma\) guessed tokens gives the target’s next-token distribution at every one of the \(\gamma + 1\) positions, exactly as in training with teacher forcing.

target model passdraft model stepaccepted draft tokentoken the target adds itself
Figure 1. Plain decoding (top) makes one token per target pass. Speculative decoding (bottom) runs the draft \(\gamma = 4\) times, then one target pass checks the 4 guesses. Here 3 guesses are accepted and the target adds a fourth token itself: 4 tokens for one target pass plus 4 short draft steps. The draft steps are drawn at 10% of a target pass to keep them visible.

The draft-then-verify loop

Let \(p(\cdot \mid x_{\lt t})\) be the target model’s next-token distribution, and \(q(\cdot \mid x_{\lt t})\) the draft model’s. Both use the same tokenizer and the same sampling settings (temperature, top-p), applied to their own logits. One step of speculative decoding does three things:

  1. Draft. Sample \(\gamma\) tokens one by one from the draft: \(\tilde x_1 \sim q(\cdot \mid x)\), \(\tilde x_2 \sim q(\cdot \mid x, \tilde x_1)\), and so on. Keep the draft probabilities \(q_i = q(\cdot \mid x, \tilde x_{\lt i})\).
  2. Verify. Run the target once on the context plus \(\tilde x_1 \dots \tilde x_\gamma\). This gives \(\gamma + 1\) target distributions \(p_1, \dots, p_{\gamma+1}\): \(p_i\) is the target’s distribution for the token at draft position \(i\), and \(p_{\gamma+1}\) is the one after the last guess.
  3. Accept or reject the guesses from left to right with the rule below. Stop at the first rejection, and replace the rejected token with a token sampled from a corrected distribution. If all \(\gamma\) guesses pass, sample one extra bonus token from \(p_{\gamma+1}\), which we already have for free.

Every step therefore produces at least 1 token (the same as plain decoding) and at most \(\gamma + 1\). After a rejection, the later guesses are thrown away: their contexts contain the rejected token, so the target’s distributions for them are no longer valid.

Algorithm 1: one speculative decoding step
  1. input: prefix \(x\), draft model \(q\), target model \(p\), draft length \(\gamma\)
  2. for \(i = 1 \dots \gamma\): \(q_i \leftarrow q(\cdot \mid x, \tilde x_{\lt i})\); sample \(\tilde x_i \sim q_i\) // \(\gamma\) cheap sequential steps
  3. \(p_1, \dots, p_{\gamma+1} \leftarrow p(\cdot \mid x),\ p(\cdot \mid x, \tilde x_1),\ \dots,\ p(\cdot \mid x, \tilde x_{1..\gamma})\) // ONE parallel target pass
  4. for \(i = 1 \dots \gamma\):
  5.   draw \(r \sim \text{Uniform}(0, 1)\)
  6.   if \(r \lt \min\big(1,\ p_i(\tilde x_i) / q_i(\tilde x_i)\big)\): append \(\tilde x_i\) to \(x\) // accept, formula (128.2)
  7.   else: sample \(y \sim \text{norm}\big(\max(0,\ p_i - q_i)\big)\); append \(y\); return // reject, formula (128.3)
  8. sample \(y \sim p_{\gamma+1}\); append \(y\) // all accepted: bonus token
  9. roll back the KV caches of both models to the accepted length

Figure 2 runs this algorithm for real, four steps in a row. The two models are tiny bigram models over 8 words, so you can see every probability. The algorithm box highlights the line that is running, and the panel under it shows the formula with the current numbers in it.

Algorithm 1, running
  1. input: prefix \(x\), draft \(q\), target \(p\), length \(\gamma\)
  2. for \(i = 1 \dots \gamma\): \(q_i \leftarrow q(\cdot \mid x, \tilde x_{\lt i})\); \(\tilde x_i \sim q_i\)
  3. \(p_1 \dots p_{\gamma+1} \leftarrow \text{target}(x, \tilde x_{1..\gamma})\) // one pass
  4. for \(i = 1 \dots \gamma\):
  5.   \(r \sim U(0,1)\); \(a_i \leftarrow \min\big(1,\ p_i(\tilde x_i) / q_i(\tilde x_i)\big)\)
  6.   if \(r \lt a_i\): accept \(\tilde x_i\)
  7.   else: \(y \sim \text{norm}\big(\max(0,\ p_i - q_i)\big)\); stop
  8. if all accepted: \(y \sim p_{\gamma+1}\) // bonus
  9. emit; roll back both KV caches
Output so far
Press “Run 5,000 steps” to measure the average number of tokens per target pass.
Figure 2. Speculative sampling, computed live in your browser with two bigram models over 8 words. The draft \(q\) is a noisy copy of the target \(p\); the word sequences are random and not meant to make sense. Top row: the last context word and the \(\gamma\) draft slots (dashed orange = guessed, green = accepted, red = rejected, faded = discarded, purple = token from the residual, teal = bonus token). Chart: blue bars are the target \(p_i\), orange bars the draft \(q_i\) at the current position. Right of the chart: the acceptance test, a random number \(r\) falling onto \([0, 1]\), accepted if it lands in the green part below \(a_i\). Bottom strip: inverse-CDF sampling, the way every token here is drawn.
How to read Figure 2

Press Play, or Step to move one line of the algorithm at a time (Back goes back). First the draft samples its guesses one by one from \(q\): the pointer on the bottom strip is the uniform random number, and the word whose segment it hits is the guess. Then one target pass gives \(p\) at every position (the numbers under each slot). Then each guess is tested: \(a_i = \min(1, p_i/q_i)\) is the green part of the bar on the right, and \(r\) falls onto it. A guess the target likes at least as much as the draft (\(p \ge q\)) has \(a_i = 1\) and always passes. At the first rejection the chart switches to the residual \(\max(0, p_i - q_i)\) in purple: only words the draft under-weights can come out. If all guesses pass, a bonus word is sampled from \(p_{\gamma+1}\). After each step the panel compares the running tokens per target pass with formula (128.7). Raise the temperature to flatten both models and watch \(\alpha\) change; change the seed for other outcomes. “Run 5,000 steps” runs the same loop from random start words without animation.

The acceptance rule

Why not just accept a guess when it equals the target’s most likely token? That works for greedy decoding (we come back to it below), but for sampling it would change the output distribution. We want each emitted token to be distributed exactly as \(p\). The tool for this is a form of rejection sampling: draw from an easy distribution, then accept or reject so that the accepted samples follow the hard one.

Drop the context for a moment and look at one position. The draft proposes \(\tilde x \sim q\). Accept it with probability

\[ P(\text{accept } \tilde x) = \min\!\left(1,\ \frac{p(\tilde x)}{q(\tilde x)}\right) \tag{128.2} \]

If the target gives the token at least as much probability as the draft, accept it always. If the draft is overconfident about the token (\(q(\tilde x) \gt p(\tilde x)\)), accept it only part of the time. When we reject, we sample a replacement from the residual distribution:

\[ p'(x) = \frac{\max\big(0,\ p(x) - q(x)\big)}{\sum_{x'} \max\big(0,\ p(x') - q(x')\big)} \tag{128.3} \]
Read the symbols
\(p(x)\)
target model’s probability of token \(x\) at this position (after temperature and top-p)
\(q(x)\)
draft model’s probability of the same token, with the same sampling settings
\(\tilde x\)
the token the draft actually sampled
\(p'\)
the residual: only the tokens where the target wants more probability than the draft gave them

The residual is exactly the “missing mass”. The accepted draft tokens supply probability \(\min(p, q)\) to each token. Some tokens need more than that: \(p(x) - q(x)\) more. The residual supplies precisely this difference.

The proof: the output follows \(p\) exactly

Fix a token \(x\). There are two ways to output \(x\): the draft proposes \(x\) and we accept it, or we reject whatever the draft proposed and then the residual gives \(x\). Let \(\beta = \sum_{x'} \min(p(x'), q(x'))\) be the total probability of accepting. Then:

\[ \begin{aligned} P(X = x) &= q(x)\,\min\!\left(1, \frac{p(x)}{q(x)}\right) + (1 - \beta)\, p'(x) \\ &= \min\big(p(x), q(x)\big) + (1 - \beta)\,\frac{\max\big(0,\ p(x) - q(x)\big)}{1 - \beta} \\ &= \min\big(p(x), q(x)\big) + \max\big(0,\ p(x) - q(x)\big) = p(x) \end{aligned} \tag{128.4} \]

Two small facts make it work. First, \(q \cdot \min(1, p/q) = \min(q, p)\). Second, the normalizer of the residual is \(\sum \max(0, p - q) = \sum (p - \min(p, q)) = 1 - \beta\), which is exactly the rejection probability, so it cancels. The last line uses \(\min(a, b) + \max(0, a - b) = a\) for any two numbers.

For a whole sequence, apply the same argument position by position. Given the accepted prefix, the next emitted token has distribution \(p(\cdot \mid \text{prefix})\), whether it came from an accepted guess, the residual, or the bonus token (which is sampled from \(p\) directly). By the chain rule of probability, the joint distribution of the full output equals the target’s. [Certain] This is Theorem 1 in Chen et al.; Leviathan et al. prove the same result.

Worked example: one position, four tokens

Target \(p = (0.45, 0.25, 0.20, 0.10)\), draft \(q = (0.30, 0.40, 0.10, 0.20)\) over tokens A, B, C, D.

Acceptance probabilities \(\min(1, p/q)\): A: \(\min(1, 1.5) = 1\); B: \(0.25/0.40 = 0.625\); C: \(1\); D: \(0.10/0.20 = 0.5\).

Accepted mass \(\min(p, q) = (0.30, 0.25, 0.10, 0.10)\), so \(\beta = 0.75\): three draft tokens out of four survive on average.

Missing mass \(\max(0, p - q) = (0.15, 0, 0.10, 0)\), total \(0.25 = 1 - \beta\). The residual is \(p' = (0.6, 0, 0.4, 0)\): after a rejection we can only output A or C, the two tokens the draft under-weights.

Check token A: \(0.30 + 0.25 \times 0.6 = 0.45\). Token B: \(0.25 + 0 = 0.25\). Token C: \(0.10 + 0.25 \times 0.4 = 0.20\). Token D: \(0.10\). This is exactly \(p\).

target p (outline)draft q (dashed)output via accepted guessoutput via residual
The rule at one position
  1. \(\tilde x \sim q\) // draft guess
  2. \(r \sim U(0,1)\); \(a \leftarrow \min\big(1,\ p(\tilde x) / q(\tilde x)\big)\)
  3. if \(r \lt a\): output \(\tilde x\) // (128.2)
  4. else: output \(y \sim \text{norm}\big(\max(0,\ p - q)\big)\) // (128.3)
Figure 3. The acceptance rule at one position, as a simulation. The draft is \(q = p + \lambda (q_0 - p)\), where \(q_0\) is the draft of the worked example; \(\lambda = 1\) is the worked example and \(\lambda = 0\) is a perfect draft. Each sample runs the four lines of the box under the chart with a seeded random generator. The wide faint bars are the theory of (128.4): \(\min(p, q)\) in green plus \((1 - \alpha)\,p'\) in purple. The narrow solid bars are the measured output frequencies, split by the path that produced them.
How to read Figure 3

Press Step to draw one sample. The orange arrow marks the draft guess \(\tilde x\); on the right, \(r\) falls onto the bar \([0, 1]\), whose green part is \(a = \min(1, p(\tilde x)/q(\tilde x))\). If \(r\) lands in the red part, the guess is rejected and a purple arrow marks the word drawn from the residual. Press Play and raise the speed: the narrow bars grow into the wide faint bars. The green part of each narrow bar tends to \(\min(p, q)\), the purple part to \(\max(0, p - q)\), and the total to \(p\), for every \(\lambda\): that is the proof (128.4) as an experiment. The table lists the same numbers. Move \(\lambda\) to the right and the draft drifts away from the target: green shrinks, purple grows, and \(\alpha = 1 - 0.25\lambda\). At \(\lambda = 0\) nothing is ever rejected; at \(\lambda = 2\) the draft never proposes C, so C comes out only through the residual.

The acceptance rate α

The probability that a guess passes, averaged over what the draft proposes, is

\[ \alpha = \sum_x q(x)\,\min\!\left(1, \frac{p(x)}{q(x)}\right) = \sum_x \min\big(p(x), q(x)\big) = 1 - \tfrac12 \sum_x \big|p(x) - q(x)\big| \tag{128.5} \]

The last term, \(\tfrac12 \sum |p - q|\), is the total variation distance (TV) between the two distributions: the largest possible difference in the probability they give to any event. So \(\alpha = 1 - \text{TV}(p, q)\). [Certain] Leviathan et al. call this distance \(D_{LK}\) and show the same identity. It tells you what a draft must do to be useful: it does not need to be a good language model in general, only close to the target on the contexts that appear.

The acceptance rate depends on the context: after “The capital of France is”, almost any draft agrees with the target; in the middle of a creative story, much less. In practice \(\alpha\) means the average over the contexts you serve. The formulas below assume that every guess passes independently with the same \(\alpha\). That is a simplification, and the real numbers in the last section show how far it can be off.

Greedy decoding and temperature

With temperature 0 (greedy decoding), \(p\) and \(q\) put all their mass on their top tokens. Then \(\min(p, q)\) is 1 if the two top tokens are the same and 0 otherwise. The rule becomes: accept a guess if it equals the target’s argmax; on rejection, output the target’s argmax. The output is exactly the target’s greedy output, and \(\alpha\) is the probability that the two models agree on the top token. Leviathan et al. note that argmax sampling is the same as zeroing out the non-max probabilities and normalizing, so the general rule covers it. [Certain]

In their T5-XXL experiments, \(\alpha\) was higher at temperature 0 than at temperature 1 for every draft: for example 0.75 versus 0.62 with T5-small on English-to-German translation. [Certain] One way to see why: agreeing on the top token is easier than matching a whole distribution, which also includes the noisy tail. [Likely] Temperature, top-k and top-p must be applied to both \(p\) and \(q\) before the rule; the rule then guarantees the target’s distribution with those settings.

Expected tokens per step

Let \(K\) be the number of accepted guesses in one step. With independent acceptances, the first rejection is like the first “tails” in a series of coin flips with “heads” probability \(\alpha\), but cut off at \(\gamma\):

\[ P(K = k) = (1 - \alpha)\,\alpha^k \ \ \text{for } k \lt \gamma, \qquad P(K = \gamma) = \alpha^\gamma \tag{128.6} \]

Each step also emits one token from the target (the replacement or the bonus), so it produces \(K + 1\) tokens. The expected value is a short geometric sum. Since \(K + 1 \ge j\) exactly when the first \(j - 1\) guesses all pass, \(P(K + 1 \ge j) = \alpha^{j-1}\) for \(j = 1 \dots \gamma + 1\), and

\[ \E[\text{tokens per step}] = \sum_{j=1}^{\gamma+1} \alpha^{j-1} = 1 + \alpha + \alpha^2 + \dots + \alpha^\gamma = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha} \tag{128.7} \]

This is equation (1) of Leviathan et al. [Certain] Two limits help to remember it. With \(\gamma = 1\) you get \(1 + \alpha\): at most 2 tokens per pass. With very long drafts the sum approaches \(1/(1 - \alpha)\): with \(\alpha = 0.8\), never more than 5 tokens per target pass, however long the draft. Long drafts have diminishing returns because a guess deep in the draft only counts if all the earlier guesses passed.

Worked example: \(\alpha = 0.8\), \(\gamma = 4\)

\(P(K = 0, 1, 2, 3) = 0.2,\ 0.16,\ 0.128,\ 0.1024\), and \(P(K = 4) = 0.8^4 = 0.4096\). They add up to 1.

Expected tokens per step: \((1 - 0.8^5)/(1 - 0.8) = (1 - 0.32768)/0.2 = 3.3616\). Check with the distribution: \(1(0.2) + 2(0.16) + 3(0.128) + 4(0.1024) + 5(0.4096) = 3.3616\).

The draft made 4 guesses per step, but on average only \(3.3616 - 1 = 2.36\) were accepted. The other 1.64 guesses per step were wasted work.

The speed-up, and the best draft length

Tokens per step is not speed. A step also costs \(\gamma\) draft calls. Let \(c\) be the cost coefficient: the time of one draft call divided by the time of one target call. If the target pass over \(\gamma + 1\) tokens costs the same as a 1-token pass (the memory-bound regime of formula 128.1), one step costs \(\gamma c + 1\) target-call times, and the wall-time speed-up over plain decoding is

\[ S(\alpha, \gamma, c) = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)} \tag{128.8} \]
Read the symbols
\(S\)
speed-up in tokens per second over plain decoding with the target alone
\(\alpha\)
acceptance rate, formula (128.5)
\(\gamma\)
number of draft tokens per step
\(c\)
draft call time divided by target call time; it depends on the hardware and software, not only on parameter counts

This is Theorem 3.8 of Leviathan et al. [Certain] The numerator grows with \(\gamma\) but saturates at \(1/(1-\alpha)\); the denominator grows linearly. So there is a best \(\gamma\). It is an integer and easy to find by trying \(\gamma = 1, 2, \dots\). When is there any gain? At \(\gamma = 1\), \(S = (1 + \alpha)/(1 + c)\), which is above 1 exactly when \(\alpha \gt c\):

\[ \alpha \gt c \ \Longrightarrow\ \max_\gamma S(\alpha, \gamma, c) \ \ge\ \frac{1 + \alpha}{1 + c} \gt 1 \tag{128.9} \]

This is Corollary 3.9 of Leviathan et al. [Certain] It is a low bar: a draft that costs 5% of the target must only be accepted more than 5% of the time. Leviathan et al. report that \(c\) was below 0.05 in most of their setups, and give \(c = 0.02, 0.04, 0.11\) for T5-small, T5-base and T5-large drafting for T5-XXL. [Certain]

Worked example: \(\alpha = 0.8\), \(c = 0.05\)

\(\gamma = 4\): \(S = 3.3616 / (1 + 4 \times 0.05) = 3.3616/1.2 = 2.80\).

Trying all \(\gamma\) from 1 to 12 gives \(1.71, 2.22, 2.57, 2.80, 2.95, 3.04, 3.08, 3.09, 3.08, 3.05, 3.00, 2.95\). The best is \(\gamma = 8\) with \(S = 3.09\), but the curve is flat: \(\gamma\) from 6 to 10 is within 2% of the best.

Now a weak, expensive draft: \(\alpha = 0.3\), \(c = 0.5\). The best is \(\gamma = 1\) with \(S = 1.3/1.5 = 0.87\): slower than no speculation, as (128.9) predicts, because \(\alpha \lt c\).

tokens per step (128.7)speed-up (128.8)\(P(\text{tokens per step} = k)\), formula (128.6)simulated frequency
Simulate steps with acceptance rate \(\alpha\): one coin flip \(r \lt \alpha\) per guess, stop at the first failure
Figure 4. Left: expected tokens per target pass and wall-time speed-up as functions of the draft length \(\gamma\). The open circle marks the best \(\gamma\); the dashed line is speed-up 1 (no gain). Right: the distribution of the number of tokens one step produces at the chosen \(\gamma\), with purple dots for the simulated steps. Below: the coin flips of the latest simulated step and formula (128.7) with the live \(\alpha\) and \(\gamma\).
How to read Figure 4

The orange curve (tokens) keeps rising but flattens at \(1/(1-\alpha)\). The blue curve (speed-up) rises, peaks, then falls, because each extra guess costs \(c\) but is worth less. Set \(c\) to 0 and the best \(\gamma\) runs to the right edge. Set \(\alpha\) below \(c\) and the blue curve stays under the dashed line for every \(\gamma\). On the right, the tall last bar is the chance that all guesses pass (\(\alpha^\gamma\)); it shrinks fast as \(\gamma\) grows. Press Play under the chart to simulate steps: each guess passes when its random number \(r\) is below \(\alpha\), the step stops at the first failure, and one more token always comes from the target. The purple dots are the measured frequencies of 1, 2, …, \(\gamma + 1\) tokens per step; they settle on the green bars of (128.6), and the running mean settles on (128.7).

The extra arithmetic

Speculation buys latency with extra compute. Rejected guesses are scored by the target and thrown away, and the draft runs too. Leviathan et al. (Theorem 3.11) give the expected factor of increase in total arithmetic operations:

\[ \text{ops factor} = \frac{(1 - \alpha)(\gamma \hat c + \gamma + 1)}{1 - \alpha^{\gamma+1}} \tag{128.10} \]

Here \(\hat c\) is the ratio of arithmetic operations per token of the draft to the target, close to the ratio of parameter counts. With \(\alpha = 0.8\), \(\gamma = 4\), \(\hat c = 0.05\), the factor is \(1.55\): 2.8 times faster, for 55% more FLOPs. At batch size 1 those FLOPs were idle anyway, so they are free. When the GPU is already busy, they are not. That is the subject of the next section.

When speculation does not pay

Formula (128.8) assumed two things: verifying \(\gamma + 1\) tokens costs the same as decoding 1, and the draft is good enough. Each can fail.

Big batches: the spare compute is gone

A serving engine runs many sequences together. With batch size \(B\), a plain decode pass processes \(B\) tokens; a verification pass processes \(B(\gamma + 1)\) tokens. Put both in formula (128.1). As long as \(B(\gamma + 1) \le 295\) on an H100, verification is still memory-bound and nearly free. Beyond that, the target becomes compute-bound and every rejected guess costs real time. The speed-up at batch \(B\) becomes

\[ S(B) = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha} \cdot \frac{t_p(B)}{\gamma\, t_q(B) + t_p\big(B(\gamma+1)\big)} \tag{128.11} \]
Read the symbols
\(t_p(n), t_q(n)\)
time of one target or draft pass over \(n\) tokens, from formula (128.1)
\(B\)
number of sequences decoded together

Take a 7B target and a 350M draft on an H100 (so \(c = 0.05\) at batch 1), with \(\alpha = 0.8\) and \(\gamma = 4\). At \(B = 1\) to \(59\), \(S = 2.80\). At \(B = 128\), the verification pass handles 640 tokens and takes 9.06 ms instead of 4.18 ms, and \(S = 1.42\). At \(B = 256\), \(S = 0.74\): speculation now makes decoding slower. The break-even point is \(B = 186\). For very large batches everything is compute-bound and \(S\) tends to \(3.3616/(4 \times 0.05 + 5) = 0.65\): you pay for 5.2 tokens of compute to get 3.36. Explore it in Figure 5.

Figure 5. Speed-up of speculative decoding versus batch size, from the roofline model (128.1) and (128.11): a 7B target and a 350M draft in BF16 on an H100 SXM (3.35 TB/s, 989 TFLOP/s), weights only, no KV cache. The batch axis is logarithmic.
How to read Figure 5

The curve is flat while the verification pass of \(B(\gamma+1)\) tokens stays under the ridge (295 tokens), then it falls and crosses the dashed line at the break-even batch. Longer drafts reach the ridge at a smaller batch, so the best \(\gamma\) shrinks as the batch grows. This model ignores the KV cache. With long contexts, reading the KV cache keeps decode memory-bound even at large batch sizes, which moves the break-even point to the right; MagicDec (Sadhukhan et al.) uses this to get speed-ups at batch sizes of 32 to 256 for long-context generation. [Certain]

So in practice, engines turn speculation down or off at high load, or choose \(\gamma\) from the current batch size. [Likely] The question to ask is not “does speculative decoding help?” but “does my GPU have idle compute at my batch size and context length?”. For a chatbot serving one user at a time, yes. For an offline batch job that already fills the GPU, usually no.

Weak or expensive drafts

A draft must be both close to the target (high \(\alpha\)) and cheap (low \(c\)). These pull in opposite directions: in Leviathan et al.’s table, T5-large had the highest \(\alpha\) (0.82) but the lowest speed-up (1.7×) because its \(c\) was 0.11. A unigram or bigram “draft” costs nothing but reaches only \(\alpha = 0.07\) to \(0.23\) on the T5-XXL tasks. [Certain] Other ways a draft becomes weak:

  • Distribution shift. A draft trained on general web text drafting for a code or medical fine-tune of the target. Distilling the draft on the target’s own outputs raises \(\alpha\) (session 119).
  • High temperature. Sampling at temperature 1 or above lowers \(\alpha\) compared with greedy decoding, as in the T5 table.
  • Different tokenizers. The rule compares probabilities of the same token, so the draft must share the target’s vocabulary.
  • Distributed serving. Chen et al. point out that a small model may not get faster when spread over the same many accelerators as the big model, because communication dominates. They trained a wide and shallow draft (8 layers) for this reason. [Certain]
Algorithm 2: choosing \(\gamma\) (and whether to speculate at all)
  1. input: measured \(\alpha\) on your traffic, measured \(t_p(n)\) and \(t_q(n)\) at your batch size \(B\)
  2. for \(\gamma = 1 \dots \gamma_{\max}\): compute \(S(B)\) with formula (128.11)
  3. \(\gamma^* \leftarrow \argmax_\gamma S(B)\)
  4. if \(S(B)\) at \(\gamma^*\) is below about 1.1: do not speculate at this load // the gain is not worth the extra memory and complexity
  5. re-measure \(\alpha\) per task and per temperature; recompute when the batch size changes

What the papers measured

Leviathan et al. used T5-XXL (11B) as the target on a single TPU-v4 at batch size 1, with T5X as the baseline. [Certain] The table below shows their reported \(\alpha\) and speed-ups, next to what formula (128.8) predicts from \(\alpha\), \(\gamma\) and their measured \(c\) (0.02, 0.04, 0.11 for small, base, large).

TaskDraftTemp\(\gamma\)\(\alpha\)Tokens per step (128.7)Predicted \(S\) (128.8)Measured
EnDeT5-small070.753.603.163.4×
EnDeT5-base070.804.163.252.8×
EnDeT5-large070.824.422.501.7×
EnDeT5-small170.622.572.262.6×
CNNDMT5-small050.652.642.403.1×
CNNDMT5-small150.532.081.892.3×

The simple model gets the ranking and the size of the effect right, but it is not exact. For T5-small it is pessimistic (measured 3.1× versus predicted 2.40× on CNNDM); for T5-large it is optimistic. Two reasons are likely: real acceptances are not independent (easy stretches of text accept long runs, which helps more than the average \(\alpha\) suggests), and the verification pass is not exactly as cheap as a 1-token pass. [Likely] Measure on your own system.

Chen et al. used Chinchilla 70B as the target on 16 TPU v4 chips, with a 4B draft (8 layers, width 6,144), batch size 1, and \(K = 4\) draft tokens (their name for \(\gamma\)). [Certain] The draft took 1.8 ms per token and the target 14.1 ms per token, so \(c \approx 0.13\). They report mean token times of 7.52 ms on XSum with nucleus sampling (speed-up 1.92×), 7.00 ms with greedy decoding (2.01×), and 5.73 ms on HumanEval (2.46×), with benchmark scores about the same (XSum ROUGE-2 0.112 versus 0.114; HumanEval 45.1% versus 47.0%). [Certain] (The ratios 14.1/7.52 and 14.1/7.00 give 1.88 and 2.01; the paper’s speed-up column comes from its own per-loop logs. [Likely]) Code reaches the largest speed-up because code has many predictable tokens.

In code

The code below implements Algorithm 1 with two tiny bigram “language models” over 6 tokens, so the exact distribution of the target is known. It checks three things: the speculative output has the target’s joint distribution over 3 tokens (its distance to the truth equals the noise of exact sampling with the same number of samples), the draft alone has a different distribution, and the measured tokens per target call match formula (128.7). It runs in about 2 seconds on a CPU.

import torch
torch.manual_seed(0)
V = 6                                    # tiny vocabulary
Zp = 2.0 * torch.randn(V, V)             # target: next-token logits given the last token (a bigram model)
Zq = Zp + 1.0 * torch.randn(V, V)        # draft: a noisy copy of the target

def p_target(x): return torch.softmax(Zp[x], -1)   # p(. | last token x)
def q_draft(x):  return torch.softmax(Zq[x], -1)   # q(. | last token x)

def speculative_step(last, gamma, g):
    """One draft-then-verify step. Returns the new tokens (1 to gamma+1 of them)."""
    xs, qs = [], []
    x = last
    for _ in range(gamma):                                   # 1) draft gamma tokens with q
        q = q_draft(x); x = int(torch.multinomial(q, 1, generator=g)); xs.append(x); qs.append(q)
    ctx = [last] + xs
    ps = [p_target(c) for c in ctx]                          # 2) ONE target pass: gamma+1 distributions
    out = []
    for i in range(gamma):                                   # 3) accept or reject, left to right
        p, q, x = ps[i], qs[i], xs[i]
        if torch.rand((), generator=g) < torch.clamp(p[x] / q[x], max=1.0):
            out.append(x)                                    # accepted, formula (128.2)
        else:
            r = torch.clamp(p - q, min=0.0)                  # residual, formula (128.3)
            out.append(int(torch.multinomial(r / r.sum(), 1, generator=g)))
            return out
    out.append(int(torch.multinomial(ps[gamma], 1, generator=g)))   # all accepted: bonus token
    return out

def speculative_generate(last, n, gamma, g):
    seq, calls = [], 0
    while len(seq) < n:
        seq += speculative_step(seq[-1] if seq else last, gamma, g); calls += 1
    return seq[:n], calls

# exact joint distribution of 3 target tokens after start token 0
P3 = torch.zeros(V, V, V)
for a in range(V):
    for b in range(V):
        P3[a, b] = p_target(0)[a] * p_target(a)[b] * p_target(b)
g = torch.Generator().manual_seed(1)
N, gamma = 20000, 4
emp, calls = torch.zeros(V, V, V), 0
for _ in range(N):
    s, c = speculative_generate(0, 3, gamma, g); emp[tuple(s)] += 1; calls += c
emp /= N
Q3 = torch.zeros(V, V, V)                                    # what the draft alone would give
for a in range(V):
    for b in range(V):
        Q3[a, b] = q_draft(0)[a] * q_draft(a)[b] * q_draft(b)
print("TV(speculative, target):", round(0.5 * (emp - P3).abs().sum().item(), 4))
ref = torch.multinomial(P3.flatten(), N, replacement=True, generator=g)   # same N, sampled exactly from p
ref = torch.bincount(ref, minlength=V ** 3).float() / N
print("TV(exact sampling, target):", round(0.5 * (ref - P3.flatten()).abs().sum().item(), 4))
print("TV(draft alone, target):", round(0.5 * (Q3 - P3).abs().sum().item(), 4))
alpha_ctx = torch.minimum(torch.softmax(Zp, -1), torch.softmax(Zq, -1)).sum(-1)   # formula (128.5), per context
print("acceptance rate per context:", [round(a, 3) for a in alpha_ctx.tolist()])

# tokens per target call on a long sequence
g = torch.Generator().manual_seed(2)
seq, calls = speculative_generate(0, 20000, gamma, g)
print("tokens per target call:", round(len(seq) / calls, 3))
a = alpha_ctx.mean().item()
print("formula (128.7) with mean alpha", round(a, 3), ":", round((1 - a ** (gamma + 1)) / (1 - a), 3))

# Output (torch 2.8, CPU):
# TV(speculative, target): 0.0181
# TV(exact sampling, target): 0.0182
# TV(draft alone, target): 0.2332
# acceptance rate per context: [0.845, 0.442, 0.903, 0.776, 0.93, 0.888]
# tokens per target call: 3.386
# formula (128.7) with mean alpha 0.798 : 3.346

The speculative output is as close to the target as exact sampling is (0.0181 versus 0.0182, both pure sampling noise for 20,000 samples), while the draft alone is far away (0.2332). The measured 3.386 tokens per call is close to the 3.346 of formula (128.7); the small gap comes from \(\alpha\) varying across contexts (from 0.44 to 0.93 here). In a real engine, steps 1 and 2 are batched GPU passes with KV caches, and the accept loop runs on the GPU as a vectorized comparison.

Where this is used

SettingHow speculation is usedWhy it pays (or not)
Interactive chat and coding assistantsSmall draft of the same family, or built-in drafters (session 129)Small batches per GPU at low load leave compute idle; code and repeated text give high \(\alpha\) (HumanEval: 2.46× in Chen et al.)
Serving engines (vLLM, SGLang, TensorRT-LLM, llama.cpp)Draft model or n-gram drafting, with a configurable number of speculative tokens [Likely]Gain shrinks as the batch grows (Figure 5); tune \(\gamma\) per load
Hugging Face transformers“Assisted generation”: pass a smaller assistant model to generate [Likely]Easy way to measure \(\alpha\) and the speed-up for your model pair
On-device generation (phones, laptops)Tiny draft next to a quantized targetBatch size 1 and very low memory bandwidth: the most memory-bound case of all [Likely]
Long-context servingDraft with a sparse or compressed KV cache (MagicDec)KV reads keep decode memory-bound even at large batches
VLMs: video captioning, video QA, key-moment descriptionsText draft model that shares the VLM’s tokenizer; the visual tokens sit in the shared prefix and are prefilled once [Speculation]Long, formulaic outputs (timestamps, dense captions) are easy to guess; but the draft cannot see the video unless it gets the visual tokens too, which lowers \(\alpha\) on visual details [Speculation]

Common mistakes

  • Using different sampling settings for \(p\) and \(q\). The rule needs the probabilities the draft actually sampled from. If the draft samples at temperature 0.7 with top-p, the \(q\) in \(p/q\) must be that processed distribution, and \(p\) must use the target’s intended settings.
  • Accepting by “token equals the target’s argmax” while sampling. That is correct only for greedy decoding. With sampling it gives a different, lower-entropy distribution.
  • Forgetting the residual. On rejection, sampling the replacement from \(p\) instead of \(\text{norm}(\max(0, p - q))\) over-weights the tokens the draft already proposes, and the output is no longer exact.
  • Forgetting the bonus token. When all \(\gamma\) guesses pass, \(p_{\gamma+1}\) is already computed. Not using it lowers formula (128.7) by \(\alpha^\gamma\): for \(\alpha = 0.8\), \(\gamma = 4\), from 3.36 to 2.95 tokens per step.
  • Not rolling back the KV cache. The draft and the target both wrote cache entries for the rejected guesses. They must be truncated to the accepted length, or the next step attends to tokens that were never emitted.
  • Benchmarking at batch size 1 and deploying at batch 128. The gain at batch 1 says nothing about the gain under load (Figure 5).
  • Using a draft with a different tokenizer without a token-level translation scheme. The ratio \(p(\tilde x)/q(\tilde x)\) is meaningless if the two models split text differently.
  • Measuring \(\alpha\) on one task and assuming it everywhere. \(\alpha\) depends on the domain and the temperature; in the T5 study it ranged from 0.53 to 0.82 for the same target.

Interview questions

Why can a large model check 5 guessed tokens for about the cost of generating 1?basic
In the decode phase, a forward pass is memory-bound: its time is the time to read all the weights (for a 7B BF16 model on an H100, about \(14\ \text{GB} / 3.35\ \text{TB/s} = 4.2\ \text{ms}\)). Processing 5 tokens instead of 1 reuses the same weight read and only adds FLOPs, and the GPU has plenty of unused compute until about \(P/BW \approx 295\) tokens per pass. The 5 positions can be scored in parallel because the guessed tokens are known, just as in training with teacher forcing.
Write the acceptance rule and the residual distribution.core
Accept the draft token \(\tilde x \sim q\) with probability \(\min(1, p(\tilde x)/q(\tilde x))\). If rejected, sample from \(p'(x) = \max(0, p(x) - q(x)) / \sum_{x'} \max(0, p(x') - q(x'))\) and stop the step. If all \(\gamma\) guesses are accepted, sample a bonus token from the target’s distribution after the last guess. Later guesses after a rejection are discarded.
Prove that the output has exactly the target distribution.core
\(P(X = x) = q(x)\min(1, p(x)/q(x)) + P(\text{reject})\,p'(x)\). The first term is \(\min(p(x), q(x))\). The rejection probability is \(1 - \sum \min(p, q) = \sum \max(0, p - q)\), which is exactly the normalizer of \(p'\), so the second term is \(\max(0, p(x) - q(x))\). The sum is \(p(x)\). For sequences, apply this at each position conditioned on the accepted prefix (the bonus token is drawn from \(p\) directly) and use the chain rule.
What is the acceptance rate \(\alpha\), and how is it related to a distance between distributions?core
\(\alpha = \E_{x\sim q}[\min(1, p/q)] = \sum_x \min(p(x), q(x)) = 1 - \text{TV}(p, q)\), where TV is half the L1 distance. So the draft only needs to be close to the target in total variation on the contexts that occur. In the worked example \(p = (0.45, 0.25, 0.2, 0.1)\), \(q = (0.3, 0.4, 0.1, 0.2)\): TV = 0.25 and \(\alpha = 0.75\).
Derive the expected number of tokens per target pass.core
Assume each guess passes independently with probability \(\alpha\). The step emits \(K + 1\) tokens, where \(K\) is the number of accepted guesses, and \(K + 1 \ge j\) iff the first \(j-1\) guesses pass, which has probability \(\alpha^{j-1}\). So \(\E[K+1] = \sum_{j=1}^{\gamma+1} \alpha^{j-1} = (1 - \alpha^{\gamma+1})/(1 - \alpha)\). For \(\alpha = 0.8\), \(\gamma = 4\): 3.36. The limit for long drafts is \(1/(1-\alpha) = 5\).
Give the speed-up formula. When is speculation guaranteed to help?core
With \(c\) = draft call time / target call time, and verification as cheap as one decode step, \(S = (1 - \alpha^{\gamma+1}) / ((1 - \alpha)(\gamma c + 1))\). At \(\gamma = 1\), \(S = (1 + \alpha)/(1 + c)\), which exceeds 1 iff \(\alpha \gt c\). So if the acceptance rate is larger than the relative draft cost, some \(\gamma\) gives a gain. The best \(\gamma\) is found by trying integers; for \(\alpha = 0.8\), \(c = 0.05\) it is 8 (\(S = 3.09\)).
What happens with greedy decoding?basic
\(p\) and \(q\) become one-hot. A guess is accepted iff it equals the target’s argmax; on rejection the residual is the target’s argmax. The output is exactly the target’s greedy output (up to floating-point ties), and \(\alpha\) is the top-1 agreement rate of the two models. In Leviathan et al., \(\alpha\) was higher at temperature 0 than at 1 for every draft.
Why does speculative decoding stop helping, or even hurt, at large batch sizes?advanced
At batch \(B\), verification processes \(B(\gamma+1)\) tokens. Once this passes the ridge point (about 295 tokens per pass on an H100 in BF16), the target is compute-bound, so the extra positions cost real time, and rejected guesses are pure waste. With a 7B target, 350M draft, \(\alpha = 0.8\), \(\gamma = 4\), a weights-only roofline gives 2.80× up to batch 59, 1.42× at 128, and a slowdown (0.74×) at 256; break-even near 186. Long contexts move this point to the right because KV-cache reads keep decode memory-bound.
How much extra compute does speculation use?advanced
Leviathan et al. give the expected factor \((1-\alpha)(\gamma\hat c + \gamma + 1)/(1 - \alpha^{\gamma+1})\), where \(\hat c\) is the draft-to-target ratio of FLOPs per token. For \(\alpha = 0.8\), \(\gamma = 4\), \(\hat c = 0.05\) it is 1.55. The extra work is the draft plus the target scoring guesses that are later rejected. It is free when compute is idle (small batch) and costly when it is not (big batch, or when energy per token matters).
With a fixed random seed, does speculative decoding give the same text as plain decoding?advanced
No. The guarantee is equality in distribution, not sample by sample: the two procedures consume random numbers differently. Even for greedy decoding, the target’s logits from a multi-token pass may differ in the last bits from single-token passes (different kernel shapes and reduction orders), which can flip near-ties. For tests, compare distributions (or greedy outputs with tolerance), not exact strings.
Your draft has \(\alpha = 0.9\) but the system is slower than plain decoding. What could be wrong?advanced
Likely causes: \(c\) is large (the draft is not much faster in your setup, for example because it is sharded over many GPUs and communication dominates, as Chen et al. warn); the batch is large so verification is compute-bound; the draft runs on the critical path with CPU overhead or kernel launches per token (CUDA graphs help); KV-cache rollback or re-allocation is expensive; or \(\gamma\) is too large for this \(c\). Measure \(t_q\), \(t_p(B)\) and \(t_p(B(\gamma+1))\) and plug them into formula (128.11).

Exercises

  1. Target \(p = (0.5, 0.3, 0.2)\), draft \(q = (0.2, 0.3, 0.5)\). Compute the acceptance probability of each token, \(\alpha\), and the residual distribution.
  2. With \(\alpha = 0.6\), \(\gamma = 3\), \(c = 0.1\): compute the expected tokens per step and the speed-up. Is \(\gamma = 3\) the best choice?
  3. Greedy decoding: the draft agrees with the target’s top token 70% of the time. How many tokens per target pass do you expect with \(\gamma = 3\)?
  4. Show that \(S \gt 1\) at \(\gamma = 1\) if and only if \(\alpha \gt c\). What is \(S\) for \(\alpha = c = 0.3\)?
  5. Chen et al. measured \(c \approx 1.8/14.1\) and \(K = 4\). Using formula (128.8), which \(\alpha\) would give their 1.92× on XSum? What is the largest speed-up possible with this \(c\) and \(K\)?
  6. In Figure 5’s model (H100, 7B target), up to which batch size is the verification pass still memory-bound for \(\gamma = 4\)? For \(\gamma = 9\)?
Answerscheck
1. Acceptance probabilities \(\min(1, p/q) = (1, 1, 0.4)\). \(\alpha = 0.2 + 0.3 + 0.2 = 0.7\). Missing mass \(\max(0, p - q) = (0.3, 0, 0)\), so the residual is \((1, 0, 0)\): after a rejection, always output the first token. Check: \(0.2 + 0.3 \times 1 = 0.5\). 2. Tokens: \((1 - 0.6^4)/0.4 = 2.176\). Speed-up \(2.176/1.3 = 1.67\). Trying \(\gamma = 1 \dots 7\) gives \(1.45, 1.63, 1.67, 1.65, 1.59, 1.52, 1.45\), so yes, \(\gamma = 3\) is best. 3. \((1 - 0.7^4)/0.3 = 2.53\). 4. At \(\gamma = 1\), \(S = (1 + \alpha)/(1 + c)\), which is above 1 iff \(1 + \alpha \gt 1 + c\). For \(\alpha = c = 0.3\), \(S = 1\) exactly. 5. \(c = 0.128\); solving \((1 - \alpha^5)/((1 - \alpha)(1 + 4 \times 0.128)) = 1.92\) gives \(\alpha \approx 0.72\). The maximum, at \(\alpha \to 1\), is \(5/1.51 = 3.31\). (Their verification pass is not free, so the real \(\alpha\) is probably higher. [Likely]) 6. Memory-bound while \(B(\gamma + 1) \le 295\): \(B \le 59\) for \(\gamma = 4\), \(B \le 29\) for \(\gamma = 9\).

Key takeaways

  • Decode passes are memory-bound (128.1), so scoring \(\gamma + 1\) tokens costs about as much as scoring 1. A cheap draft proposes \(\gamma\) tokens; the target checks them all in one pass.
  • Accept each guess with probability \(\min(1, p/q)\) (128.2); on rejection, sample from the residual \(\text{norm}(\max(0, p - q))\) (128.3). The output then has exactly the target’s distribution (128.4).
  • The acceptance rate is \(\alpha = \sum \min(p, q) = 1 - \text{TV}(p, q)\) (128.5). Greedy decoding is the special case “accept if it matches the argmax”.
  • Each step gives \((1 - \alpha^{\gamma+1})/(1-\alpha)\) tokens (128.7), at most \(1/(1-\alpha)\). The speed-up is that divided by \(1 + \gamma c\) (128.8); some \(\gamma\) helps whenever \(\alpha \gt c\) (128.9).
  • Speculation spends extra compute (128.10). At large batch sizes the compute is no longer idle, and the speed-up falls below 1 (128.11); weak or expensive drafts fail the \(\alpha \gt c\) test.
  • Reported results: T5-XXL 2.3× to 3.4× with a T5-small draft (Leviathan et al.), Chinchilla 70B 1.92× to 2.46× (Chen et al.), both at batch size 1.

Key formulas

\[ P(\text{accept}) = \min\!\left(1, \frac{p(\tilde x)}{q(\tilde x)}\right), \quad p' = \text{norm}\big(\max(0, p - q)\big), \quad \alpha = \sum_x \min(p, q) \]
\[ \E[\text{tokens}] = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}, \qquad S = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}, \qquad \text{gain possible iff } \alpha \gt c \]

References for this session

  1. Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
  2. Chen et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling
  3. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  4. Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
  5. Chen et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling
  6. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  7. Stern, Shazeer, Uszkoreit (2018). Blockwise Parallel Decoding for Deep Autoregressive Models
  8. Sadhukhan, Chen, Chen et al. (2024). MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
  9. Elhoushi et al. (2024). LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
  10. Cai et al. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
  11. Li et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty