LLM & VLM courseBlog
132Track 5: LLM inference and serving · session 55 of 65

profile, fuse and speculate

In this lab you take one real model, Qwen2.5-1.5B-Instruct, and make its decoding faster in three steps. You profile one decode step and find that hundreds of tiny kernels and the CPU that launches them cost a large part of the time. You fuse the residual add with RMSNorm into one kernel and measure the gain. Then you run speculative decoding with a small draft model and with an EAGLE-style head that you train yourself, and you measure the acceptance rate and the tokens per second at batch 1, 4 and 16.

Time: 4 to 5 hours of reading and hands-on work, plus about 2 hours of laptop GPU time (head training and measurements)Level: Intermediate · hands-on labBefore this: Sessions 87 (KV-cached generation lab), 123 (runtime profiling), 127 (graph-level optimization), 128 and 129 (speculative decoding); PyTorch and a little Triton

What you will learn

  • How to profile one decode step: the operator table, the count of small kernels, issue time against GPU time, and the Nsight Systems commands for an NVIDIA GPU
  • How to estimate the bytes of an unfused residual add + RMSNorm (42 bytes per element in eager PyTorch) against a fused kernel (8 bytes), and why fusion helps even more at batch 1, where launches dominate
  • How to write the fused kernel in Triton (and in Metal, which is what we ran), test it bit for bit against the eager code, and measure it alone and inside the model
  • How to run greedy speculative decoding in a batch where each row accepts a different number of tokens, using a validity mask over the KV cache
  • How to train a small EAGLE-style draft head on the target model's own features, and how it compares with a separate small draft model
  • How to measure acceptance rate, accepted length and tokens per second at batch 1, 4 and 16, and explain with a cost model why the gain shrinks as the batch grows
  • Why the speculative output must equal plain greedy decoding in FP32, and why it can differ in BF16

Why this matters

The last ten sessions gave you many ways to make decoding faster: read a profile, fuse small kernels, launch a whole step at once, and guess several tokens with a cheap model. Each idea comes with a promise, such as “fusion saves memory traffic” or “speculation gives 2 to 3 times”. In this lab you test the promises on one real model and one real machine. You profile one decode step, write one fused kernel and measure it, and run speculative decoding with two kinds of drafts at three batch sizes. At the end you can say, with your own numbers, which trick helped, by how much, and why.

The lab connects four sessions. Session 123 taught you to read an operator table and an Nsight Systems trace. Session 127 explained fusion and CUDA graphs. Session 128 derived the acceptance rate and the speed-up formula, and Session 129 showed the kinds of drafts, including EAGLE. Session 131 showed how a production engine packs all of this together. Here you do each step by hand, so that nothing is a black box.

The work has four parts, one per key point of the session:

  1. Profile one decode step and find where the time goes.
  2. Fuse the residual add and RMSNorm into one kernel, check it, and measure it alone and inside the model.
  3. Run speculative decoding with a small draft model and with an EAGLE-style head that you train.
  4. Measure the acceptance rate and the tokens per second at batch 1, 4 and 16.

Our machine and what we could not run. All numbers come from one laptop: an Apple M5 Pro (20-core GPU, 48 GB of unified memory), PyTorch 2.8 with the MPS back-end and Hugging Face transformers 4.57, in BF16. Other jobs shared the GPU while we measured, so our times are noisy; we report the best of several repeats unless we say otherwise. Two tools of the key points need an NVIDIA GPU: Nsight Systems and Triton. For both we give the exact CUDA commands and code, marked [not run by us]. On our machine we used the closest equivalents: the PyTorch profiler, and the same fused kernel written in Metal (Apple’s GPU language) through torch.mps.compile_shader. The method and the byte counts are the same on any GPU; the absolute numbers are not.

The lab models

The target (the model whose output we want) is Qwen2.5-1.5B-Instruct. The draft model is Qwen2.5-0.5B-Instruct: same tokenizer, same chat format, about a third of the bytes. Their shapes, from each model’s config.json [Certain]:

Qwen2.5-1.5B-Instruct (target)Qwen2.5-0.5B-Instruct (draft)
layers \(L\), width \(d\)28, 1,53624, 896
query heads, KV heads, head size12, 2, 12814, 2, 64
FFN width (SwiGLU)8,9604,864
vocabulary, tied output layer151,936, yes151,936, yes
parameters (PyTorch count)1,543,714,304 (3.09 GB in BF16)494,032,768 (0.99 GB)
KV cache per token28,672 bytes12,288 bytes

The third draft is a small network that you train in Part 3: an EAGLE-style head, one transformer layer of the target’s shape plus a projection, 51.5 million parameters. It reuses the target’s embedding and output layer.

Worked example: the best possible decode step

A decode step makes one token per sequence. It must read every weight once: 3.09 GB. On our machine a large copy reaches about 233 GB/s (measured in session 87, same laptop). So one step at batch 1 cannot take less than \(3.087 / 233 = 13.2\) ms, and the fastest possible decoding is about 75 tokens per second. A context of 512 tokens adds \(512 \times 28{,}672 = 14.7\) MB of cache, which is less than 0.5% of the weights. So at batch 1 the bytes are the weights, and the bound is 13.2 ms. Keep this number in mind: every measurement below is compared with it.

\[ t_{\text{step}} \;\ge\; \frac{s\,N + b\,c\,\text{kv}_{\text{tok}}}{\text{BW}} \tag{132.1} \]
Read the symbols
\(N, s\)
number of weights (1.54 billion) and bytes per number (2 for BF16)
\(b, c\)
sequences in the batch and tokens already in each one’s cache
\(\text{kv}_{\text{tok}}\)
bytes of keys and values per token: \(2 \cdot L \cdot n_{kv} \cdot d_h \cdot s\), which is \(2 \cdot 28 \cdot 2 \cdot 128 \cdot 2\) \(= 28{,}672\)
\(\text{BW}\)
memory bandwidth you can really reach (measured, not the spec sheet)

Part 1: profile one decode step

A profile is a record of what ran and how long it took. For one decode step you want three things: how many GPU operations (kernels) the step launches, how long the CPU needs to issue them, and how long the GPU needs to run them. If the CPU needs longer than the GPU, the GPU waits between kernels and the step is launch-bound (session 123).

Algorithm 1: profile one decode step
  1. input: model, a prompt of \(c\) tokens, batch \(b\)
  2. prefill the prompt once; keep the cache; remember its length \(c\)
  3. define step(): crop the cache back to \(c\); run the model on one new token; take the argmax // the same work every time
  4. run step() 3 times // warm-up: allocation, kernel choice, shader compilation
  5. repeat 15 times: \(t_0\); step(); \(t_1\) // issue time: the CPU is done; copy the token to the CPU; \(t_2\) // step time: the GPU is done
  6. report min and median of \(t_2 - t_0\), and the median of \(t_1 - t_0\)
  7. run step() once under the profiler; count the operations by name // what is launched, and how often

The trick in line 3 matters: a decode step changes the cache, so a timing loop that calls the model again and again would measure longer and longer contexts. Cropping the cache back gives the same step every time.

What the profile shows

We profiled one eager step at batch 1 (our own forward pass with the eager norms, which runs the same operations as the Hugging Face model; see Part 2) with a 512-token context. The PyTorch profiler lists 1,324 operations that do real work (we counted the lowest-level aten:: calls and left out those that only make a view of a tensor or allocate memory) [Certain, our run]. The largest groups:

OperationsCount per stepWhere they come from
matrix multiplies (aten::linear)1977 per layer (q, k, v, o, gate, up, down) × 28, plus the output layer
multiplies and adds (aten::mul, aten::add)254 and 112RMSNorm, the residual stream, RoPE, the SwiGLU product
type casts (aten::copy_)234mostly the casts of RMSNorm, which works in FP32 (BF16 → FP32 and back, 57 norms per step); after fusion 66 remain
concatenations (aten::cat)113RoPE’s “rotate half” and the cache append (the cache grows by concatenation)
pow, mean, rsqrt57 eachone of each per RMSNorm: 2 per layer plus the final norm, \(2 \times 28 + 1 = 57\)
attention (scaled_dot_product_attention)28one per layer

So only about 15% of the operations are the matrix multiplies that read the weights. The rest are small element-wise operations on a vector of 1,536 numbers. Each of them moves only a few kilobytes, but each one costs a launch: the CPU must run Python, dispatch the operator and send a command to the GPU.

Worked example: our measured step, against the bound

Batch 1, context 512, eager: the step took 29.9 ms (best of 15; median 30.2 ms). The CPU needed 12.0 ms just to issue it. The bound (132.1) is 13.2 ms. So we are at 2.3× the bound. The issue time alone is 0.9× the bound, about 9 µs of CPU time for each of the 1,324 operations: even with an infinitely fast GPU, this step could not take less than 12.0 ms. [Certain, our run]

At batch 16 the step took 76.5 ms, for 16 tokens. The bound grows only a little (16 caches of 512 tokens add 0.23 GB), so batching is still almost free; but the gap to the bound tells you that much of the step is overhead, not memory traffic.

The profiler has a limit you must know. On MPS (and on CUDA with only CPU activities on) it measures CPU time per operator. When the CPU must wait for the GPU, for example in the copy of the token to the CPU, that wait is charged to whatever operator is waiting. In our table, aten::copy_ had the largest “self CPU time” for exactly this reason, not because type casts are slow. A CPU-side table tells you how many operations there are; only a GPU timeline tells you when the GPU was busy. That is what Nsight Systems is for.

On an NVIDIA GPU: Nsight Systems

On a CUDA machine, wrap the step in an NVTX range (a named time span that shows up in the trace) and record it with nsys. [not run by us]

# profile_step.py: the same step() as above, marked for Nsight Systems (CUDA only)
import torch
for i in range(20):
    if i == 10: torch.cuda.cudart().cudaProfilerStart()      # skip the warm-up steps
    torch.cuda.nvtx.range_push(f"decode step {i}")
    tok = step(); tok.cpu()
    torch.cuda.nvtx.range_pop()
torch.cuda.cudart().cudaProfilerStop()

# shell: record CUDA API calls, kernels and NVTX ranges, only between Start and Stop
#   nsys profile -t cuda,nvtx,osrt --capture-range=cudaProfilerApi -o step python profile_step.py
#   nsys stats --report cuda_gpu_kern_sum step.nsys-rep     # time per kernel name
#   nsys stats --report cuda_api_sum step.nsys-rep          # time in cudaLaunchKernel etc.

In the trace, read the GPU row of one step: the sum of kernel times against the length of the NVTX range. Their ratio is the GPU utilization of the step. If the CUDA API row is solid and the GPU row has gaps, the step is launch-bound, as ours is.

CodeBatch
CPU issues a kernelmatmulattentionnorm and residualother small kernels
Figure 1. A timeline model (not a trace) of one decode step of Qwen2.5-1.5B with a 512-token context. Every kernel is listed with its bytes; its GPU time is the larger of bytes / 233 GB/s and a fixed shortest time. The CPU issues one kernel at a time; a kernel starts when it has been issued and the previous one is done. The top rows zoom into the first 1.5 layers; the bottom bar is the whole step. The readout adds our measured step times.
How to read Figure 1

Start with “eager” at batch 1 and an issue cost of 12 µs: the step has about 1,130 kernels, the CPU needs about 14 ms to issue them, the GPU is busy for about 17 ms, and the GPU row has gaps wherever several tiny kernels follow each other. Raise the issue cost to 25 µs: the step becomes launch-bound and grows to about 30 ms. Now press “fused add + RMSNorm”: about 450 kernels disappear, and the step gets shorter. Press “fused + one graph launch”: the CPU issues the whole step at once (this is what a CUDA graph does), the gaps close, and the step is set by the GPU alone. Switch to batch 16: the matmuls take about the same time (the weights are shared by all rows) and only attention grows, because it reads 16 caches, so the overhead stays a large part of the step.

The model of Figure 1 is simple: real issue costs differ by operator, and real GPU kernels overlap a little. But it explains the two numbers we measured: a step far above the bandwidth bound, and an issue time of the same size as the step. It also tells you what to do. Make fewer kernels (fusion, Part 2), or issue them all at once (a CUDA graph or torch.compile(mode="reduce-overhead"), from session 127). The second is not available on MPS, so in this lab we do the first.

Part 2: fuse the residual add and RMSNorm

Every layer of a pre-norm transformer does this twice: add the output of a sub-layer to the residual stream, then normalize the result for the next sub-layer (session 22):

\[ \mathbf{h} = \mathbf{x} + \mathbf{r}, \qquad \mathbf{y} = \mathbf{w} \odot \frac{\mathbf{h}}{\sqrt{\tfrac{1}{d}\sum_{j=1}^{d} h_j^2 + \epsilon}} \tag{132.2} \]
Read the symbols
\(\mathbf{x}\)
output of the attention or FFN sub-layer, one row of \(d = 1{,}536\) numbers per token
\(\mathbf{r}, \mathbf{h}\)
the residual stream before and after the add; \(\mathbf{h}\) is kept for the next residual add
\(\mathbf{w}, \epsilon\)
the learned scale of RMSNorm and a small constant (\(10^{-6}\) in Qwen2.5)
\(\mathbf{y}\)
the normalized input to the next sub-layer

In eager PyTorch this is nine kernels, because each line of the Hugging Face code is a separate operation: the add, a cast to FP32, pow, mean, the add of \(\epsilon\), rsqrt, a multiply, a cast back to BF16, and the multiply by \(\mathbf{w}\). Each kernel reads its inputs from memory and writes its output back. A fused kernel reads \(\mathbf{x}\) and \(\mathbf{r}\) once and writes \(\mathbf{h}\) and \(\mathbf{y}\) once.

\[ Q_{\text{eager}} = \underbrace{(6 + 6 + 8 + 4 + 8 + 6 + 4)}_{42}\; n\,d \ \text{bytes}, \qquad Q_{\text{fused}} = (2 + 2 + 2 + 2)\; n\,d = 8\,n\,d \ \text{bytes} \tag{132.3} \]
Read the symbols
\(n\)
tokens in the pass (the batch at decode, the prompt length at prefill)
42
bytes per number for the eager kernels: add (read 2+2, write 2), cast to FP32 (read 2, write 4), pow (4+4), mean (read 4), multiply by the inverse root (4+4), cast to BF16 (4+2), multiply by \(\mathbf{w}\) (2+2); the tiny per-row kernels for \(\epsilon\) and the root are left out
8
bytes per number fused: read \(\mathbf{x}\) and \(\mathbf{r}\), write \(\mathbf{h}\) and \(\mathbf{y}\), all BF16

So fusion should cut the bytes by \(42/8 = 5.25\) times. At decode with batch 1, the bytes are tiny (\(8 \times 1{,}536 = 12\) kB), so the time is the fixed cost of the kernels, and fusion turns nine fixed costs into one.

Algorithm 2: fused residual add + RMSNorm (one program per row)
  1. input: \(\mathbf{x}, \mathbf{r}\) of shape \((n, d)\) in BF16; \(\mathbf{w}\) of size \(d\); \(\epsilon\)
  2. for each row \(i\) in parallel: // one GPU program (Triton) or threadgroup (Metal) per token
  3.   load \(\mathbf{x}_i, \mathbf{r}_i\); \(\;\mathbf{h}_i \leftarrow \text{bf16}(\mathbf{x}_i + \mathbf{r}_i)\) // round like eager does
  4.   store \(\mathbf{h}_i\) into \(\mathbf{r}_i\) // the new residual, in place
  5.   \(\sigma \leftarrow \sum_j h_{ij}^2\) in FP32, reduced over the row; \(\;\rho \leftarrow 1/\sqrt{\sigma/d + \epsilon}\)
  6.   \(\mathbf{y}_i \leftarrow \text{bf16}\big(\mathbf{w} \odot \text{bf16}(\mathbf{h}_i \rho)\big)\) // the same two roundings as eager
  7.   store \(\mathbf{y}_i\)

The two roundings in line 6 are there on purpose. Hugging Face casts the normalized vector to BF16 before it multiplies by \(\mathbf{w}\). If the kernel skipped that rounding, its output would be slightly more accurate, but it would not match the eager model bit for bit, and you would lose the simplest test you have.

The Triton kernel

This is the kernel in Triton, for an NVIDIA GPU (the language is in session 86). One program handles one row; BLOCK is the row size rounded up to a power of two, so the whole row of 1,536 numbers sits in registers. [not run by us: it needs a CUDA GPU]

# fused_add_rmsnorm.py (Triton, CUDA)
import torch, triton, triton.language as tl

@triton.jit
def add_rmsnorm_kernel(X, R, W, Y, stride, D, eps, BLOCK: tl.constexpr):
    row = tl.program_id(0)
    cols = tl.arange(0, BLOCK)
    m = cols < D
    x = tl.load(X + row * stride + cols, mask=m, other=0.0).to(tl.float32)
    r = tl.load(R + row * stride + cols, mask=m, other=0.0).to(tl.float32)
    h = (x + r).to(tl.bfloat16)                        # residual add, rounded like eager
    tl.store(R + row * stride + cols, h, mask=m)       # new residual stream, in place
    hf = h.to(tl.float32)
    inv = tl.rsqrt(tl.sum(hf * hf, axis=0) / D + eps)  # one reduction over the row
    w = tl.load(W + cols, mask=m, other=0.0).to(tl.float32)
    y = (hf * inv).to(tl.bfloat16).to(tl.float32) * w
    tl.store(Y + row * stride + cols, y.to(tl.bfloat16), mask=m)

def add_rmsnorm(x, r, w, eps=1e-6):
    """h = x + r (written into r), y = RMSNorm(h) * w. x, r: (..., D) contiguous BF16."""
    D = x.shape[-1]; x2, r2 = x.view(-1, D), r.view(-1, D)
    y = torch.empty_like(x2)
    BLOCK = triton.next_power_of_2(D)
    add_rmsnorm_kernel[(x2.shape[0],)](x2, r2, w, y, x2.stride(0), D, eps,
                                       BLOCK=BLOCK, num_warps=4 if BLOCK <= 2048 else 8)
    return r, y.view_as(x)

On our machine we ran the same algorithm as a Metal kernel. The only differences are the language and the reduction: Metal adds up the row with simd_sum over groups of 32 threads and a small shared array, where Triton’s tl.sum does it for you. The full Metal source is in “The lab code” below.

Test, then measure

Test first. For 1 to 16,384 rows of random BF16 numbers, the fused kernel’s new residual \(\mathbf{h}\) was equal bit for bit to eager PyTorch. The output \(\mathbf{y}\) was equal bit for bit up to 256 rows; at 16,384 rows, 84 of 25.2 million numbers differed by one BF16 step (the two codes compute the reciprocal square root in slightly different ways). [Certain, our run] Then measure each version alone, with the timer of session 87 (warm-up, 50 calls, one synchronize, best of 5):

eager kernels (alternating shades)fused kernel
Figure 2. Residual add + RMSNorm with \(d = 1{,}536\) in BF16. Top: the bytes of each eager kernel and of the fused kernel, from (132.3). Bottom: our measured time per call (Apple M5 Pro, MPS), for the five sizes we measured. The readout also gives torch.compile, which fuses the same code automatically.
How to read Figure 2

At \(n = 1\) (decode, batch 1) the bars of bytes are tiny, but eager takes 122 µs and fused 17 µs: the cost is nine kernel launches against one. Move to 256 rows: still mostly fixed cost. At 4,096 and 16,384 rows (a long prefill) the bytes matter: the fused kernel moves 201 MB in 843 µs, about 239 GB/s, which is our machine’s copy bandwidth. Eager is 8.0 times slower, more than the 5.25 times of the byte count, because its small reductions and casts do not reach full bandwidth either. torch.compile comes within 4% of the hand-written kernel at the largest size (874 against 843 µs), but is slower at small sizes.

Inside the model

A kernel that is 7 times faster alone does not make the model 7 times faster. To measure the effect on the whole step, we wrote the model’s forward pass ourselves (about 25 lines, using the Hugging Face layers) so that each residual add is fused with the norm that follows it. One step then has 56 fused kernels instead of \(56 \times 9 = 504\) small ones. First we checked in FP32 that our forward pass (with the eager norms) gives exactly the same final features as the Hugging Face forward pass [Certain, our run]; in BF16 they differ a little, see the “Watch out” in Part 3. Then we timed one decode step (context 512, Algorithm 1):

Batcheager stepfused stepgainbound (132.1)
129.9 ms25.7 ms14%13.2 ms
439.5 ms35.8 ms10%13.5 ms
1676.5 ms71.7 ms6%14.3 ms

[Certain, our run; best of 15 steps] Fusion removed 504 of the 1,324 operations of a step (the count fell to 820), the issue time fell from 12.0 to 8.7 ms, and the step from 29.9 to 25.7 ms at batch 1. The gain is real but far from the 7× of the kernel alone, because the nine small kernels were only a part of the step. In percent the gain shrinks with the batch, because the matmuls and attention take longer while the norm kernels stay small.

Part 3: speculative decoding with two kinds of drafts

Decode is memory-bound: a pass over 5 tokens reads the same 3.09 GB of weights as a pass over 1 token. Speculative decoding (session 128) uses this. A cheap drafter guesses \(k\) tokens; the target checks all of them in one pass. With greedy decoding the check is simple: accept the guesses up to the first one that differs from the target’s own argmax, then add one token from the target (the bonus token). So every round makes between 1 and \(k + 1\) tokens, and the output is exactly what plain greedy decoding of the target would make.

Algorithm 3: greedy speculative decoding, one round, batch \(b\)
  1. input: the pending token \(g\) of each row (made by the target, not yet in its cache); draft length \(k\)
  2. drafter: \(d_1, \dots, d_k\) // \(k\) cheap passes, one token each
  3. target: one pass over \([g, d_1, \dots, d_k]\) \(\rightarrow\) greedy tokens \(G_0, \dots, G_k\) // \(G_j\) is the target's choice after \(d_j\)
  4. for each row: \(n \leftarrow\) the number of leading \(j\) with \(d_j = G_{j-1}\) // accepted guesses, \(0\) to \(k\)
  5.   output \(d_1, \dots, d_n\) and the bonus \(G_n\) // \(n + 1\) new tokens
  6.   mark the cache slots of \(d_{n+1}, \dots, d_k\) as invalid in the target’s and the drafter’s masks
  7.   \(g \leftarrow G_n\)

Line 6 is how the lab handles a batch. Each row accepts a different number of guesses, so after one round the rows have different lengths. Instead of moving data, we keep a boolean validity mask over the KV cache: rejected slots stay in memory but are masked out of attention, and each token’s position (for RoPE) is the count of valid slots before it. The cache grows by \(k + 1\) slots per round for every row; that is a little wasted memory, but no copying. Production engines use paged caches instead (session 134); the idea of “keep, but do not attend” is the same.

Draft 1: a small model

Qwen2.5-0.5B-Instruct runs Algorithm 3 directly. It keeps its own KV cache. In each round its first pass re-reads the guesses that were accepted and the bonus token (so its cache matches the target’s text), and its other \(k - 1\) passes are single tokens. One draft step reads 0.99 GB, about a third of the target.

Draft 2: an EAGLE-style head

EAGLE (Li et al., 2024) does not use a separate small model. It trains a small head that predicts the target’s next feature: the last hidden vector before the output layer. Its input at position \(t\) is the target’s feature at \(t-1\) and the embedding of the token at \(t\), joined, projected from \(2d\) to \(d\) by a linear layer, and passed through one transformer decoder layer. The output goes through the target’s own output layer to give a token. [Certain] Features carry much more information than tokens, so a one-layer head can guess well.

\[ \hat{\mathbf{f}}_t = \text{Layer}\big(\mathbf{W}_{fc}\,[\mathbf{f}_{t-1};\ \mathbf{e}(x_t)]\big), \qquad \mathcal{L} = \text{SmoothL1}\big(\hat{\mathbf{f}}_t, \mathbf{f}_t\big) + 0.1 \cdot \text{CE}\big(\mathbf{W}_{out}\hat{\mathbf{f}}_t,\ x_{t+1}\big) \tag{132.4} \]
Read the symbols
\(\mathbf{f}_t\)
the target’s final hidden vector at position \(t\) (after the last norm); the target’s next-token logits are \(\mathbf{W}_{out}\mathbf{f}_t\)
\(\mathbf{e}(x_t)\)
the target’s embedding of token \(x_t\) (frozen, shared)
\(\mathbf{W}_{fc}\)
the trained projection from \(2d = 3{,}072\) to \(d = 1{,}536\)
Layer
one trained decoder layer with the target’s shape (attention with its own small KV cache, SwiGLU FFN)
SmoothL1, CE
a regression loss on the feature and a cross-entropy loss on the token; EAGLE weights them 1 and 0.1 [Certain]

When drafting, the head feeds its own predicted feature \(\hat{\mathbf{f}}\) back in, step after step. After verification, the target gives true features for every accepted token, and the head re-reads those in its first pass of the next round. To make the head robust to its own small errors, EAGLE adds uniform noise in \([-0.1, 0.1]\) to the input features during training [Certain]; we did the same.

Algorithm 4: train an EAGLE-style head
  1. data: 2,048 GSM8K training questions with their reference answers, plus the target’s own greedy answers (160 tokens) to 1,024 other training questions, all in the target’s chat format // 3,072 sequences
  2. run the target once over each sequence with the same forward pass that will serve it; store \(\mathbf{f}_1, \dots, \mathbf{f}_T\) in BF16 // 683,011 tokens, about 2.1 GB
  3. initialize the head’s decoder layer and final norm from the target’s last layer and final norm // right feature scale from step 0
  4. for 4 epochs, batches of 8 sequences:
  5.   inputs \([\mathbf{f}_{t-1} + \text{noise};\ \mathbf{e}(x_t)]\), targets \(\mathbf{f}_t\) and \(x_{t+1}\)
  6.   loss (132.4), with the cross-entropy on 1,024 random positions per batch // the 151,936-wide output layer is the costly part
  7.   AdamW, learning rate \(5 \times 10^{-4}\) with warm-up and cosine decay, gradient clipping at 0.5

Feature extraction and training took about 40 minutes on our (very busy) laptop. At the end the head’s top-1 token accuracy on training batches was about 0.75 (one step ahead, with true input features). [Certain, our run] This is a toy version of EAGLE: the paper trains on a fixed dialogue dataset (ShareGPT, 68,000 dialogues) for 1 to 2 days on 4 A100 GPUs with a learning rate of \(3 \times 10^{-5}\), and drafts a tree of guesses instead of one chain [Certain]. Our head sees two kinds of text: human-written GSM8K answers (short, with calculations written as <<48/2=24>>) and the target’s own answers, which are longer explanations and are what the head must guess at test time. The test questions below are not in either training set. Testing on other kinds of text (chat, code) would lower the acceptance rate [Likely].

The correctness test

Greedy speculation must give exactly the same tokens as plain greedy decoding. We tested this in FP32 (4 prompts, 64 tokens, draft model, \(k = 4\)): all 256 tokens were equal. [Certain, our run] In BF16 they are not always equal: at batch 1, 10 of the 16 rows of the draft-model run were identical to plain decoding over all 128 tokens, and 81% of all token positions matched (EAGLE-style head: 11 of 16 rows, 78%). [Certain, our run] The reason is the one from session 87: the verification pass works on 5 tokens at a time and plain decoding on 1, so the matrix multiplies add numbers in a different order, and BF16 can flip a near-tie. After one flip, the two texts go different ways. Test the logic in FP32; in BF16, compare logits with a tolerance.

Part 4: acceptance and tokens per second at batch 1, 4 and 16

Two numbers describe a drafter. The acceptance rate \(\hat\alpha\) is the fraction of all guesses that the target accepted. The accepted length \(\tau\) is the average number of tokens a row gets per target pass (accepted guesses plus the bonus).

\[ \hat\alpha = \frac{\sum_{\text{rounds, rows}} n}{k \cdot \#(\text{rounds, rows})}, \qquad \tau = \overline{n + 1}, \qquad \tau(\alpha, k) = \frac{1 - \alpha^{k+1}}{1 - \alpha} \ \text{ if each guess is accepted with probability } \alpha \tag{132.5} \]
Read the symbols
\(n\)
accepted guesses of one row in one round (0 to \(k\))
\(\tau(\alpha, k)\)
the expected tokens per round from session 128, if every guess is accepted independently with probability \(\alpha\)

The speed-up is the accepted length times the time of one plain step, divided by the time of one round:

\[ S = \frac{\tau \cdot t_1(b)}{k\, t_d(b) + t_v(b, k+1)} \tag{132.6} \]
Read the symbols
\(t_1(b)\)
one plain target step for \(b\) rows
\(t_d(b)\)
one drafter step; \(c = t_d / t_1\) is the cost ratio of session 128
\(t_v(b, k+1)\)
one target pass over \(k + 1\) tokens per row: close to \(t_1\) while the pass is memory-bound, larger once it is not

The test: the first 16 questions of the GSM8K test set, in Qwen’s chat format, 128 new tokens per question, greedy, \(k = 4\), all with the fused forward pass. Batch 1 runs the 16 questions one by one, batch 4 in 4 groups, batch 16 in one group. Each group runs all four methods back to back, twice, and we keep the faster run of each, because other jobs shared our GPU. Tokens per second counts all rows and includes the prefill. Rows keep going after an end-of-text token, so that every row makes exactly 128 tokens.

Show
plain, eager normsplain, fused normsdraft model (0.5B)EAGLE-style head
Figure 3. Our measurements: Qwen2.5-1.5B-Instruct on 16 GSM8K test questions, 128 greedy tokens each, \(k = 4\), Apple M5 Pro, BF16. Each bar is the sum over the 16 questions of the faster of two runs.
How to read Figure 3

Start with “tokens per second”. At batch 1 both drafters beat plain decoding: the draft model by 1.22× and the EAGLE-style head by 1.27×. At batch 4 and 16 both are slower than plain decoding (0.86× and 0.91× for the draft model, 0.85× and 0.81× for the head). Compare also the first two bars of each group: the fused norms help plain decoding at every batch, by 4% to 14% in this noisy run. Switch to “accepted length \(\tau\)”: \(\tau\) hardly depends on the batch, because acceptance is a property of the text and the drafter, not of the hardware. Switch to “speed-up”: the gain falls with the batch even though \(\tau\) stays the same. Figure 4 explains why.

What the numbers say

  • The draft model guesses well but is not cheap. It reached \(\alpha = 0.78\) and \(\tau = 4.14\) out of 5 possible. But one draft step took 15.0 ms, 0.58 of a target step (25.9 ms), not the 0.32 that its bytes suggest (0.99 against 3.09 GB): at batch 1 both models are launch-bound, and the draft has 24 layers against 28. So the gain at batch 1 was only 1.22×. [Certain, our run]
  • The EAGLE-style head guesses worse but is very cheap. \(\alpha = 0.34\) and \(\tau = 2.38\), but one head step took 3.1 ms (0.12 of a target step), so it was the faster drafter at batch 1: 1.27×. Its acceptance falls fast along the chain: \(\alpha = 0.63\) with one guess per round, 0.20 with eight. [Certain, our run]
  • Acceptance hardly depends on the batch. \(\tau\) of the draft model was 4.14, 4.09 and 4.12 at batch 1, 4 and 16. Acceptance is a property of the text and the drafter. [Certain, our run]
  • At batch 4 and 16, speculation lost. Plain decoding uses the spare compute by batching: from batch 1 to 16 it went from 15.5 to 308.7 tokens per second. Speculation then fights three costs: the 5-token verification pass costs 1.14× a 1-token step at batch 4 and 1.15× at batch 16; a group runs until its slowest row is done (the counted tokens per row and round fell from 4.09 to 3.61 for the draft model and from 2.31 to 1.80 for the head); and each round has fixed CPU bookkeeping [Likely].
  • The numbers are noisy. Other jobs shared our GPU. Read the ratios inside one batch size, which were measured back to back, more than the absolute tokens per second.
DrafterBatch
Figure 4. The cost model (132.6) with our measured component times: one target step, one target pass over 5 tokens, and one drafter step, each at a 300-token context and the chosen batch (best of three timing runs). \(\tau\) comes from (132.5), with every guess accepted independently with probability \(\alpha\). The verification time for other \(k\) is interpolated linearly from the 1-token and 5-token passes (an assumption). The square is the speed-up we measured at \(k = 4\) in Figure 3.
How to read Figure 4

The slider starts at the acceptance rate we measured at \(k = 4\). At batch 1 the curve is highest at small \(k\) and falls as \(k\) grows: with these drafter costs, each extra guess costs more than the tokens it adds. Both squares lie above the curve at batch 1. For the draft model, the formula with \(\alpha = 0.78\) gives \(\tau = 3.26\), but we measured 4.14: acceptances are not independent; they come in runs, on easy stretches of text. Now switch to batch 4 and 16. The curve hardly moves, because our component times grow together (a 5-token pass costs 1.14 to 1.15 times a 1-token step), but the squares fall below 1. The readout shows the cost the model leaves out: a group runs until its slowest row is done, so each row gets fewer counted tokens per round than its own \(\tau\). Each round also has fixed bookkeeping on the CPU. Move the slider to see which \(\alpha\) would make longer drafts worth it.

Worked example: predict the batch-1 speed-up from three timings

Batch 1, draft model, \(k = 4\). We measured \(t_1 = 25.9\) ms, \(t_v = 26.1\) ms (5 tokens cost the same as 1) and \(t_d = 15.0\) ms, so \(c = 0.58\). With the measured \(\tau = 4.14\), (132.6) gives \(S = 4.14 \times 25.9 / (4 \times 15.0 + 26.1) = 1.24\). We measured 1.22×: close. For the EAGLE-style head, \(t_d = 3.1\) ms and \(\tau = 2.38\) give \(S = 1.60\), but we measured only 1.27×. The formula leaves out the costs of each round: the drafter’s first pass reads up to 5 tokens, and the CPU must read the accepted counts before it can start the next round, which empties the GPU queue [Likely]. With a cheap drafter these fixed costs are a large part of the round.

The lab code

Three files. lab.py holds the fused kernel and our forward pass with a validity mask; spec.py holds the drafters and the generation loop; the training script for the head follows the same pattern and is listed after them. We ran exactly this code on our machine. [Certain] It needs PyTorch 2.8 with MPS (for the Metal kernel; on CUDA, replace addnorm_fused by the Triton add_rmsnorm above) and transformers 4.57; other versions of transformers may change the layer call signatures.

# lab.py: Qwen2 forward pass with fused add + RMSNorm and a KV-validity mask (PyTorch 2.8, MPS)
import time, torch
from transformers import AutoModelForCausalLM, DynamicCache
dev = "mps"; sync = torch.mps.synchronize

SRC = r'''
#include <metal_stdlib>
using namespace metal;
kernel void add_rmsnorm(device const bfloat* x [[buffer(0)]], device bfloat* r [[buffer(1)]],
    device const bfloat* w [[buffer(2)]], device bfloat* y [[buffer(3)]],
    constant uint& D [[buffer(4)]], constant float& eps [[buffer(5)]],
    uint row [[threadgroup_position_in_grid]], uint tid [[thread_position_in_threadgroup]],
    uint nt [[threads_per_threadgroup]], uint lane [[thread_index_in_simdgroup]],
    uint sg [[simdgroup_index_in_threadgroup]]) {
  threadgroup float part[32];
  float acc = 0.0f;
  device const bfloat* xr = x + row * D; device bfloat* rr = r + row * D;
  for (uint i = tid; i < D; i += nt) {
    bfloat h = bfloat(float(xr[i]) + float(rr[i]));    // residual add, rounded like eager
    rr[i] = h; float hf = float(h); acc += hf * hf;      // new residual, in place
  }
  acc = simd_sum(acc);                                   // reduce 32 lanes
  if (lane == 0) part[sg] = acc;
  threadgroup_barrier(mem_flags::mem_threadgroup);
  if (sg == 0) { float v = lane < (nt + 31) / 32 ? part[lane] : 0.0f; v = simd_sum(v); if (lane == 0) part[0] = v; }
  threadgroup_barrier(mem_flags::mem_threadgroup);
  float inv = rsqrt(part[0] / float(D) + eps);
  for (uint i = tid; i < D; i += nt) y[row * D + i] = bfloat(float(bfloat(float(rr[i]) * inv)) * float(w[i]));
}
'''
lib = torch.mps.compile_shader(SRC)

def addnorm_eager(x, r, w, eps):          # what Hugging Face does: 9 kernels
    h = x + r
    hf = h.to(torch.float32)
    hf = hf * torch.rsqrt(hf.pow(2).mean(-1, keepdim=True) + eps)
    return h, w * hf.to(x.dtype)

def addnorm_fused(x, r, w, eps):          # 1 kernel: h is written into r
    D = x.shape[-1]; x2 = x.reshape(-1, D).contiguous(); r2 = r.reshape(-1, D).contiguous()
    y = torch.empty_like(x2)
    lib.add_rmsnorm(x2, r2, w, y, D, eps, threads=x2.shape[0] * 256, group_size=256)
    return r2.view(x.shape), y.view(x.shape)

def load(name):
    return AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16).to(dev).eval()

def mask4d(am, T):                        # am: (B, Ltot) bool validity; the last T columns are new
    Lt = am.shape[1]; q = torch.arange(Lt - T, Lt, device=am.device); k = torch.arange(Lt, device=am.device)
    m = (k[None, :] <= q[:, None])[None] & am[:, None, :]          # causal and valid
    m = m | (k[None, None, :] == q[None, :, None])                  # a slot always sees itself
    return m[:, None]                                               # (B, 1, T, Ltot), True = attend

def positions(am, T):                     # RoPE position = number of valid slots before
    return (am.long().cumsum(-1) - 1).clamp(min=0)[:, -T:]

@torch.no_grad()
def run_layers(M, layers, x, past, am, fused, final_w):
    T = x.shape[1]; L0 = past.get_seq_length()
    cp = torch.arange(L0, L0 + T, device=x.device); mask = mask4d(am, T)
    pe = M.rotary_emb(x, positions(am, T))
    an = addnorm_fused if fused else addnorm_eager; eps = M.config.rms_norm_eps
    r = x; h = layers[0].input_layernorm(x)
    for i, lyr in enumerate(layers):
        a, _ = lyr.self_attn(hidden_states=h, attention_mask=mask, past_key_values=past,
                             cache_position=cp, position_embeddings=pe)
        r, h = an(a, r, lyr.post_attention_layernorm.weight, eps)               # add + norm
        f = lyr.mlp(h)
        nxt = layers[i + 1].input_layernorm.weight if i + 1 < len(layers) else final_w
        r, h = an(f, r, nxt, eps)                                                # add + next norm
    return h                              # final normalized hidden states (B, T, d)

def fwd(m, ids, past, am, fused):
    return run_layers(m.model, m.model.layers, m.model.embed_tokens(ids), past, am, fused, m.model.norm.weight)

def bench(f, warmup=3, reps=10):          # best and median wall time, GPU queue drained
    for _ in range(warmup): f()
    sync(); ts = []
    for _ in range(reps):
        t = time.perf_counter(); f(); sync(); ts.append(time.perf_counter() - t)
    ts.sort(); return ts[0], ts[len(ts) // 2]
# spec.py: drafters and greedy (speculative) generation with per-row acceptance
import copy, time, torch, torch.nn as nn
from transformers.models.qwen2.modeling_qwen2 import Qwen2DecoderLayer, Qwen2RMSNorm, Qwen2RotaryEmbedding
from lab import *

class Head(nn.Module):                    # EAGLE-style: [f_(t-1); e(x_t)] -> fc -> one decoder layer -> f_t
    def __init__(s, cfg):
        super().__init__(); c = copy.deepcopy(cfg); c.num_hidden_layers = 1; s.config = c
        s.fc = nn.Linear(2 * c.hidden_size, c.hidden_size, bias=False)
        s.layers = nn.ModuleList([Qwen2DecoderLayer(c, 0)])
        s.norm = Qwen2RMSNorm(c.hidden_size, c.rms_norm_eps); s.rotary_emb = Qwen2RotaryEmbedding(c)
    def inp(s, feat, emb): return s.fc(torch.cat([feat, emb], -1))

class Drafter:
    """kind = "model" (a small LM) or "eagle" (a Head). Keeps its own KV cache and validity mask."""
    def __init__(s, kind, net, target, fused): s.kind, s.net, s.t, s.fused = kind, net, target, fused
    def _run(s, X, F, v):
        s.am = torch.cat([s.am, v], 1)
        if s.kind == "model":
            h = fwd(s.net, X, s.past, s.am, s.fused); return h, s.net.lm_head(h)
        x = s.net.inp(F, s.t.model.embed_tokens(X))
        h = run_layers(s.net, s.net.layers, x, s.past, s.am, s.fused, s.net.norm.weight)
        return h, s.t.lm_head(h)                         # the target's own output layer
    def start(s, ids, am, g, H):                        # prompt (+ target features H) and first token g
        s.past = DynamicCache(); s.am = am[:, :0]; s.steps = []
        one = am[:, :1] | True
        if s.kind == "model": X, v, F = torch.cat([ids, g[:, None]], 1), torch.cat([am, one], 1), None
        else: X, v, F = torch.cat([ids[:, 1:], g[:, None]], 1), torch.cat([am[:, 1:], one], 1), H
        return s.first(X, F, v)
    def first(s, X, F, v):                              # re-read accepted tokens + bonus (width k+1)
        if s.steps: s.am[:, s.steps] = False; s.steps = []          # forget last round's guesses
        h, lg = s._run(X, F, v)
        idx = (v.long() * torch.arange(1, v.shape[1] + 1, device=v.device)).argmax(1)   # last valid slot
        ar = torch.arange(X.shape[0], device=X.device)
        return lg[ar, idx].argmax(-1), h[ar, idx][:, None]
    def step(s, tok, fh):                               # one more guess
        s.steps.append(s.am.shape[1])
        h, lg = s._run(tok[:, None], fh, torch.ones_like(tok[:, None], dtype=torch.bool))
        return lg[:, -1].argmax(-1), h[:, -1:]

def left_pad(seqs, pad):
    P = max(map(len, seqs))
    ids = torch.tensor([[pad] * (P - len(x)) + x for x in seqs], device=dev)
    am = torch.tensor([[False] * (P - len(x)) + [True] * len(x) for x in seqs], device=dev)
    return ids, am

@torch.no_grad()
def generate(target, seqs, N, drafter=None, k=4, fused=True, pad=151643):
    ids, am = left_pad(seqs, pad); B = ids.shape[0]; sync(); t0 = time.perf_counter()
    past = DynamicCache(); H = fwd(target, ids, past, am, fused)
    g = target.lm_head(H[:, -1]).argmax(-1); out = [[x] for x in g.tolist()]; acc = []
    if drafter is None:                                 # plain greedy decoding
        while min(map(len, out)) < N:
            am = torch.cat([am, am[:, :1] | True], 1)
            g = target.lm_head(fwd(target, g[:, None], past, am, fused)[:, -1]).argmax(-1)
            for b, x in enumerate(g.tolist()): out[b].append(x)
        sync(); return torch.tensor([o[:N] for o in out]), dict(time=time.perf_counter() - t0)
    d, fh = drafter.start(ids, am, g, H)
    while min(map(len, out)) < N:
        D = [d]
        for _ in range(k - 1):
            d, fh = drafter.step(d, fh); D.append(d)
        D = torch.stack(D, 1)                                        # (B, k) guesses
        V = torch.cat([g[:, None], D], 1); am = torch.cat([am, V > -1], 1)
        H = fwd(target, V, past, am, fused); G = target.lm_head(H).argmax(-1)   # one verify pass
        n = (D == G[:, :k]).long().cumprod(1).sum(1)                  # accepted guesses per row
        bonus = G.gather(1, n[:, None])[:, 0]
        j = torch.arange(k + 1, device=dev)
        am[:, -k - 1:] = j[None] <= n[:, None]                       # rejected slots: masked out
        for b, (nb, Db, bb) in enumerate(zip(n.tolist(), D.tolist(), bonus.tolist())):
            if len(out[b]) < N: acc.append(nb)
            out[b] += Db[:nb] + [bb]
        X = torch.cat([D, D[:, :1]], 1).scatter(1, n[:, None], bonus[:, None])
        d, fh = drafter.first(X, H, j[None] <= n[:, None]); g = bonus
    sync(); a = torch.tensor(acc, dtype=torch.float)
    return torch.tensor([o[:N] for o in out]), dict(time=time.perf_counter() - t0,
        alpha=(a.sum() / (k * len(a))).item(), tau=(a + 1).mean().item())

if __name__ == "__main__":
    import json
    from transformers import AutoTokenizer
    T = "Qwen/Qwen2.5-1.5B-Instruct"; tok = AutoTokenizer.from_pretrained(T)
    tg, dm = load(T), load("Qwen/Qwen2.5-0.5B-Instruct")
    qs = ["Janet has 3 apples and buys 5 more. She eats 2. How many apples does she have?",
          "A train travels 60 km per hour for 2.5 hours. How far does it go?"]
    seqs = [tok.apply_chat_template([{"role": "user", "content": q}], add_generation_prompt=True) for q in qs]
    a, s0 = generate(tg, seqs, 128)                                   # plain greedy
    b, s1 = generate(tg, seqs, 128, Drafter("model", dm, tg, True), k=4)
    print(f"plain {2 * 128 / s0['time']:.0f} tok/s; draft model {2 * 128 / s1['time']:.0f} tok/s, "
          f"alpha {s1['alpha']:.2f}, tau {s1['tau']:.2f}, same tokens {(a == b).float().mean():.3f}")

The training script for the head (Algorithm 4) uses the same Head. Its setup and one batch:

# train_eagle.py (setup): features from our own forward pass, head started from the target's last layer
feats = [fwd(tg, torch.tensor([q], device=dev), DynamicCache(), torch.ones(1, len(q), dtype=torch.bool, device=dev), True)[0].cpu()
         for q in seqs]                                        # not tg.model(...): see "Watch out" in Part 3
head = Head(tg.config).to(dev, torch.float32)
for p in head.parameters():
    if p.dim() > 1: torch.nn.init.normal_(p, std=0.02)
head.layers[0].load_state_dict(tg.model.layers[-1].state_dict()); head.norm.weight.data.copy_(tg.model.norm.weight.float())
# train_eagle.py (core, one batch): X tokens x_t, Fin features f_(t-1), Ft features f_t, Y next tokens x_(t+1)
Fin = Fin + (torch.rand_like(Fin) - 0.5) * 0.2                 # uniform noise in [-0.1, 0.1]
x = head.inp(Fin, E[X].float())                                # E: the target's embedding matrix
pos = torch.arange(Lm, device=dev)[None].expand(BS, -1)
mask = torch.ones(Lm, Lm, device=dev, dtype=torch.bool).tril()[None, None]
h = head.norm(head.layers[0](x, attention_mask=mask, position_embeddings=head.rotary_emb(x, pos), position_ids=pos))
lreg = (F.smooth_l1_loss(h, Ft, reduction="none").mean(-1) * Mk).sum() / Mk.sum()   # Mk: real tokens
sel = (Y >= 0).view(-1).nonzero()[:, 0]
sel = sel[torch.randperm(len(sel), generator=g)[:1024].to(dev)]  # CE on 1,024 positions
lcls = F.cross_entropy((h.reshape(-1, h.shape[-1])[sel].bfloat16() @ W.T).float(), Y.view(-1)[sel])
loss = lreg + 0.1 * lcls                                       # formula (132.4)
opt.zero_grad(); loss.backward(); torch.nn.utils.clip_grad_norm_(head.parameters(), 0.5); opt.step(); sched.step()

Things to try next: on a CUDA GPU, wrap the decode step in a CUDA graph and see how much of the gap to the bound disappears; fuse RoPE into the attention call; draft a small tree of guesses instead of one chain (EAGLE’s default) and measure how much \(\tau\) grows; test the head on non-math prompts and watch \(\hat\alpha\) drop.

Where this is used

WhereWhat from this labWhat it decides
Serving engines (vLLM, SGLang, TensorRT-LLM)Fused norm kernels, CUDA graphs for decode, speculative decoding with draft models and EAGLE heads [Certain]Which features to turn on for a given model, batch size and traffic
Performance work on a new modelAlgorithm 1: step time, issue time, operation counts; the ratio to the bound (132.1)Whether to fix launches, kernels or bytes first
Kernel libraries (for example vLLM’s custom ops, FlashInfer) [Likely]Fused add + RMSNorm and similar kernels, tested against eager codeHow much a kernel saves alone and in a full step
Interactive chat at batch 1 (laptops, phones, a single user)Speculative decoding with a small drafterThe largest latency gain is at low batch, where the GPU has spare compute
High-throughput batch servingAcceptance and speed-up measured at your real batch size (Figure 4)Whether speculation still pays, and with which \(k\); often a smaller \(k\) at larger batch
Long-output tasks: code, math, video captioning and dense video descriptionEAGLE-style heads trained on the target’s own outputs for that domainPredictable outputs (code, structured captions, timestamps in key-moment lists) give high acceptance
Vision-language modelsThe same drafters over the language model; the image or video tokens are only in the prefillDecode speed of answers in video question answering and Video-RAG pipelines

Common mistakes

  • Timing a decode step that changes the cache. Each call adds a token, so you measure longer and longer contexts. Crop the cache back before every timed call, or use a static cache.
  • Reading a CPU-only profile as GPU time. The operator that synchronizes collects the waiting time. Count operations with the profiler; time the GPU with a trace (Nsight Systems) or with events.
  • Measuring a fused kernel only alone. 7× faster alone was 14% faster in the step. Always measure the end-to-end step too.
  • Skipping the bit-for-bit test. Write the fused kernel to reproduce eager’s roundings first; only then make it “more accurate”, and switch to a tolerance test.
  • Testing speculative decoding for equal tokens in BF16. Near-ties flip. Test exact equality in FP32; in BF16, compare logits with a tolerance.
  • Using “the minimum acceptance of the batch” for every row. Correct, but at batch 16 the minimum is often 0, and speculation looks useless. Accept per row and mask rejected cache slots.
  • Reporting acceptance from one prompt type. Our head is trained and tested on math answers. Acceptance depends on the text: measure it on traffic like yours.
  • Assuming the batch-1 speed-up holds at batch 16. The verification pass over \(b(k+1)\) tokens stops being free as the batch grows (Figure 4). Measure the speed-up at the batch you serve.

Interview questions

Your decode step takes twice the bandwidth bound. How do you find out why?core
Compute the bound first: weight bytes plus cache bytes divided by the measured bandwidth (132.1). Then measure the step correctly (warm-up, fixed context by cropping the cache, synchronize, repeat) and also the issue time, the time until the CPU has queued all kernels. If the issue time is close to the step time, the step is launch-bound: count the operations in a profiler (ours had about 1,324 per step) and reduce them with fusion or a CUDA graph. If the GPU is busy all the time, look at the long kernels in an Nsight Systems trace: matrix-vector products that do not reach full bandwidth, attention that copies the cache, hidden syncs such as .item().
Why can a CPU-side profiler table mislead you about GPU time?basic
GPU work is asynchronous: the CPU queues a kernel and moves on. A CPU-side profiler measures how long each operator call took on the CPU. When an operator has to wait for the GPU, for example a copy of the result to the CPU, all the waiting time is charged to that operator. In our profile, a type-cast operation (aten::copy_) had the most self CPU time for this reason. Use the table to count calls, and use a GPU timeline (Nsight Systems, or the profiler with CUDA activities) to see kernel durations and gaps.
How many bytes does an unfused residual add + RMSNorm move, and how many does a fused kernel move?core
In eager PyTorch with BF16 inputs and an FP32 norm, about 42 bytes per element: the add (6), the cast to FP32 (6), pow (8), mean (4), the multiply by the inverse root (8), the cast back (6) and the multiply by the weight (4). A fused kernel reads the input and the residual and writes the new residual and the output: 8 bytes per element. So fusion cuts traffic by 5.25×. At decode the bytes are tiny and the gain comes from launching one kernel instead of nine; at prefill the gain follows the bytes. We measured 7.2× alone at one row and 8.0× at 16,384 rows.
Why did your fused kernel make the whole decode step only a little faster?core
Because add + RMSNorm is a small part of the step. The step also has 197 matmuls that read 3.09 GB of weights, attention, RoPE, cache appends and many other small kernels. Fusion removed 504 of 1,324 operations and saved 4.3 ms at batch 1 (14%). Amdahl’s law applies: the speed-up of the whole is limited by the part you did not change. The remaining launch overhead needs more fusion or a CUDA graph, and the remaining memory time needs fewer bytes (quantization).
How do you test that a fused kernel is correct?basic
Write it to follow the eager code’s rounding steps exactly (in our case: round the sum to BF16, compute the norm in FP32, round the normalized value to BF16 before the weight multiply), then compare bit for bit on random inputs of many sizes, including sizes that are not a multiple of the block. Check both outputs (the new residual and the normalized value). Then run the whole model and check that the generated tokens are the same. If you later change the rounding for accuracy, switch to a tolerance test of a few BF16 steps.
Speculative decoding with greedy acceptance: is the output exactly the same as plain greedy decoding?basic
In exact arithmetic yes: every output token is the target’s argmax given the tokens before it, whether it came from an accepted guess or from the bonus. In our FP32 test all 256 tokens matched. In BF16 the verification pass computes the logits of 5 positions at once, with different matrix shapes and summation order than a 1-token step, so a near-tie can flip and the texts diverge. That is a numerical effect, not a bug, and production engines have the same behavior.
How does an EAGLE head differ from a small draft model, and why can it be better?core
A draft model is a separate language model that sees only tokens. An EAGLE head is one transformer layer that sees the target’s own last hidden features plus the next token’s embedding, predicts the next feature, and uses the target’s output layer to get a token. The features contain what the target “is about to say”, so a much smaller network can match the target’s choices. It also needs no separate tokenizer or pretraining, only training on target features. Costs: it must be trained for each target model, and it needs the target’s features, so it lives inside the serving engine. In our lab the head (51.5 million parameters plus the shared output layer) reached \(\alpha = 0.34\) and \(\tau = 2.38\), against 0.78 and 4.14 for the 0.5B draft model, yet it was slightly faster at batch 1 (1.27× against 1.22×), because one head step costs 3.1 ms against 15.0 ms. With far more training data, as in the paper, its acceptance would rise [Likely].
Why does the speed-up of speculative decoding shrink at larger batch, even when the acceptance rate stays the same?advanced
The speed-up is \(\tau t_1 / (k t_d + t_v)\) (132.6). At batch 1 a decode step is memory-bound, so verifying 5 tokens costs about the same as 1, and \(t_v \approx t_1\). At larger batch the verification pass processes \(b(k+1)\) tokens; once that crosses the hardware’s ridge (or the kernels stop being memory-bound), \(t_v\) grows with the tokens. Plain decoding already used the spare compute by batching. In our measurements the 5-token pass cost 1.01× a 1-token step at batch 1 and 1.15× at batch 16. Draft steps also grow with the batch. So the same \(\tau\) buys less time. In our lab a second effect was even larger: a batch runs until its slowest row is done, so the counted tokens per round fall below the average \(\tau\) (for the draft model, from 4.09 at batch 1 to 3.61 at batch 4). Continuous batching, where finished requests leave and new ones join, reduces this waste in real engines [Likely].
How do you handle different numbers of accepted tokens per row in a batch?advanced
Options: (1) accept the minimum of the batch for all rows: simple, correct, wasteful; (2) keep the rejected KV slots in memory but mask them out of attention, with RoPE positions taken from the count of valid slots, as in this lab: no copying, a little wasted memory; (3) a paged KV cache, where each row owns its blocks and the engine frees or overwrites rejected slots: the production solution in vLLM-style engines. In every case the drafter’s cache must also be corrected, because its guesses after the first rejection are now wrong context.
How would you choose the draft length \(k\)?core
Measure \(\alpha\) on representative traffic, measure \(t_1\), \(t_d\) and the verification time for a few \(k\) at your batch size, and maximize (132.6). The tokens per round \((1 - \alpha^{k+1})/(1 - \alpha)\) saturate, while the cost grows linearly in \(k\), so the best \(k\) is small when \(\alpha\) is low or the drafter is expensive, and larger when \(\alpha\) is high and the drafter is cheap. At larger batch the best \(k\) usually drops. Engines often adapt \(k\) per request or use draft trees with a confidence threshold (EAGLE-2).
Your offline benchmark shows a 2× speed-up from speculation, but production shows 1.1×. What could explain it?advanced
Different text: the drafter was tuned or measured on easy, predictable prompts, and real traffic has lower acceptance. Sampling: with temperature above 0, acceptance drops compared with greedy. Batch size: production runs at large batch, where verification is no longer free (Figure 4). Extra overheads: the drafter’s launches, the bookkeeping of variable accepted lengths, and memory for the drafter’s KV cache, which reduces the batch the engine can hold. Measure acceptance and step times on production traffic and at production batch.

Exercises

  1. Use (132.1) to compute the decode bound of Qwen2.5-0.5B (0.99 GB) on our machine (233 GB/s). If one draft step took 3 times that bound, what cost ratio \(c\) would it have against a target step of 25.7 ms?
  2. With (132.3), how many bytes do the eager and the fused add + RMSNorm move in one decode step at batch 16 (56 fused pairs, \(d = 1{,}536\))? Compare with the 3.09 GB of weights. What does this tell you about where the fusion gain at decode comes from?
  3. With \(\alpha = 0.7\), compute \(\tau\) from (132.5) for \(k = 2, 4, 8\). How many extra tokens per round does going from 4 to 8 guesses buy?
  4. At batch 1, suppose \(t_1 = 27\) ms, \(t_v = 29\) ms for 5 tokens, and a drafter step costs 9 ms. With \(\tau = 3.5\) and \(k = 4\), what speed-up does (132.6) predict? What if the drafter step cost 3 ms?
  5. The batch-16 verification pass handles \(16 \times 5 = 80\) tokens. Using the ridge of our machine (about 125 FLOP per byte, from session 87) and the fact that a pass of \(n\) tokens has intensity about \(n\) FLOP per byte in BF16, is this pass memory-bound or compute-bound in theory? Why can the measured cost still grow?
  6. Run the lab on your own GPU. Report the eager and fused step times at batch 1, \(\hat\alpha\) and \(\tau\) for the draft model, and the speed-up at batch 1 and 16. On a CUDA GPU, add a CUDA graph and report the new step time.
Answerscheck
1. \(0.988 / 233 = 4.24\) ms; three times that is 12.7 ms, so \(c = 12.7 / 25.7 = 0.50\). (We measured 15.0 ms for a draft step, \(c = 0.58\) against our 25.9 ms target step: the draft model is far from its bound.) 2. Eager: \(42 \times 16 \times 1{,}536 \times 56 = 57.8\) MB; fused: \(8 \times 16 \times 1{,}536 \times 56 = 11.0\) MB. That is 1.9% and 0.36% of the 3.09 GB of weights, so the bytes saved are tiny; the gain at decode comes from launching 56 kernels instead of 504. 3. \(\tau(0.7, 2) = 2.19\), \(\tau(0.7, 4) = 2.77\), \(\tau(0.7, 8) = 3.20\): four more guesses buy only 0.43 tokens per round. 4. \(S = 3.5 \times 27 / (4 \times 9 + 29) = 94.5 / 65 = 1.45\); with a 3 ms drafter \(S = 94.5 / 41 = 2.30\). 5. 80 FLOP per byte is below the ridge of about 125, so in theory the pass is still memory-bound. It still costs more than a 1-token pass because the small kernels and attention scale with the tokens, the matmul kernels for 80 rows do not run at full bandwidth, and the output layer writes 80 rows of 151,936 logits. 6. Your own numbers; compare the ratios (fused against eager, speculative against plain) with ours, not the absolute times.

Key takeaways

  • Profile before you optimize: time one decode step at a fixed context (crop the cache), measure its issue time, and count its operations. Ours had about 1,324 operations and took 2.3× the bandwidth bound at batch 1.
  • A CPU-side profiler counts operations; only a GPU timeline (Nsight Systems on NVIDIA) shows when the GPU is busy. Synchronizing operations collect the waiting time.
  • Eager residual add + RMSNorm is 9 kernels and about 42 bytes per element; fused, it is 1 kernel and 8 bytes. Alone it was 7 to 8× faster; in the whole step it saved 14% at batch 1.
  • Write a fused kernel to match eager’s roundings, test it bit for bit, measure it alone, then measure the step.
  • Greedy speculative decoding gives the target’s exact greedy output (in FP32). Per-row acceptance with a validity mask over the KV cache makes it work in a batch.
  • On our machine speculation paid only at batch 1: 1.22× with a 0.5B draft model and 1.27× with a small EAGLE-style head (cheaper steps beat higher acceptance). At batch 4 and 16 it was slower than plain decoding. Train a head on the features your serving code produces, and measure at the batch you serve.

Key formulas

\[ t_{\text{step}} \ge \frac{sN + bc\,\text{kv}_{\text{tok}}}{\text{BW}}, \qquad Q_{\text{eager}} \approx 42\,nd, \qquad Q_{\text{fused}} = 8\,nd \]
\[ \tau(\alpha, k) = \frac{1 - \alpha^{k+1}}{1 - \alpha}, \qquad S = \frac{\tau\, t_1}{k\, t_d + t_v} \]
\[ \mathcal{L}_{\text{EAGLE}} = \text{SmoothL1}(\hat{\mathbf{f}}_t, \mathbf{f}_t) + 0.1\,\text{CE}(\mathbf{W}_{out}\hat{\mathbf{f}}_t, x_{t+1}) \]

References for this session

  1. OpenAI and community (docs). Triton language documentation
  2. Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
  3. Li et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
  4. Chen et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling
  5. Tillet et al. (2019). Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations
  6. Zhang & Sennrich (2019). Root Mean Square Layer Normalization
  7. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  8. Cai et al. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
  9. Qwen Team (2024). Qwen2.5 Technical Report
  10. Li, Wei, Zhang, Zhang (2024). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
  11. Li et al. (2025). EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
  12. NVIDIA (docs). Nsight Systems User Guide
  13. PyTorch (docs). torch.profiler (documentation)
  14. PyTorch (docs). torch.mps.compile_shader (documentation)