profile, fuse and speculate
In this lab you take one real model, Qwen2.5-1.5B-Instruct, and make its decoding faster in three steps. You profile one decode step and find that hundreds of tiny kernels and the CPU that launches them cost a large part of the time. You fuse the residual add with RMSNorm into one kernel and measure the gain. Then you run speculative decoding with a small draft model and with an EAGLE-style head that you train yourself, and you measure the acceptance rate and the tokens per second at batch 1, 4 and 16.
What you will learn
- How to profile one decode step: the operator table, the count of small kernels, issue time against GPU time, and the Nsight Systems commands for an NVIDIA GPU
- How to estimate the bytes of an unfused residual add + RMSNorm (42 bytes per element in eager PyTorch) against a fused kernel (8 bytes), and why fusion helps even more at batch 1, where launches dominate
- How to write the fused kernel in Triton (and in Metal, which is what we ran), test it bit for bit against the eager code, and measure it alone and inside the model
- How to run greedy speculative decoding in a batch where each row accepts a different number of tokens, using a validity mask over the KV cache
- How to train a small EAGLE-style draft head on the target model's own features, and how it compares with a separate small draft model
- How to measure acceptance rate, accepted length and tokens per second at batch 1, 4 and 16, and explain with a cost model why the gain shrinks as the batch grows
- Why the speculative output must equal plain greedy decoding in FP32, and why it can differ in BF16
Why this matters
The last ten sessions gave you many ways to make decoding faster: read a profile, fuse small kernels, launch a whole step at once, and guess several tokens with a cheap model. Each idea comes with a promise, such as “fusion saves memory traffic” or “speculation gives 2 to 3 times”. In this lab you test the promises on one real model and one real machine. You profile one decode step, write one fused kernel and measure it, and run speculative decoding with two kinds of drafts at three batch sizes. At the end you can say, with your own numbers, which trick helped, by how much, and why.
The lab connects four sessions. Session 123 taught you to read an operator table and an Nsight Systems trace. Session 127 explained fusion and CUDA graphs. Session 128 derived the acceptance rate and the speed-up formula, and Session 129 showed the kinds of drafts, including EAGLE. Session 131 showed how a production engine packs all of this together. Here you do each step by hand, so that nothing is a black box.
The work has four parts, one per key point of the session:
- Profile one decode step and find where the time goes.
- Fuse the residual add and RMSNorm into one kernel, check it, and measure it alone and inside the model.
- Run speculative decoding with a small draft model and with an EAGLE-style head that you train.
- Measure the acceptance rate and the tokens per second at batch 1, 4 and 16.
Our machine and what we could not run. All numbers come from one laptop: an Apple M5 Pro (20-core GPU, 48 GB of unified memory), PyTorch 2.8 with the MPS back-end and Hugging Face transformers 4.57, in BF16. Other jobs shared the GPU while we measured, so our times are noisy; we report the best of several repeats unless we say otherwise. Two tools of the key points need an NVIDIA GPU: Nsight Systems and Triton. For both we give the exact CUDA commands and code, marked [not run by us]. On our machine we used the closest equivalents: the PyTorch profiler, and the same fused kernel written in Metal (Apple’s GPU language) through torch.mps.compile_shader. The method and the byte counts are the same on any GPU; the absolute numbers are not.
The lab models
The target (the model whose output we want) is Qwen2.5-1.5B-Instruct. The draft model is Qwen2.5-0.5B-Instruct: same tokenizer, same chat format, about a third of the bytes. Their shapes, from each model’s config.json [Certain]:
| Qwen2.5-1.5B-Instruct (target) | Qwen2.5-0.5B-Instruct (draft) | |
|---|---|---|
| layers \(L\), width \(d\) | 28, 1,536 | 24, 896 |
| query heads, KV heads, head size | 12, 2, 128 | 14, 2, 64 |
| FFN width (SwiGLU) | 8,960 | 4,864 |
| vocabulary, tied output layer | 151,936, yes | 151,936, yes |
| parameters (PyTorch count) | 1,543,714,304 (3.09 GB in BF16) | 494,032,768 (0.99 GB) |
| KV cache per token | 28,672 bytes | 12,288 bytes |
The third draft is a small network that you train in Part 3: an EAGLE-style head, one transformer layer of the target’s shape plus a projection, 51.5 million parameters. It reuses the target’s embedding and output layer.
A decode step makes one token per sequence. It must read every weight once: 3.09 GB. On our machine a large copy reaches about 233 GB/s (measured in session 87, same laptop). So one step at batch 1 cannot take less than \(3.087 / 233 = 13.2\) ms, and the fastest possible decoding is about 75 tokens per second. A context of 512 tokens adds \(512 \times 28{,}672 = 14.7\) MB of cache, which is less than 0.5% of the weights. So at batch 1 the bytes are the weights, and the bound is 13.2 ms. Keep this number in mind: every measurement below is compared with it.
- Read the symbols
- \(N, s\)
- number of weights (1.54 billion) and bytes per number (2 for BF16)
- \(b, c\)
- sequences in the batch and tokens already in each one’s cache
- \(\text{kv}_{\text{tok}}\)
- bytes of keys and values per token: \(2 \cdot L \cdot n_{kv} \cdot d_h \cdot s\), which is \(2 \cdot 28 \cdot 2 \cdot 128 \cdot 2\) \(= 28{,}672\)
- \(\text{BW}\)
- memory bandwidth you can really reach (measured, not the spec sheet)
Part 1: profile one decode step
A profile is a record of what ran and how long it took. For one decode step you want three things: how many GPU operations (kernels) the step launches, how long the CPU needs to issue them, and how long the GPU needs to run them. If the CPU needs longer than the GPU, the GPU waits between kernels and the step is launch-bound (session 123).
- input: model, a prompt of \(c\) tokens, batch \(b\)
- prefill the prompt once; keep the cache; remember its length \(c\)
- define step(): crop the cache back to \(c\); run the model on one new token; take the argmax // the same work every time
- run step() 3 times // warm-up: allocation, kernel choice, shader compilation
- repeat 15 times: \(t_0\); step(); \(t_1\) // issue time: the CPU is done; copy the token to the CPU; \(t_2\) // step time: the GPU is done
- report min and median of \(t_2 - t_0\), and the median of \(t_1 - t_0\)
- run step() once under the profiler; count the operations by name // what is launched, and how often
The trick in line 3 matters: a decode step changes the cache, so a timing loop that calls the model again and again would measure longer and longer contexts. Cropping the cache back gives the same step every time.
What the profile shows
We profiled one eager step at batch 1 (our own forward pass with the eager norms, which runs the same operations as the Hugging Face model; see Part 2) with a 512-token context. The PyTorch profiler lists 1,324 operations that do real work (we counted the lowest-level aten:: calls and left out those that only make a view of a tensor or allocate memory) [Certain, our run]. The largest groups:
| Operations | Count per step | Where they come from |
|---|---|---|
matrix multiplies (aten::linear) | 197 | 7 per layer (q, k, v, o, gate, up, down) × 28, plus the output layer |
multiplies and adds (aten::mul, aten::add) | 254 and 112 | RMSNorm, the residual stream, RoPE, the SwiGLU product |
type casts (aten::copy_) | 234 | mostly the casts of RMSNorm, which works in FP32 (BF16 → FP32 and back, 57 norms per step); after fusion 66 remain |
concatenations (aten::cat) | 113 | RoPE’s “rotate half” and the cache append (the cache grows by concatenation) |
pow, mean, rsqrt | 57 each | one of each per RMSNorm: 2 per layer plus the final norm, \(2 \times 28 + 1 = 57\) |
attention (scaled_dot_product_attention) | 28 | one per layer |
So only about 15% of the operations are the matrix multiplies that read the weights. The rest are small element-wise operations on a vector of 1,536 numbers. Each of them moves only a few kilobytes, but each one costs a launch: the CPU must run Python, dispatch the operator and send a command to the GPU.
Batch 1, context 512, eager: the step took 29.9 ms (best of 15; median 30.2 ms). The CPU needed 12.0 ms just to issue it. The bound (132.1) is 13.2 ms. So we are at 2.3× the bound. The issue time alone is 0.9× the bound, about 9 µs of CPU time for each of the 1,324 operations: even with an infinitely fast GPU, this step could not take less than 12.0 ms. [Certain, our run]
At batch 16 the step took 76.5 ms, for 16 tokens. The bound grows only a little (16 caches of 512 tokens add 0.23 GB), so batching is still almost free; but the gap to the bound tells you that much of the step is overhead, not memory traffic.
The profiler has a limit you must know. On MPS (and on CUDA with only CPU activities on) it measures CPU time per operator. When the CPU must wait for the GPU, for example in the copy of the token to the CPU, that wait is charged to whatever operator is waiting. In our table, aten::copy_ had the largest “self CPU time” for exactly this reason, not because type casts are slow. A CPU-side table tells you how many operations there are; only a GPU timeline tells you when the GPU was busy. That is what Nsight Systems is for.
On an NVIDIA GPU: Nsight Systems
On a CUDA machine, wrap the step in an NVTX range (a named time span that shows up in the trace) and record it with nsys. [not run by us]
# profile_step.py: the same step() as above, marked for Nsight Systems (CUDA only)
import torch
for i in range(20):
if i == 10: torch.cuda.cudart().cudaProfilerStart() # skip the warm-up steps
torch.cuda.nvtx.range_push(f"decode step {i}")
tok = step(); tok.cpu()
torch.cuda.nvtx.range_pop()
torch.cuda.cudart().cudaProfilerStop()
# shell: record CUDA API calls, kernels and NVTX ranges, only between Start and Stop
# nsys profile -t cuda,nvtx,osrt --capture-range=cudaProfilerApi -o step python profile_step.py
# nsys stats --report cuda_gpu_kern_sum step.nsys-rep # time per kernel name
# nsys stats --report cuda_api_sum step.nsys-rep # time in cudaLaunchKernel etc.
In the trace, read the GPU row of one step: the sum of kernel times against the length of the NVTX range. Their ratio is the GPU utilization of the step. If the CUDA API row is solid and the GPU row has gaps, the step is launch-bound, as ours is.
How to read Figure 1
Start with “eager” at batch 1 and an issue cost of 12 µs: the step has about 1,130 kernels, the CPU needs about 14 ms to issue them, the GPU is busy for about 17 ms, and the GPU row has gaps wherever several tiny kernels follow each other. Raise the issue cost to 25 µs: the step becomes launch-bound and grows to about 30 ms. Now press “fused add + RMSNorm”: about 450 kernels disappear, and the step gets shorter. Press “fused + one graph launch”: the CPU issues the whole step at once (this is what a CUDA graph does), the gaps close, and the step is set by the GPU alone. Switch to batch 16: the matmuls take about the same time (the weights are shared by all rows) and only attention grows, because it reads 16 caches, so the overhead stays a large part of the step.
The model of Figure 1 is simple: real issue costs differ by operator, and real GPU kernels overlap a little. But it explains the two numbers we measured: a step far above the bandwidth bound, and an issue time of the same size as the step. It also tells you what to do. Make fewer kernels (fusion, Part 2), or issue them all at once (a CUDA graph or torch.compile(mode="reduce-overhead"), from session 127). The second is not available on MPS, so in this lab we do the first.
Part 2: fuse the residual add and RMSNorm
Every layer of a pre-norm transformer does this twice: add the output of a sub-layer to the residual stream, then normalize the result for the next sub-layer (session 22):
- Read the symbols
- \(\mathbf{x}\)
- output of the attention or FFN sub-layer, one row of \(d = 1{,}536\) numbers per token
- \(\mathbf{r}, \mathbf{h}\)
- the residual stream before and after the add; \(\mathbf{h}\) is kept for the next residual add
- \(\mathbf{w}, \epsilon\)
- the learned scale of RMSNorm and a small constant (\(10^{-6}\) in Qwen2.5)
- \(\mathbf{y}\)
- the normalized input to the next sub-layer
In eager PyTorch this is nine kernels, because each line of the Hugging Face code is a separate operation: the add, a cast to FP32, pow, mean, the add of \(\epsilon\), rsqrt, a multiply, a cast back to BF16, and the multiply by \(\mathbf{w}\). Each kernel reads its inputs from memory and writes its output back. A fused kernel reads \(\mathbf{x}\) and \(\mathbf{r}\) once and writes \(\mathbf{h}\) and \(\mathbf{y}\) once.
- Read the symbols
- \(n\)
- tokens in the pass (the batch at decode, the prompt length at prefill)
- 42
- bytes per number for the eager kernels: add (read 2+2, write 2), cast to FP32 (read 2, write 4), pow (4+4), mean (read 4), multiply by the inverse root (4+4), cast to BF16 (4+2), multiply by \(\mathbf{w}\) (2+2); the tiny per-row kernels for \(\epsilon\) and the root are left out
- 8
- bytes per number fused: read \(\mathbf{x}\) and \(\mathbf{r}\), write \(\mathbf{h}\) and \(\mathbf{y}\), all BF16
So fusion should cut the bytes by \(42/8 = 5.25\) times. At decode with batch 1, the bytes are tiny (\(8 \times 1{,}536 = 12\) kB), so the time is the fixed cost of the kernels, and fusion turns nine fixed costs into one.
- input: \(\mathbf{x}, \mathbf{r}\) of shape \((n, d)\) in BF16; \(\mathbf{w}\) of size \(d\); \(\epsilon\)
- for each row \(i\) in parallel: // one GPU program (Triton) or threadgroup (Metal) per token
- load \(\mathbf{x}_i, \mathbf{r}_i\); \(\;\mathbf{h}_i \leftarrow \text{bf16}(\mathbf{x}_i + \mathbf{r}_i)\) // round like eager does
- store \(\mathbf{h}_i\) into \(\mathbf{r}_i\) // the new residual, in place
- \(\sigma \leftarrow \sum_j h_{ij}^2\) in FP32, reduced over the row; \(\;\rho \leftarrow 1/\sqrt{\sigma/d + \epsilon}\)
- \(\mathbf{y}_i \leftarrow \text{bf16}\big(\mathbf{w} \odot \text{bf16}(\mathbf{h}_i \rho)\big)\) // the same two roundings as eager
- store \(\mathbf{y}_i\)
The two roundings in line 6 are there on purpose. Hugging Face casts the normalized vector to BF16 before it multiplies by \(\mathbf{w}\). If the kernel skipped that rounding, its output would be slightly more accurate, but it would not match the eager model bit for bit, and you would lose the simplest test you have.
The Triton kernel
This is the kernel in Triton, for an NVIDIA GPU (the language is in session 86). One program handles one row; BLOCK is the row size rounded up to a power of two, so the whole row of 1,536 numbers sits in registers. [not run by us: it needs a CUDA GPU]
# fused_add_rmsnorm.py (Triton, CUDA)
import torch, triton, triton.language as tl
@triton.jit
def add_rmsnorm_kernel(X, R, W, Y, stride, D, eps, BLOCK: tl.constexpr):
row = tl.program_id(0)
cols = tl.arange(0, BLOCK)
m = cols < D
x = tl.load(X + row * stride + cols, mask=m, other=0.0).to(tl.float32)
r = tl.load(R + row * stride + cols, mask=m, other=0.0).to(tl.float32)
h = (x + r).to(tl.bfloat16) # residual add, rounded like eager
tl.store(R + row * stride + cols, h, mask=m) # new residual stream, in place
hf = h.to(tl.float32)
inv = tl.rsqrt(tl.sum(hf * hf, axis=0) / D + eps) # one reduction over the row
w = tl.load(W + cols, mask=m, other=0.0).to(tl.float32)
y = (hf * inv).to(tl.bfloat16).to(tl.float32) * w
tl.store(Y + row * stride + cols, y.to(tl.bfloat16), mask=m)
def add_rmsnorm(x, r, w, eps=1e-6):
"""h = x + r (written into r), y = RMSNorm(h) * w. x, r: (..., D) contiguous BF16."""
D = x.shape[-1]; x2, r2 = x.view(-1, D), r.view(-1, D)
y = torch.empty_like(x2)
BLOCK = triton.next_power_of_2(D)
add_rmsnorm_kernel[(x2.shape[0],)](x2, r2, w, y, x2.stride(0), D, eps,
BLOCK=BLOCK, num_warps=4 if BLOCK <= 2048 else 8)
return r, y.view_as(x)
On our machine we ran the same algorithm as a Metal kernel. The only differences are the language and the reduction: Metal adds up the row with simd_sum over groups of 32 threads and a small shared array, where Triton’s tl.sum does it for you. The full Metal source is in “The lab code” below.
Test, then measure
Test first. For 1 to 16,384 rows of random BF16 numbers, the fused kernel’s new residual \(\mathbf{h}\) was equal bit for bit to eager PyTorch. The output \(\mathbf{y}\) was equal bit for bit up to 256 rows; at 16,384 rows, 84 of 25.2 million numbers differed by one BF16 step (the two codes compute the reciprocal square root in slightly different ways). [Certain, our run] Then measure each version alone, with the timer of session 87 (warm-up, 50 calls, one synchronize, best of 5):
torch.compile, which fuses the same code automatically.How to read Figure 2
At \(n = 1\) (decode, batch 1) the bars of bytes are tiny, but eager takes 122 µs and fused 17 µs: the cost is nine kernel launches against one. Move to 256 rows: still mostly fixed cost. At 4,096 and 16,384 rows (a long prefill) the bytes matter: the fused kernel moves 201 MB in 843 µs, about 239 GB/s, which is our machine’s copy bandwidth. Eager is 8.0 times slower, more than the 5.25 times of the byte count, because its small reductions and casts do not reach full bandwidth either. torch.compile comes within 4% of the hand-written kernel at the largest size (874 against 843 µs), but is slower at small sizes.
Inside the model
A kernel that is 7 times faster alone does not make the model 7 times faster. To measure the effect on the whole step, we wrote the model’s forward pass ourselves (about 25 lines, using the Hugging Face layers) so that each residual add is fused with the norm that follows it. One step then has 56 fused kernels instead of \(56 \times 9 = 504\) small ones. First we checked in FP32 that our forward pass (with the eager norms) gives exactly the same final features as the Hugging Face forward pass [Certain, our run]; in BF16 they differ a little, see the “Watch out” in Part 3. Then we timed one decode step (context 512, Algorithm 1):
| Batch | eager step | fused step | gain | bound (132.1) |
|---|---|---|---|---|
| 1 | 29.9 ms | 25.7 ms | 14% | 13.2 ms |
| 4 | 39.5 ms | 35.8 ms | 10% | 13.5 ms |
| 16 | 76.5 ms | 71.7 ms | 6% | 14.3 ms |
[Certain, our run; best of 15 steps] Fusion removed 504 of the 1,324 operations of a step (the count fell to 820), the issue time fell from 12.0 to 8.7 ms, and the step from 29.9 to 25.7 ms at batch 1. The gain is real but far from the 7× of the kernel alone, because the nine small kernels were only a part of the step. In percent the gain shrinks with the batch, because the matmuls and attention take longer while the norm kernels stay small.
Part 3: speculative decoding with two kinds of drafts
Decode is memory-bound: a pass over 5 tokens reads the same 3.09 GB of weights as a pass over 1 token. Speculative decoding (session 128) uses this. A cheap drafter guesses \(k\) tokens; the target checks all of them in one pass. With greedy decoding the check is simple: accept the guesses up to the first one that differs from the target’s own argmax, then add one token from the target (the bonus token). So every round makes between 1 and \(k + 1\) tokens, and the output is exactly what plain greedy decoding of the target would make.
- input: the pending token \(g\) of each row (made by the target, not yet in its cache); draft length \(k\)
- drafter: \(d_1, \dots, d_k\) // \(k\) cheap passes, one token each
- target: one pass over \([g, d_1, \dots, d_k]\) \(\rightarrow\) greedy tokens \(G_0, \dots, G_k\) // \(G_j\) is the target's choice after \(d_j\)
- for each row: \(n \leftarrow\) the number of leading \(j\) with \(d_j = G_{j-1}\) // accepted guesses, \(0\) to \(k\)
- output \(d_1, \dots, d_n\) and the bonus \(G_n\) // \(n + 1\) new tokens
- mark the cache slots of \(d_{n+1}, \dots, d_k\) as invalid in the target’s and the drafter’s masks
- \(g \leftarrow G_n\)
Line 6 is how the lab handles a batch. Each row accepts a different number of guesses, so after one round the rows have different lengths. Instead of moving data, we keep a boolean validity mask over the KV cache: rejected slots stay in memory but are masked out of attention, and each token’s position (for RoPE) is the count of valid slots before it. The cache grows by \(k + 1\) slots per round for every row; that is a little wasted memory, but no copying. Production engines use paged caches instead (session 134); the idea of “keep, but do not attend” is the same.
Draft 1: a small model
Qwen2.5-0.5B-Instruct runs Algorithm 3 directly. It keeps its own KV cache. In each round its first pass re-reads the guesses that were accepted and the bonus token (so its cache matches the target’s text), and its other \(k - 1\) passes are single tokens. One draft step reads 0.99 GB, about a third of the target.
Draft 2: an EAGLE-style head
EAGLE (Li et al., 2024) does not use a separate small model. It trains a small head that predicts the target’s next feature: the last hidden vector before the output layer. Its input at position \(t\) is the target’s feature at \(t-1\) and the embedding of the token at \(t\), joined, projected from \(2d\) to \(d\) by a linear layer, and passed through one transformer decoder layer. The output goes through the target’s own output layer to give a token. [Certain] Features carry much more information than tokens, so a one-layer head can guess well.
- Read the symbols
- \(\mathbf{f}_t\)
- the target’s final hidden vector at position \(t\) (after the last norm); the target’s next-token logits are \(\mathbf{W}_{out}\mathbf{f}_t\)
- \(\mathbf{e}(x_t)\)
- the target’s embedding of token \(x_t\) (frozen, shared)
- \(\mathbf{W}_{fc}\)
- the trained projection from \(2d = 3{,}072\) to \(d = 1{,}536\)
- Layer
- one trained decoder layer with the target’s shape (attention with its own small KV cache, SwiGLU FFN)
- SmoothL1, CE
- a regression loss on the feature and a cross-entropy loss on the token; EAGLE weights them 1 and 0.1 [Certain]
When drafting, the head feeds its own predicted feature \(\hat{\mathbf{f}}\) back in, step after step. After verification, the target gives true features for every accepted token, and the head re-reads those in its first pass of the next round. To make the head robust to its own small errors, EAGLE adds uniform noise in \([-0.1, 0.1]\) to the input features during training [Certain]; we did the same.
- data: 2,048 GSM8K training questions with their reference answers, plus the target’s own greedy answers (160 tokens) to 1,024 other training questions, all in the target’s chat format // 3,072 sequences
- run the target once over each sequence with the same forward pass that will serve it; store \(\mathbf{f}_1, \dots, \mathbf{f}_T\) in BF16 // 683,011 tokens, about 2.1 GB
- initialize the head’s decoder layer and final norm from the target’s last layer and final norm // right feature scale from step 0
- for 4 epochs, batches of 8 sequences:
- inputs \([\mathbf{f}_{t-1} + \text{noise};\ \mathbf{e}(x_t)]\), targets \(\mathbf{f}_t\) and \(x_{t+1}\)
- loss (132.4), with the cross-entropy on 1,024 random positions per batch // the 151,936-wide output layer is the costly part
- AdamW, learning rate \(5 \times 10^{-4}\) with warm-up and cosine decay, gradient clipping at 0.5
Feature extraction and training took about 40 minutes on our (very busy) laptop. At the end the head’s top-1 token accuracy on training batches was about 0.75 (one step ahead, with true input features). [Certain, our run] This is a toy version of EAGLE: the paper trains on a fixed dialogue dataset (ShareGPT, 68,000 dialogues) for 1 to 2 days on 4 A100 GPUs with a learning rate of \(3 \times 10^{-5}\), and drafts a tree of guesses instead of one chain [Certain]. Our head sees two kinds of text: human-written GSM8K answers (short, with calculations written as <<48/2=24>>) and the target’s own answers, which are longer explanations and are what the head must guess at test time. The test questions below are not in either training set. Testing on other kinds of text (chat, code) would lower the acceptance rate [Likely].
The correctness test
Greedy speculation must give exactly the same tokens as plain greedy decoding. We tested this in FP32 (4 prompts, 64 tokens, draft model, \(k = 4\)): all 256 tokens were equal. [Certain, our run] In BF16 they are not always equal: at batch 1, 10 of the 16 rows of the draft-model run were identical to plain decoding over all 128 tokens, and 81% of all token positions matched (EAGLE-style head: 11 of 16 rows, 78%). [Certain, our run] The reason is the one from session 87: the verification pass works on 5 tokens at a time and plain decoding on 1, so the matrix multiplies add numbers in a different order, and BF16 can flip a near-tie. After one flip, the two texts go different ways. Test the logic in FP32; in BF16, compare logits with a tolerance.
Part 4: acceptance and tokens per second at batch 1, 4 and 16
Two numbers describe a drafter. The acceptance rate \(\hat\alpha\) is the fraction of all guesses that the target accepted. The accepted length \(\tau\) is the average number of tokens a row gets per target pass (accepted guesses plus the bonus).
- Read the symbols
- \(n\)
- accepted guesses of one row in one round (0 to \(k\))
- \(\tau(\alpha, k)\)
- the expected tokens per round from session 128, if every guess is accepted independently with probability \(\alpha\)
The speed-up is the accepted length times the time of one plain step, divided by the time of one round:
- Read the symbols
- \(t_1(b)\)
- one plain target step for \(b\) rows
- \(t_d(b)\)
- one drafter step; \(c = t_d / t_1\) is the cost ratio of session 128
- \(t_v(b, k+1)\)
- one target pass over \(k + 1\) tokens per row: close to \(t_1\) while the pass is memory-bound, larger once it is not
The test: the first 16 questions of the GSM8K test set, in Qwen’s chat format, 128 new tokens per question, greedy, \(k = 4\), all with the fused forward pass. Batch 1 runs the 16 questions one by one, batch 4 in 4 groups, batch 16 in one group. Each group runs all four methods back to back, twice, and we keep the faster run of each, because other jobs shared our GPU. Tokens per second counts all rows and includes the prefill. Rows keep going after an end-of-text token, so that every row makes exactly 128 tokens.
How to read Figure 3
Start with “tokens per second”. At batch 1 both drafters beat plain decoding: the draft model by 1.22× and the EAGLE-style head by 1.27×. At batch 4 and 16 both are slower than plain decoding (0.86× and 0.91× for the draft model, 0.85× and 0.81× for the head). Compare also the first two bars of each group: the fused norms help plain decoding at every batch, by 4% to 14% in this noisy run. Switch to “accepted length \(\tau\)”: \(\tau\) hardly depends on the batch, because acceptance is a property of the text and the drafter, not of the hardware. Switch to “speed-up”: the gain falls with the batch even though \(\tau\) stays the same. Figure 4 explains why.
What the numbers say
- The draft model guesses well but is not cheap. It reached \(\alpha = 0.78\) and \(\tau = 4.14\) out of 5 possible. But one draft step took 15.0 ms, 0.58 of a target step (25.9 ms), not the 0.32 that its bytes suggest (0.99 against 3.09 GB): at batch 1 both models are launch-bound, and the draft has 24 layers against 28. So the gain at batch 1 was only 1.22×. [Certain, our run]
- The EAGLE-style head guesses worse but is very cheap. \(\alpha = 0.34\) and \(\tau = 2.38\), but one head step took 3.1 ms (0.12 of a target step), so it was the faster drafter at batch 1: 1.27×. Its acceptance falls fast along the chain: \(\alpha = 0.63\) with one guess per round, 0.20 with eight. [Certain, our run]
- Acceptance hardly depends on the batch. \(\tau\) of the draft model was 4.14, 4.09 and 4.12 at batch 1, 4 and 16. Acceptance is a property of the text and the drafter. [Certain, our run]
- At batch 4 and 16, speculation lost. Plain decoding uses the spare compute by batching: from batch 1 to 16 it went from 15.5 to 308.7 tokens per second. Speculation then fights three costs: the 5-token verification pass costs 1.14× a 1-token step at batch 4 and 1.15× at batch 16; a group runs until its slowest row is done (the counted tokens per row and round fell from 4.09 to 3.61 for the draft model and from 2.31 to 1.80 for the head); and each round has fixed CPU bookkeeping [Likely].
- The numbers are noisy. Other jobs shared our GPU. Read the ratios inside one batch size, which were measured back to back, more than the absolute tokens per second.
How to read Figure 4
The slider starts at the acceptance rate we measured at \(k = 4\). At batch 1 the curve is highest at small \(k\) and falls as \(k\) grows: with these drafter costs, each extra guess costs more than the tokens it adds. Both squares lie above the curve at batch 1. For the draft model, the formula with \(\alpha = 0.78\) gives \(\tau = 3.26\), but we measured 4.14: acceptances are not independent; they come in runs, on easy stretches of text. Now switch to batch 4 and 16. The curve hardly moves, because our component times grow together (a 5-token pass costs 1.14 to 1.15 times a 1-token step), but the squares fall below 1. The readout shows the cost the model leaves out: a group runs until its slowest row is done, so each row gets fewer counted tokens per round than its own \(\tau\). Each round also has fixed bookkeeping on the CPU. Move the slider to see which \(\alpha\) would make longer drafts worth it.
Batch 1, draft model, \(k = 4\). We measured \(t_1 = 25.9\) ms, \(t_v = 26.1\) ms (5 tokens cost the same as 1) and \(t_d = 15.0\) ms, so \(c = 0.58\). With the measured \(\tau = 4.14\), (132.6) gives \(S = 4.14 \times 25.9 / (4 \times 15.0 + 26.1) = 1.24\). We measured 1.22×: close. For the EAGLE-style head, \(t_d = 3.1\) ms and \(\tau = 2.38\) give \(S = 1.60\), but we measured only 1.27×. The formula leaves out the costs of each round: the drafter’s first pass reads up to 5 tokens, and the CPU must read the accepted counts before it can start the next round, which empties the GPU queue [Likely]. With a cheap drafter these fixed costs are a large part of the round.
The lab code
Three files. lab.py holds the fused kernel and our forward pass with a validity mask; spec.py holds the drafters and the generation loop; the training script for the head follows the same pattern and is listed after them. We ran exactly this code on our machine. [Certain] It needs PyTorch 2.8 with MPS (for the Metal kernel; on CUDA, replace addnorm_fused by the Triton add_rmsnorm above) and transformers 4.57; other versions of transformers may change the layer call signatures.
# lab.py: Qwen2 forward pass with fused add + RMSNorm and a KV-validity mask (PyTorch 2.8, MPS)
import time, torch
from transformers import AutoModelForCausalLM, DynamicCache
dev = "mps"; sync = torch.mps.synchronize
SRC = r'''
#include <metal_stdlib>
using namespace metal;
kernel void add_rmsnorm(device const bfloat* x [[buffer(0)]], device bfloat* r [[buffer(1)]],
device const bfloat* w [[buffer(2)]], device bfloat* y [[buffer(3)]],
constant uint& D [[buffer(4)]], constant float& eps [[buffer(5)]],
uint row [[threadgroup_position_in_grid]], uint tid [[thread_position_in_threadgroup]],
uint nt [[threads_per_threadgroup]], uint lane [[thread_index_in_simdgroup]],
uint sg [[simdgroup_index_in_threadgroup]]) {
threadgroup float part[32];
float acc = 0.0f;
device const bfloat* xr = x + row * D; device bfloat* rr = r + row * D;
for (uint i = tid; i < D; i += nt) {
bfloat h = bfloat(float(xr[i]) + float(rr[i])); // residual add, rounded like eager
rr[i] = h; float hf = float(h); acc += hf * hf; // new residual, in place
}
acc = simd_sum(acc); // reduce 32 lanes
if (lane == 0) part[sg] = acc;
threadgroup_barrier(mem_flags::mem_threadgroup);
if (sg == 0) { float v = lane < (nt + 31) / 32 ? part[lane] : 0.0f; v = simd_sum(v); if (lane == 0) part[0] = v; }
threadgroup_barrier(mem_flags::mem_threadgroup);
float inv = rsqrt(part[0] / float(D) + eps);
for (uint i = tid; i < D; i += nt) y[row * D + i] = bfloat(float(bfloat(float(rr[i]) * inv)) * float(w[i]));
}
'''
lib = torch.mps.compile_shader(SRC)
def addnorm_eager(x, r, w, eps): # what Hugging Face does: 9 kernels
h = x + r
hf = h.to(torch.float32)
hf = hf * torch.rsqrt(hf.pow(2).mean(-1, keepdim=True) + eps)
return h, w * hf.to(x.dtype)
def addnorm_fused(x, r, w, eps): # 1 kernel: h is written into r
D = x.shape[-1]; x2 = x.reshape(-1, D).contiguous(); r2 = r.reshape(-1, D).contiguous()
y = torch.empty_like(x2)
lib.add_rmsnorm(x2, r2, w, y, D, eps, threads=x2.shape[0] * 256, group_size=256)
return r2.view(x.shape), y.view(x.shape)
def load(name):
return AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16).to(dev).eval()
def mask4d(am, T): # am: (B, Ltot) bool validity; the last T columns are new
Lt = am.shape[1]; q = torch.arange(Lt - T, Lt, device=am.device); k = torch.arange(Lt, device=am.device)
m = (k[None, :] <= q[:, None])[None] & am[:, None, :] # causal and valid
m = m | (k[None, None, :] == q[None, :, None]) # a slot always sees itself
return m[:, None] # (B, 1, T, Ltot), True = attend
def positions(am, T): # RoPE position = number of valid slots before
return (am.long().cumsum(-1) - 1).clamp(min=0)[:, -T:]
@torch.no_grad()
def run_layers(M, layers, x, past, am, fused, final_w):
T = x.shape[1]; L0 = past.get_seq_length()
cp = torch.arange(L0, L0 + T, device=x.device); mask = mask4d(am, T)
pe = M.rotary_emb(x, positions(am, T))
an = addnorm_fused if fused else addnorm_eager; eps = M.config.rms_norm_eps
r = x; h = layers[0].input_layernorm(x)
for i, lyr in enumerate(layers):
a, _ = lyr.self_attn(hidden_states=h, attention_mask=mask, past_key_values=past,
cache_position=cp, position_embeddings=pe)
r, h = an(a, r, lyr.post_attention_layernorm.weight, eps) # add + norm
f = lyr.mlp(h)
nxt = layers[i + 1].input_layernorm.weight if i + 1 < len(layers) else final_w
r, h = an(f, r, nxt, eps) # add + next norm
return h # final normalized hidden states (B, T, d)
def fwd(m, ids, past, am, fused):
return run_layers(m.model, m.model.layers, m.model.embed_tokens(ids), past, am, fused, m.model.norm.weight)
def bench(f, warmup=3, reps=10): # best and median wall time, GPU queue drained
for _ in range(warmup): f()
sync(); ts = []
for _ in range(reps):
t = time.perf_counter(); f(); sync(); ts.append(time.perf_counter() - t)
ts.sort(); return ts[0], ts[len(ts) // 2]
# spec.py: drafters and greedy (speculative) generation with per-row acceptance
import copy, time, torch, torch.nn as nn
from transformers.models.qwen2.modeling_qwen2 import Qwen2DecoderLayer, Qwen2RMSNorm, Qwen2RotaryEmbedding
from lab import *
class Head(nn.Module): # EAGLE-style: [f_(t-1); e(x_t)] -> fc -> one decoder layer -> f_t
def __init__(s, cfg):
super().__init__(); c = copy.deepcopy(cfg); c.num_hidden_layers = 1; s.config = c
s.fc = nn.Linear(2 * c.hidden_size, c.hidden_size, bias=False)
s.layers = nn.ModuleList([Qwen2DecoderLayer(c, 0)])
s.norm = Qwen2RMSNorm(c.hidden_size, c.rms_norm_eps); s.rotary_emb = Qwen2RotaryEmbedding(c)
def inp(s, feat, emb): return s.fc(torch.cat([feat, emb], -1))
class Drafter:
"""kind = "model" (a small LM) or "eagle" (a Head). Keeps its own KV cache and validity mask."""
def __init__(s, kind, net, target, fused): s.kind, s.net, s.t, s.fused = kind, net, target, fused
def _run(s, X, F, v):
s.am = torch.cat([s.am, v], 1)
if s.kind == "model":
h = fwd(s.net, X, s.past, s.am, s.fused); return h, s.net.lm_head(h)
x = s.net.inp(F, s.t.model.embed_tokens(X))
h = run_layers(s.net, s.net.layers, x, s.past, s.am, s.fused, s.net.norm.weight)
return h, s.t.lm_head(h) # the target's own output layer
def start(s, ids, am, g, H): # prompt (+ target features H) and first token g
s.past = DynamicCache(); s.am = am[:, :0]; s.steps = []
one = am[:, :1] | True
if s.kind == "model": X, v, F = torch.cat([ids, g[:, None]], 1), torch.cat([am, one], 1), None
else: X, v, F = torch.cat([ids[:, 1:], g[:, None]], 1), torch.cat([am[:, 1:], one], 1), H
return s.first(X, F, v)
def first(s, X, F, v): # re-read accepted tokens + bonus (width k+1)
if s.steps: s.am[:, s.steps] = False; s.steps = [] # forget last round's guesses
h, lg = s._run(X, F, v)
idx = (v.long() * torch.arange(1, v.shape[1] + 1, device=v.device)).argmax(1) # last valid slot
ar = torch.arange(X.shape[0], device=X.device)
return lg[ar, idx].argmax(-1), h[ar, idx][:, None]
def step(s, tok, fh): # one more guess
s.steps.append(s.am.shape[1])
h, lg = s._run(tok[:, None], fh, torch.ones_like(tok[:, None], dtype=torch.bool))
return lg[:, -1].argmax(-1), h[:, -1:]
def left_pad(seqs, pad):
P = max(map(len, seqs))
ids = torch.tensor([[pad] * (P - len(x)) + x for x in seqs], device=dev)
am = torch.tensor([[False] * (P - len(x)) + [True] * len(x) for x in seqs], device=dev)
return ids, am
@torch.no_grad()
def generate(target, seqs, N, drafter=None, k=4, fused=True, pad=151643):
ids, am = left_pad(seqs, pad); B = ids.shape[0]; sync(); t0 = time.perf_counter()
past = DynamicCache(); H = fwd(target, ids, past, am, fused)
g = target.lm_head(H[:, -1]).argmax(-1); out = [[x] for x in g.tolist()]; acc = []
if drafter is None: # plain greedy decoding
while min(map(len, out)) < N:
am = torch.cat([am, am[:, :1] | True], 1)
g = target.lm_head(fwd(target, g[:, None], past, am, fused)[:, -1]).argmax(-1)
for b, x in enumerate(g.tolist()): out[b].append(x)
sync(); return torch.tensor([o[:N] for o in out]), dict(time=time.perf_counter() - t0)
d, fh = drafter.start(ids, am, g, H)
while min(map(len, out)) < N:
D = [d]
for _ in range(k - 1):
d, fh = drafter.step(d, fh); D.append(d)
D = torch.stack(D, 1) # (B, k) guesses
V = torch.cat([g[:, None], D], 1); am = torch.cat([am, V > -1], 1)
H = fwd(target, V, past, am, fused); G = target.lm_head(H).argmax(-1) # one verify pass
n = (D == G[:, :k]).long().cumprod(1).sum(1) # accepted guesses per row
bonus = G.gather(1, n[:, None])[:, 0]
j = torch.arange(k + 1, device=dev)
am[:, -k - 1:] = j[None] <= n[:, None] # rejected slots: masked out
for b, (nb, Db, bb) in enumerate(zip(n.tolist(), D.tolist(), bonus.tolist())):
if len(out[b]) < N: acc.append(nb)
out[b] += Db[:nb] + [bb]
X = torch.cat([D, D[:, :1]], 1).scatter(1, n[:, None], bonus[:, None])
d, fh = drafter.first(X, H, j[None] <= n[:, None]); g = bonus
sync(); a = torch.tensor(acc, dtype=torch.float)
return torch.tensor([o[:N] for o in out]), dict(time=time.perf_counter() - t0,
alpha=(a.sum() / (k * len(a))).item(), tau=(a + 1).mean().item())
if __name__ == "__main__":
import json
from transformers import AutoTokenizer
T = "Qwen/Qwen2.5-1.5B-Instruct"; tok = AutoTokenizer.from_pretrained(T)
tg, dm = load(T), load("Qwen/Qwen2.5-0.5B-Instruct")
qs = ["Janet has 3 apples and buys 5 more. She eats 2. How many apples does she have?",
"A train travels 60 km per hour for 2.5 hours. How far does it go?"]
seqs = [tok.apply_chat_template([{"role": "user", "content": q}], add_generation_prompt=True) for q in qs]
a, s0 = generate(tg, seqs, 128) # plain greedy
b, s1 = generate(tg, seqs, 128, Drafter("model", dm, tg, True), k=4)
print(f"plain {2 * 128 / s0['time']:.0f} tok/s; draft model {2 * 128 / s1['time']:.0f} tok/s, "
f"alpha {s1['alpha']:.2f}, tau {s1['tau']:.2f}, same tokens {(a == b).float().mean():.3f}")
The training script for the head (Algorithm 4) uses the same Head. Its setup and one batch:
# train_eagle.py (setup): features from our own forward pass, head started from the target's last layer
feats = [fwd(tg, torch.tensor([q], device=dev), DynamicCache(), torch.ones(1, len(q), dtype=torch.bool, device=dev), True)[0].cpu()
for q in seqs] # not tg.model(...): see "Watch out" in Part 3
head = Head(tg.config).to(dev, torch.float32)
for p in head.parameters():
if p.dim() > 1: torch.nn.init.normal_(p, std=0.02)
head.layers[0].load_state_dict(tg.model.layers[-1].state_dict()); head.norm.weight.data.copy_(tg.model.norm.weight.float())
# train_eagle.py (core, one batch): X tokens x_t, Fin features f_(t-1), Ft features f_t, Y next tokens x_(t+1)
Fin = Fin + (torch.rand_like(Fin) - 0.5) * 0.2 # uniform noise in [-0.1, 0.1]
x = head.inp(Fin, E[X].float()) # E: the target's embedding matrix
pos = torch.arange(Lm, device=dev)[None].expand(BS, -1)
mask = torch.ones(Lm, Lm, device=dev, dtype=torch.bool).tril()[None, None]
h = head.norm(head.layers[0](x, attention_mask=mask, position_embeddings=head.rotary_emb(x, pos), position_ids=pos))
lreg = (F.smooth_l1_loss(h, Ft, reduction="none").mean(-1) * Mk).sum() / Mk.sum() # Mk: real tokens
sel = (Y >= 0).view(-1).nonzero()[:, 0]
sel = sel[torch.randperm(len(sel), generator=g)[:1024].to(dev)] # CE on 1,024 positions
lcls = F.cross_entropy((h.reshape(-1, h.shape[-1])[sel].bfloat16() @ W.T).float(), Y.view(-1)[sel])
loss = lreg + 0.1 * lcls # formula (132.4)
opt.zero_grad(); loss.backward(); torch.nn.utils.clip_grad_norm_(head.parameters(), 0.5); opt.step(); sched.step()
Things to try next: on a CUDA GPU, wrap the decode step in a CUDA graph and see how much of the gap to the bound disappears; fuse RoPE into the attention call; draft a small tree of guesses instead of one chain (EAGLE’s default) and measure how much \(\tau\) grows; test the head on non-math prompts and watch \(\hat\alpha\) drop.
Where this is used
| Where | What from this lab | What it decides |
|---|---|---|
| Serving engines (vLLM, SGLang, TensorRT-LLM) | Fused norm kernels, CUDA graphs for decode, speculative decoding with draft models and EAGLE heads [Certain] | Which features to turn on for a given model, batch size and traffic |
| Performance work on a new model | Algorithm 1: step time, issue time, operation counts; the ratio to the bound (132.1) | Whether to fix launches, kernels or bytes first |
| Kernel libraries (for example vLLM’s custom ops, FlashInfer) [Likely] | Fused add + RMSNorm and similar kernels, tested against eager code | How much a kernel saves alone and in a full step |
| Interactive chat at batch 1 (laptops, phones, a single user) | Speculative decoding with a small drafter | The largest latency gain is at low batch, where the GPU has spare compute |
| High-throughput batch serving | Acceptance and speed-up measured at your real batch size (Figure 4) | Whether speculation still pays, and with which \(k\); often a smaller \(k\) at larger batch |
| Long-output tasks: code, math, video captioning and dense video description | EAGLE-style heads trained on the target’s own outputs for that domain | Predictable outputs (code, structured captions, timestamps in key-moment lists) give high acceptance |
| Vision-language models | The same drafters over the language model; the image or video tokens are only in the prefill | Decode speed of answers in video question answering and Video-RAG pipelines |
Common mistakes
- Timing a decode step that changes the cache. Each call adds a token, so you measure longer and longer contexts. Crop the cache back before every timed call, or use a static cache.
- Reading a CPU-only profile as GPU time. The operator that synchronizes collects the waiting time. Count operations with the profiler; time the GPU with a trace (Nsight Systems) or with events.
- Measuring a fused kernel only alone. 7× faster alone was 14% faster in the step. Always measure the end-to-end step too.
- Skipping the bit-for-bit test. Write the fused kernel to reproduce eager’s roundings first; only then make it “more accurate”, and switch to a tolerance test.
- Testing speculative decoding for equal tokens in BF16. Near-ties flip. Test exact equality in FP32; in BF16, compare logits with a tolerance.
- Using “the minimum acceptance of the batch” for every row. Correct, but at batch 16 the minimum is often 0, and speculation looks useless. Accept per row and mask rejected cache slots.
- Reporting acceptance from one prompt type. Our head is trained and tested on math answers. Acceptance depends on the text: measure it on traffic like yours.
- Assuming the batch-1 speed-up holds at batch 16. The verification pass over \(b(k+1)\) tokens stops being free as the batch grows (Figure 4). Measure the speed-up at the batch you serve.
Interview questions
Your decode step takes twice the bandwidth bound. How do you find out why?core
.item().Why can a CPU-side profiler table mislead you about GPU time?basic
aten::copy_) had the most self CPU time for this reason. Use the table to count calls, and use a GPU timeline (Nsight Systems, or the profiler with CUDA activities) to see kernel durations and gaps.How many bytes does an unfused residual add + RMSNorm move, and how many does a fused kernel move?core
Why did your fused kernel make the whole decode step only a little faster?core
How do you test that a fused kernel is correct?basic
Speculative decoding with greedy acceptance: is the output exactly the same as plain greedy decoding?basic
How does an EAGLE head differ from a small draft model, and why can it be better?core
Why does the speed-up of speculative decoding shrink at larger batch, even when the acceptance rate stays the same?advanced
How do you handle different numbers of accepted tokens per row in a batch?advanced
How would you choose the draft length \(k\)?core
Your offline benchmark shows a 2× speed-up from speculation, but production shows 1.1×. What could explain it?advanced
Exercises
- Use (132.1) to compute the decode bound of Qwen2.5-0.5B (0.99 GB) on our machine (233 GB/s). If one draft step took 3 times that bound, what cost ratio \(c\) would it have against a target step of 25.7 ms?
- With (132.3), how many bytes do the eager and the fused add + RMSNorm move in one decode step at batch 16 (56 fused pairs, \(d = 1{,}536\))? Compare with the 3.09 GB of weights. What does this tell you about where the fusion gain at decode comes from?
- With \(\alpha = 0.7\), compute \(\tau\) from (132.5) for \(k = 2, 4, 8\). How many extra tokens per round does going from 4 to 8 guesses buy?
- At batch 1, suppose \(t_1 = 27\) ms, \(t_v = 29\) ms for 5 tokens, and a drafter step costs 9 ms. With \(\tau = 3.5\) and \(k = 4\), what speed-up does (132.6) predict? What if the drafter step cost 3 ms?
- The batch-16 verification pass handles \(16 \times 5 = 80\) tokens. Using the ridge of our machine (about 125 FLOP per byte, from session 87) and the fact that a pass of \(n\) tokens has intensity about \(n\) FLOP per byte in BF16, is this pass memory-bound or compute-bound in theory? Why can the measured cost still grow?
- Run the lab on your own GPU. Report the eager and fused step times at batch 1, \(\hat\alpha\) and \(\tau\) for the draft model, and the speed-up at batch 1 and 16. On a CUDA GPU, add a CUDA graph and report the new step time.
Answerscheck
Key takeaways
- Profile before you optimize: time one decode step at a fixed context (crop the cache), measure its issue time, and count its operations. Ours had about 1,324 operations and took 2.3× the bandwidth bound at batch 1.
- A CPU-side profiler counts operations; only a GPU timeline (Nsight Systems on NVIDIA) shows when the GPU is busy. Synchronizing operations collect the waiting time.
- Eager residual add + RMSNorm is 9 kernels and about 42 bytes per element; fused, it is 1 kernel and 8 bytes. Alone it was 7 to 8× faster; in the whole step it saved 14% at batch 1.
- Write a fused kernel to match eager’s roundings, test it bit for bit, measure it alone, then measure the step.
- Greedy speculative decoding gives the target’s exact greedy output (in FP32). Per-row acceptance with a validity mask over the KV cache makes it work in a batch.
- On our machine speculation paid only at batch 1: 1.22× with a 0.5B draft model and 1.27× with a small EAGLE-style head (cheaper steps beat higher acceptance). At batch 4 and 16 it was slower than plain decoding. Train a head on the features your serving code produces, and measure at the batch you serve.
Key formulas
References for this session
- OpenAI and community (docs). Triton language documentation
- Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
- Li et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- Chen et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling
- Tillet et al. (2019). Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations
- Zhang & Sennrich (2019). Root Mean Square Layer Normalization
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Cai et al. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Qwen Team (2024). Qwen2.5 Technical Report
- Li, Wei, Zhang, Zhang (2024). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
- Li et al. (2025). EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- NVIDIA (docs). Nsight Systems User Guide
- PyTorch (docs). torch.profiler (documentation)
- PyTorch (docs). torch.mps.compile_shader (documentation)