LLM & VLM courseBlog
130Track 5: LLM inference and serving · session 53 of 65

Offloading: FlexGen, llama.cpp, PowerInfer

When a model does not fit in GPU memory, part of it can live in CPU memory or on an SSD. This session shows, with simple bandwidth math, why that is slow and how three systems make it work: FlexGen plans where every tensor lives and shares each transfer over a huge batch, llama.cpp splits the layers and computes the CPU part in place, and PowerInfer keeps the often-used (“hot”) neurons on the GPU and the rarely-used (“cold”) ones on the CPU.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: The KV cache (session 80), Why inference is slow and the roofline (session 82), Quantization math (session 103)

What you will learn

  • The memory hierarchy (GPU memory, PCIe, DRAM, SSD) and the bandwidth lower bound on time per token
  • Why computing offloaded layers on the CPU beats streaming them over PCIe at batch size 1, and why the answer flips for large batches
  • llama.cpp’s layer split with -ngl, KV-cache placement and expert offloading for MoE models, with a cost model for choosing the split
  • FlexGen: the zig-zag block schedule, its cost model and linear program for tensor placement, CPU attention, and 4-bit group-wise compression
  • PowerInfer: activation sparsity, the power law of neuron activations, hot/cold placement, predictors and the hybrid CPU-GPU FFN
  • LLM in a flash: windowing and row-column bundling for models larger than DRAM
  • What the papers measured, and how to read offloading benchmarks

Why this matters

Llama 3.1 70B has 70.6 billion weights. In BF16 that is 141 GB. Even in 4-bit form it is about 40 GB. A good consumer GPU, the RTX 4090, has 24 GB. The model does not fit. What can you do if you have one such GPU, a computer with a lot of normal memory, and a fast SSD?

There are three answers. You can make the model smaller (quantization, session 103, and pruning, session 115). You can buy more GPUs and split the model (Track 5 covers this later). Or you can offload: keep part of the model, or part of the KV cache, in CPU memory (DRAM) or on the SSD, and use it from there. This session is about the third answer. The efficient-inference survey of Zhou et al. lists offloading as one of the techniques of the inference engine and names the systems we study: FlexGen, llama.cpp, PowerInfer, and FastDecode. [Certain]

Offloading is not one idea. It is a set of choices, and the right choice depends on the goal:

  • Throughput, latency does not matter. You want to label 10 million documents overnight, or run a benchmark, on one GPU. FlexGen (Sheng et al., 2023) plans where every tensor lives (GPU, CPU or disk) and in which order to compute, so that every weight it moves over the bus is used by hundreds of sequences. On one 16 GB GPU it ran OPT-175B at about 1 token per second, using an effective batch of 144 sequences. [Certain]
  • Latency for one user. You chat with a model on your own PC. llama.cpp puts some layers on the GPU and runs the other layers on the CPU, where those weights already live. One number, -ngl (the number of layers on the GPU), controls the split.
  • Latency, and the model has sparse activations. PowerInfer (Song et al., 2023) notices that a few FFN neurons fire for almost every token (“hot”) and most fire rarely (“cold”). It keeps the hot ones on the GPU and computes the cold ones on the CPU, and only when a predictor says they will fire. It reports up to 11.69× faster generation than llama.cpp on one RTX 4090. [Certain]
  • Even CPU memory is too small. On a phone or a laptop, LLM in a flash (Alizadeh et al., 2023) keeps the weights on flash storage and loads only the rows that will be used. It runs models up to twice the size of the available DRAM. [Certain]

In session 129 we made each pass over the weights produce more tokens. Offloading has the same enemy, the cost of moving bytes, but at a much worse ratio: the bus between CPU and GPU is about 40 times slower than the GPU’s own memory. Everything in this session follows from that one fact.

Where bytes live and how fast they move

A computer has a memory hierarchy: several kinds of memory, from small and fast to large and slow. For LLM inference, four levels matter:

  • GPU memory (HBM on data-center GPUs, GDDR on consumer GPUs). The GPU’s math units read from it. An RTX 4090 has 24 GB at about 1,008 GB/s. [Certain]
  • The bus between CPU and GPU, usually PCIe. PCIe 4.0 with 16 lanes is about 32 GB/s in each direction on paper; in practice about 25 GB/s. [Likely]
  • CPU memory (DRAM). Much larger (64 to 512 GB) and cheap, but slower. The desktop in the PowerInfer paper (Intel i9-13900K) has 192 GB at 67.2 GB/s. [Certain]
  • SSD or flash. Terabytes, but slow: about 2 GB/s for reads on the FlexGen cloud machine, about 6 GB/s for large reads on the MacBook M1 Max in LLM in a flash. [Certain]

First, how much memory does a model need? For weights alone:

\[ M_{\text{weights}} = \frac{N \cdot b}{8} \ \text{bytes} \tag{130.1} \]
Read the symbols
\(N\)
number of weights (parameters)
\(b\)
bits per weight: 16 for BF16, 8.5 for llama.cpp’s Q8_0 format, 4.5 for its Q4_0 format (a block of 32 four-bit weights plus one 16-bit scale is 18 bytes, and \(18 \cdot 8 / 32 = 4.5\))

Add the KV cache on top. For Llama 3.1 70B (80 layers, 8 key-value heads of size 128) it is \(2 \times 80 \times 8 \times 128 \times 2 = 327{,}680\) bytes, or 320 KiB, per token in BF16 (session 80). A 4,096-token context needs 1.34 GB.

Second, how fast can we go? In the decode phase, each new token reads every weight once (session 82). At batch size 1 the time per token is set by how many bytes we read and from where:

\[ t_{\text{token}} \;\ge\; \sum_{\text{levels } \ell} \frac{\text{bytes read from } \ell}{\text{bandwidth of } \ell} \tag{130.2} \]

This is a lower bound: it ignores compute and assumes that the reads from different levels happen one after the other, which is what happens when layer \(i+1\) needs the output of layer \(i\). Figure 1 puts numbers on it.

Figure 1. The memory hierarchy of two machines. Box width grows with capacity (log scale). The time on the right is how long it takes to read 39.1 GB once from that level: the bytes that a 4.5-bit Llama 3.1 70B reads for every generated token. The desktop SSD speed (7 GB/s, a fast PCIe 4.0 NVMe drive) and the 25 GB/s effective PCIe speed are our assumptions; the paper does not give the DRAM bandwidth of the T4 machine.
How to read Figure 1

Compare the times, not the capacities. On the PC, one pass over the model from GPU memory takes about 39 ms; from DRAM, 0.58 s; over PCIe, 1.56 s; from the SSD, 5.6 s. Notice that DRAM is faster than PCIe. If the weights are already in DRAM, a CPU that reads them directly is faster than copying them to the GPU. That is the key idea behind llama.cpp’s split. On the T4 machine of FlexGen the gaps are even larger: PCIe is 12 GB/s and the SSD 2 GB/s.

Three ways to run a layer that is not on the GPU

Suppose layer \(i\)’s weights live in DRAM. There are three ways to use them:

  1. Stream the weights to the GPU, compute there, throw them away. This is what Hugging Face Accelerate (device_map="auto") and DeepSpeed ZeRO-Inference do. [Likely] The cost is the PCIe transfer of all offloaded weights, for every forward pass.
  2. Compute on the CPU, where the weights are. Only the activation vector (a few kilobytes per token) crosses the bus. The cost is reading the weights from DRAM at DRAM speed, plus the CPU’s slower arithmetic. This is llama.cpp.
  3. Split inside the layer: the most-used part on the GPU, the rest on the CPU, and skip what is not needed. This is PowerInfer.

Let \(W_g\) be the bytes of the weights on the GPU and \(W_c\) the bytes in DRAM. With equation 130.2, the time per token at batch size 1 is:

\[ t_{\text{stream}} = \frac{W_g}{B_{\text{gpu}}} + \frac{W_c}{B_{\text{pcie}}}, \qquad t_{\text{cpu}} = \frac{W_g}{B_{\text{gpu}}} + \frac{W_c}{B_{\text{dram}}} \tag{130.3} \]
Read the symbols
\(B_{\text{gpu}}\)
GPU memory bandwidth (1,008 GB/s on an RTX 4090)
\(B_{\text{pcie}}\)
effective CPU-to-GPU bandwidth (about 25 GB/s for PCIe 4.0 ×16)
\(B_{\text{dram}}\)
DRAM bandwidth the CPU can use (67.2 GB/s on the PowerInfer desktop)

When \(B_{\text{dram}} \gt B_{\text{pcie}}\), which is true on most desktops, computing on the CPU wins at batch size 1. The CPU’s arithmetic is slow, but at batch size 1 decode needs only about 2 FLOPs per weight, so the CPU is limited by memory bandwidth too, not by arithmetic. [Likely]

The picture changes with a batch. If \(B\) sequences share one pass over the weights, streaming pays the PCIe cost once for all of them, while the CPU must do \(B\) times more arithmetic. The weight-transfer cost per generated token is

\[ \frac{t_{\text{transfer}}}{\text{token}} = \frac{W_c}{B \cdot B_{\text{pcie}}} \tag{130.4} \]

With \(B = 144\), the PCIe cost per token is 144 times smaller. This is the whole idea of FlexGen: if you do not care about latency, make the batch huge, and the slow bus stops mattering.

Worked example: Llama 3.1 70B on one RTX 4090

At 4.5 bits per weight the model file is 39.7 GB. Each token reads 39.1 GB: the 80 layers (0.481 GB each) and the output head (0.59 GB); the input embedding table is only looked up, one row per token.

Put 45 layers on the GPU: 21.66 GB of weights, plus 0.75 GB of KV cache for 4,096 tokens in those layers, plus about 1.5 GB of working buffers (our assumption) gives 23.9 GB. That fits in 24 GB; 46 layers would not. The other 35 layers and the head, 17.44 GB, stay in DRAM.

Compute on the CPU: \(t = 21.66/1008 + 17.44/67.2 = 0.0215 + 0.2595 = 0.281\) s, so at most 3.56 tokens/s. Stream over PCIe instead: \(t = 0.0215 + 17.44/25 = 0.719\) s, at most 1.39 tokens/s. All on the CPU: \(39.1/67.2 = 0.58\) s, 1.72 tokens/s. A GPU with 48 GB that holds everything: \(39.1/1008 = 38.8\) ms, 25.8 tokens/s.

Two lessons. First, computing where the weights live is 2.6× faster than streaming them. Second, even the best split is 7× slower than a GPU that holds the whole model, because the 17.44 GB on the CPU side dominate. These are bandwidth-only upper bounds, not measurements.

llama.cpp: split the layers between CPU and GPU

llama.cpp is a C/C++ inference engine built on the ggml tensor library. It runs on CPUs (x86 and ARM), Apple GPUs (Metal), NVIDIA GPUs (CUDA), and others, and it stores models in the GGUF file format with many quantization types (Q4_0, Q4_K, Q8_0, ...). [Certain] It is the engine under several popular local-model apps. [Likely]

Its offloading rule is simple: a transformer is a stack of identical layers, so cut the stack. The option -ngl N (--n-gpu-layers) sets the “max. number of layers to store in VRAM”. [Certain] The first layers run on the CPU, the last \(N\) on the GPU. Between the two parts, only the hidden-state vector crosses the bus: 8,192 numbers per token for a 70B model, about 16 KB in FP16. That is tiny next to the 0.48 GB of weights per layer.

Algorithm 1: one decode step with a layer split (llama.cpp style)
  1. input: token id, layers \(1..L\), the last \(n_{\text{gl}}\) of them on the GPU, KV cache of each layer stored next to its weights
  2. \(\mathbf{h} \leftarrow\) embedding row of the token // CPU, one row lookup
  3. for \(i = 1\) to \(L - n_{\text{gl}}\): \(\mathbf{h} \leftarrow \text{layer}_i(\mathbf{h})\) on the CPU, reading weights and KV from DRAM // many threads, SIMD
  4. copy \(\mathbf{h}\) to the GPU // a few KB over PCIe
  5. for \(i = L - n_{\text{gl}} + 1\) to \(L\): \(\mathbf{h} \leftarrow \text{layer}_i(\mathbf{h})\) on the GPU
  6. logits \(\leftarrow\) output head (GPU if offloaded too, else CPU); sample the next token
  7. time \(\approx\) GPU bytes \(/ B_{\text{gpu}}\) + CPU bytes \(/ B_{\text{dram}}\) // equation 130.3; the parts run one after the other

Figure 2 lets you choose the split. The model is Llama 3.1 70B in Q4_0 (4.5 bits per weight) or Q8_0 (8.5 bits), on a 24 GB GPU with the bandwidths of Figure 1. We count the output head as one extra “layer”: with an 80-layer model, \(N = 81\) means everything is on the GPU (an 8B model with 32 layers would need \(N = 33\)). [Likely]

on the GPUin DRAM (CPU)KV cache on the GPUworking buffers (1.5 GB, assumed)
Figure 2. The llama.cpp layer split for Llama 3.1 70B on a 24 GB GPU. Top: the 80 layers and the output head, colored by where they live. Middle: GPU memory use. Bottom: time per token from equation 130.3, split into GPU time and CPU (or PCIe) time.
How to read Figure 2

Move the slider from 0 to 81 and watch the time bar. Each layer you move to the GPU saves its DRAM time (7.2 ms at 4.5 bits) and costs only 0.48 ms of GPU time, so the speed grows steadily, then jumps when the last layers and the head arrive. Past about 45 layers the memory bar turns red: out of memory. A longer context uses more KV memory on the GPU, so fewer layers fit. Switch to “Stream CPU layers to GPU” to see why llama.cpp does not do that at batch size 1. Switch to Q8_0: almost twice the bytes, so about half the speed, and fewer layers fit.

Details that matter in practice

  • Memory-mapped files. By default llama.cpp maps the GGUF file into memory (mmap): the operating system loads pages of the file when they are first touched, and can share them between processes. Startup is fast, and the CPU layers are read straight from the page cache. [Likely] If the file does not fit in DRAM, pages are read from the SSD again and again, and speed falls to SSD speed.
  • Where the KV cache goes. The KV cache of each layer is kept on the same device as the layer. The option -nkvo (--no-kv-offload) keeps all of it in DRAM to save GPU memory. [Certain]
  • Several GPUs. --tensor-split 3,1 gives the “fraction of the model to offload to each GPU”; by default the split is by layers. [Certain]
  • Mixture-of-Experts models. In an MoE layer only a few experts run per token (session 95). So the experts are a good thing to leave on the CPU, while attention and the shared parts go to the GPU. llama.cpp has options for exactly this: --cpu-moe keeps “all Mixture of Experts (MoE) weights in the CPU”, --n-cpu-moe N those of the first \(N\) layers, and --override-tensor places any tensor by name. [Certain] This is a coarse form of the PowerInfer idea: put the rarely used weights where memory is cheap.
  • Prefill is different. The prompt is processed as a batch of many tokens, so it is compute-bound, and the CPU is slow at compute. llama.cpp can copy CPU-side weights to the GPU for large prompt batches. [Likely] So long prompts suffer less than you might fear.

FlexGen: plan where every tensor lives

FlexGen asks a different question: with one GPU, how many tokens per second can we produce in total, if each request may take minutes? Its setting is a Google Cloud machine with an NVIDIA T4 (16 GB), 208 GB of DRAM, and a 1.5 TB SSD. [Certain] OPT-175B has 174 billion weights in its layers, 348 GB in FP16. It does not even fit in DRAM; part must live on the SSD.

Step 1: the order of computation

Think of the work for one generated token as a grid: one row per batch of sequences (a “GPU batch”, the batch that the GPU computes at once), one column per layer. Earlier offloading systems walk the grid row by row: finish all layers for batch 1, then all layers for batch 2. [Certain] That is the fastest way to finish one batch, but two neighbouring squares in a row use different weights, so every square loads its layer’s weights again.

FlexGen walks column by column inside a block of GPU batches: load layer \(j\)’s weights once, run them on all GPU batches in the block, then move to layer \(j+1\). The weights are used many times per load. The price: the activations and KV cache of all batches in the block must be stored somewhere while they wait, so the block size is limited by memory. FlexGen calls this the zig-zag block schedule and proves that its I/O cost is within 2× of the optimal schedule. [Certain] Step through Figure 3.

computedcomputing nowweights of this layer are on the GPU
Figure 3. Two orders for one decoding step of 4 GPU batches through 4 layers. Each weight load costs one layer of OPT-30B in FP16 (1.23 GB) over the T4 machine’s 12 GB/s bus: 0.103 s.
How to read Figure 3

In “Row by row”, the weight counter goes up at every square: 16 loads, 1.64 s of transfer. In “Zig-zag block”, it goes up only when the column changes: 4 loads, 0.41 s. The cost appears in the other counter: in the block order, up to 4 batches of activations and KV cache are “parked” (stored on CPU) while they wait for the next layer. A real block in FlexGen has many more GPU batches, for example 8 GPU batches of 32 sequences for OPT-175B. [Certain]

FlexGen also overlaps everything it can. While the GPU computes batch \(k\) of layer \(j\), it already loads the weights of layer \(j+1\), loads the KV cache and activations of batch \(k+1\), and stores those of batch \(k-1\). This is Algorithm 1 of the paper:

Algorithm 2: FlexGen block schedule with overlapping (after Sheng et al., Algorithm 1)
  1. for \(i = 1\) to generation length do
  2.   for \(j = 1\) to number of layers do // one column of the grid
  3.     for \(k = 1\) to number of GPU batches in the block do
  4.       load weight\((i, j+1, k)\) // prefetch the next layer
  5.       store activation\((i, j, k-1)\); store cache\((i, j, k-1)\)
  6.       load cache\((i, j, k+1)\); load activation\((i, j, k+1)\)
  7.       compute\((i, j, k)\) // the six operations above run in parallel with this
  8.       synchronize()

The block size (GPU batch size × number of GPU batches) is the effective batch size. [Certain]

Step 2: a cost model

With perfect overlap, the slowest of the parallel streams sets the time. FlexGen estimates the average decode time of one layer as

\[ T_{\text{gen}} = \max\left(\text{ctog}_g,\ \text{gtoc}_g,\ \text{dtoc}_g,\ \text{ctod}_g,\ \text{comp}_g\right), \qquad T = T_{\text{pre}} \cdot l + T_{\text{gen}} \cdot (n - 1) \cdot l \tag{130.5} \]
Read the symbols
\(\text{ctog}, \text{gtoc}\)
time to read from CPU to GPU, and to write from GPU to CPU, for one layer
\(\text{dtoc}, \text{ctod}\)
time to read from disk to CPU, and to write from CPU to disk
\(\text{comp}\)
compute time for one layer, on GPU and CPU together
\(T_{\text{pre}}\)
the same maximum for one layer in the prefill phase
\(l, n\)
number of layers, number of tokens to generate
\(T\)
time to finish one block of sequences

Each I/O term is a sum of bytes divided by a bandwidth. For example, in FP16 one OPT layer has \(8h_1^2 + 4h_1h_2\) bytes of weights (\(h_1\) is the hidden size, \(h_2\) the FFN size), and the disk read time is the disk share of the weights, the KV cache and the activations, divided by the disk bandwidth. [Certain]

Step 3: a linear program for placement

Where should each kind of tensor live? FlexGen uses nine numbers: \(w_g, w_c, w_d\) are the fractions of the weights on GPU, CPU and disk; \(h_g, h_c, h_d\) the same for the activations; \(c_g, c_c, c_d\) for the KV cache. Together with the block size \(\text{bls}\) and the GPU batch size \(\text{gbs}\), that is a policy of 11 variables. [Certain] It enumerates a few \((\text{bls}, \text{gbs})\) pairs and, for each, solves a small linear program:

\[ \begin{aligned} \min_{p}\ & T / \text{bls} \\ \text{s.t. }\ & \text{peak memory on GPU, CPU, disk} \le \text{its capacity} \\ & w_g + w_c + w_d = 1, \quad c_g + c_c + c_d = 1, \quad h_g + h_c + h_d = 1 \end{aligned} \tag{130.6} \]

The objective \(T / \text{bls}\) is the time per sequence in the block, so minimizing it maximizes throughput. The program has only 9 variables, so it solves very quickly; the paper notes that the result sometimes runs out of memory and must be adjusted by hand, and that hand tuning can beat it. [Certain] Weights are placed with layer granularity, activations and KV cache with tensor granularity. [Certain]

Step 4: compute attention on the CPU

The KV cache of a big batch is huge, and in decode, attention reads all of it to do very little arithmetic. If the KV cache lives in DRAM, moving it to the GPU costs much more than moving the small query to the CPU. FlexGen’s numbers, for GPU batch size \(b\), sequence length \(s\) and hidden size \(h_1\) in FP16:

\[ \frac{\text{bytes to move the KV cache}}{\text{bytes to move the activations}} = \frac{b \cdot s \cdot h_1 \cdot 4}{b \cdot h_1 \cdot 4} = s \tag{130.7} \]

So for \(s = 512\), computing attention on the CPU moves 512 times fewer bytes. FlexGen calls this computation delegation and uses it for long sequences (\(s \ge 512\)) when the KV cache is not on the GPU. [Certain] FastDecode, also named by the survey, goes further and keeps the whole attention operator and the KV cache on CPUs. [Certain]

Step 5: compress to 4 bits

The more compact the tensors, the fewer bytes cross the bus and the more sequences fit. FlexGen quantizes both the weights and the KV cache to 4 bits with group-wise min-max (asymmetric) quantization, in groups of 64 elements (session 103):

\[ x_{\text{quant}} = \operatorname{round}\!\left( \frac{x - \min}{\max - \min} \cdot (2^b - 1) \right) \tag{130.8} \]

Here \(\min\) and \(\max\) are taken over the group and \(b = 4\). Tensors are stored this way and turned back into FP16 just before the computation, so the goal is compression, not faster arithmetic. With one FP16 min and one FP16 max per 64 values, the cost is \(4 + 32/64 = 4.5\) bits per value. Weights are grouped along the output channel and the KV cache along the hidden dimension; the paper reports negligible accuracy loss on OPT-175B. Decompression is slow on the CPU, so FlexGen turns off CPU delegation when compression is on. [Certain]

The same paper also shows a simple approximation for attention: load only the top 10% of the value cache for each query (Top-K sparse attention), with model quality kept on OPT-175B. [Certain] Compare with dynamic sparse attention in session 117.

Results

Generation throughput (tokens/s) with prompts of 512 tokens and 32 generated tokens per prompt, on one T4 (Table 2 of the paper; Petals is a distributed system and is omitted here): [Certain]

SystemOPT-6.7BOPT-30BOPT-175B
HF Accelerate25.120.620.01
DeepSpeed ZeRO-Inference9.280.600.01
FlexGen25.267.320.69
FlexGen, 4-bit compression29.128.701.12

For OPT-6.7B the model fits on the GPU and the systems are similar. For OPT-175B the baselines could use a batch of only 2, while FlexGen used a GPU batch of 32 and a block of 32 × 8: 69× higher throughput. With compression, an effective batch of 144 and everything in DRAM (no disk), 112×. [Certain] Note the price: a block of OPT-175B takes thousands of seconds to finish. FlexGen is for offline work.

Figure 4 lets you play with the throughput side of this trade-off with a simple bandwidth model of OPT-30B on the T4 machine.

throughput (tokens/s, left axis)time to finish the block (s, right axis)CPU memory full
Figure 4. A FlexGen-style bound for OPT-30B (59.2 GB of FP16 layer weights) on 16 GB T4 + 208 GB DRAM. All weights and the KV cache live in DRAM; weights stream over PCIe at 12 GB/s once per step for the whole block. Prompt 512 tokens, 32 generated. Our assumptions: perfect overlap, GPU at 30 TFLOP/s, CPU attention reading DRAM at 50 GB/s.
How to read Figure 4

At batch 1 the weight transfer (4.93 s per token) is everything: 0.2 tokens/s. Raise the batch: the transfer is shared, and throughput grows almost linearly while the time to finish grows slowly. At large batches, prefill compute (2 FLOPs per weight per prompt token) starts to dominate and throughput flattens. Turn off delegation: the KV cache must cross PCIe every step, and above about 80 sequences it costs more than the weights. Turn on 4-bit: fewer bytes and more room in DRAM. The bound is higher than the 7.32 tokens/s that FlexGen measured for OPT-30B, because real overlap and kernels are not perfect.

PowerInfer: hot neurons on the GPU, cold neurons on the CPU

llama.cpp cuts the model between layers. PowerInfer cuts it inside each FFN, one neuron at a time. A neuron of an FFN is one hidden unit: one row of the up-projection (and of the gate, if there is one) and the matching column of the down-projection (session 21).

Activation sparsity

With a ReLU activation, every neuron whose pre-activation is negative outputs exactly 0, and its column of the down-projection contributes nothing. For one token, most neurons are 0. LLM in a flash reports 97% sparsity in the FFN of OPT 6.7B. [Certain] So for each token we could skip most FFN rows, if we knew in advance which ones. This is contextual sparsity: which neurons are active depends on the input. Deja Vu (Liu et al., 2023) showed that small predictors can guess the active set from the layer input. [Certain]

Models with SwiGLU, such as Llama, are much less sparse, because SiLU is rarely exactly 0. PowerInfer’s best results are on ReLU-family models (OPT, Falcon with ReLU, Llama 2 with ReGLU); models with SiLU or SwiGLU show smaller speed-ups. [Certain] Turning a SwiGLU model into a ReLU model needs extra training (“ReLU-fication”). [Likely]

The power law

Across many tokens, the activation frequency of neurons is very skewed. PowerInfer reports that about 17% of the neurons of OPT-30B, and 26% of those of LLaMA2 (ReGLU) 70B, produce 80% of all activations. [Certain] So: put the hot neurons, the ones that fire often, permanently on the GPU. Put the cold neurons in DRAM and let the CPU compute the few cold neurons that fire for the current token.

Let neuron \(i\) have activation probability \(p_i\), sorted so that \(p_1 \ge p_2 \ge \dots \ge p_n\). With the top \(k = g \cdot n\) neurons on the GPU, the share of activations served by the GPU (the coverage) and the cold bytes the CPU reads per token are:

\[ c(g) = \frac{\sum_{i \le gn} p_i}{\sum_{i=1}^{n} p_i}, \qquad \E[\text{CPU bytes per token}] = L \cdot \beta \sum_{i \gt gn} p_i \tag{130.9} \]
Read the symbols
\(n, L\)
neurons per FFN layer (28,672 for the Llama 3.1 70B shape), number of layers
\(g\)
fraction of neurons kept on the GPU
\(p_i\)
how often neuron \(i\) is active, measured offline on general text
\(\beta\)
bytes per neuron: gate row + up row + down column, \(3 \times 8192 \times 4.5/8 = 13{,}824\) bytes at 4.5 bits

Then, like equation 130.3, the time per token is about

\[ t_{\text{token}} \approx \frac{W_{\text{always}} + L\beta \sum_{i \le gn} p_i}{B_{\text{gpu}}} + \frac{L\beta \sum_{i \gt gn} p_i}{B_{\text{dram}}} \tag{130.10} \]

where \(W_{\text{always}}\) are the bytes that every token reads on the GPU: attention weights, the output head and the predictors. Both sums are small when the model is sparse, and the expensive term (the CPU one) shrinks fast as \(g\) grows, because the first neurons carry most of the probability.

How PowerInfer works

Algorithm 3: PowerInfer, offline and online
  1. Offline profiling: run the model on general text and count how often each neuron is active, giving \(p_i\)
  2. Offline placement: solve an integer linear program that maximizes the total \(p_i\) placed on the GPU, subject to GPU memory, one device per neuron, and a rule that a layer gets either enough GPU neurons to pay for the CPU-GPU synchronization or none // neurons are grouped in batches of 64; solving takes about 10 s
  3. Online, for every token and every layer:
  4.   GPU: attention; run the small predictor for this FFN on the layer input
  5.   GPU: compute the predicted-active hot neurons with a sparse kernel
  6.   CPU, in parallel: compute the predicted-active cold neurons, reading only their rows from DRAM
  7.   send the CPU’s partial result (one vector) to the GPU and add it to the GPU’s partial result

The details are from the paper. [Certain] The predictors are small networks per layer; the paper says it limits them to about 6% of the model’s parameters (its tables give 4.9% to 8.7% for different models). [Likely] The engine is built on llama.cpp, adding about 4,200 lines of C++ and CUDA. [Certain]

Figure 5 is a bandwidth model of this idea for a hypothetical ReLU version of Llama 3.1 70B at 4.5 bits on the 24 GB desktop. Activation probabilities follow a power law \(p_i \propto i^{-a}\) (capped at 1), scaled so that a chosen fraction \(\rho\) of neurons is active per token.

hot neurons (GPU)cold neurons (CPU)attention, head, predictors, KV, buffers
Figure 5. Hot/cold neuron placement. Left: the activation probability of the 28,672 neurons of one FFN layer, sorted (log scale on both axes). Right: GPU memory against 24 GB, and the time per token from equation 130.10 compared with a dense llama.cpp layer split that uses the same GPU.
How to read Figure 5

With the defaults (10% active, skew 1.2), about 28% of the neurons produce 80% of the activations, close to PowerInfer’s 26% for LLaMA2 (ReGLU) 70B. The largest hot set that fits is 35% of the neurons; it covers 84% of activations. Move “Neurons on the GPU” to 0: everything FFN is on the CPU, but sparse, and it is already much faster than the dense split. That part of the gain comes from sparsity. Move it up: the extra gain comes from placement. Raise \(\rho\) to 40%, like a model that is not very sparse: the gain over the dense split shrinks a lot. Lower the skew: the hot set covers less. The bound ignores predictor errors, CPU-GPU synchronization and the cost of scattered reads, so real systems are slower.

What did PowerInfer measure? On a desktop with an RTX 4090, it reports an average of 13.20 tokens/s and a peak of 29.08 tokens/s with INT4 models (8.32 and 16.06 with FP16), up to 11.69× faster than llama.cpp, and a generation rate of 82% of a server A100 GPU. [Certain] Compare with our bound: the real speed-up is smaller than the bound, as expected.

LLM in a flash: when even DRAM is too small

On a phone or laptop, the model may not fit in DRAM either. LLM in a flash (Alizadeh et al., Apple) keeps the weights on flash storage and builds on the same sparsity. Its numbers for OPT 6.7B (13.4 GB in FP16): loading everything from flash at 6.1 GB/s takes about 2.2 s per token. [Certain] Three ideas cut that down:

  • Keep the dense part in DRAM. Attention weights, about one third of the model, stay in DRAM. Only FFN rows are loaded from flash, using a low-rank predictor of the active neurons. [Certain]
  • Windowing. Keep in DRAM the neurons that were active for the last \(k\) tokens (the paper uses windows of 4 or 5 tokens). Consecutive tokens use many of the same neurons, so each new token loads only the difference: [Certain]
    \[ \text{rows to load at step } t = \left|\, A_t \setminus \textstyle\bigcup_{\tau = t-k}^{t-1} A_\tau \right| \tag{130.11} \]
    where \(A_t\) is the set of neurons predicted active for token \(t\).
  • Row-column bundling. Flash is fast only for large reads. Neuron \(i\) needs row \(i\) of the up-projection and column \(i\) of the down-projection, so store them next to each other. One read then fetches both, and the chunk size doubles from \(d_{\text{model}} \cdot \text{bytes}\) to \(2 \cdot d_{\text{model}} \cdot \text{bytes}\) (16 KB instead of 8 KB for OPT 6.7B in FP16). [Certain]

The paper reports a 4–5× speed-up on CPU and 20–25× on GPU over naive loading, for models up to twice the size of DRAM. For OPT 6.7B on an M1 Max CPU, the time per token went from 3,182 ms to 669 ms. [Certain]

PyTorch: a hot/cold FFN and 4-bit group quantization

The code below builds a toy ReLU FFN whose neurons fire with skewed frequencies. It profiles the frequencies, keeps the top 20% of neurons as the “GPU” part, and computes the cold neurons only where a cheap rank-16 predictor says they will fire (Algorithm 3). With a perfect predictor the hybrid result equals the dense result; with the cheap predictor it misses a few neurons. The last part is FlexGen’s group-wise 4-bit quantization (equation 130.8). Everything runs on the CPU and is deterministic.

import torch
torch.manual_seed(0)
d, n, r = 256, 1024, 16                           # model width, FFN neurons, predictor rank

# A toy ReLU FFN. W_up has low-rank structure plus noise, and the biases make
# some neurons fire often and most fire rarely (a skewed, power-law-like pattern).
W_up = (torch.randn(n, r) @ torch.randn(r, d) / (r * d) ** 0.5) * 2 + 0.02 * torch.randn(n, d)
b_up = 3.0 - torch.arange(1, n + 1).float().log()
W_dn = torch.randn(d, n) / n ** 0.5
def ffn_dense(x):                                  # x: (B, d)
    return torch.relu(x @ W_up.T + b_up) @ W_dn.T

# 1) Offline profiling: activation frequency of every neuron on calibration data
calib = torch.randn(4096, d)
freq = (calib @ W_up.T + b_up > 0).float().mean(0)
order = torch.argsort(freq, descending=True)
hot, cold = order[: n // 5], order[n // 5 :]       # GPU budget: 20% of the neurons
print(f"active per token: {freq.mean():.1%}; hot 20% cover {freq[hot].sum() / freq.sum():.1%} of activations")

# 2) A cheap predictor for the cold neurons: a rank-r copy of their up-projection rows
U, S, Vh = torch.linalg.svd(W_up[cold], full_matrices=False)
A, Bm = U[:, :r] * S[:r], Vh[:r]                   # W_up[cold] is close to A @ Bm
def lowrank_pred(x, margin=0.2):
    return (x @ Bm.T) @ A.T + b_up[cold] > -margin # True = "will be active"
def oracle_pred(x):
    return x @ W_up[cold].T + b_up[cold] > 0

# 3) Hybrid forward: hot neurons dense ("GPU"), cold neurons only where predicted ("CPU")
def ffn_hybrid(x, predict):
    y = torch.relu(x @ W_up[hot].T + b_up[hot]) @ W_dn[:, hot].T     # GPU part
    mask = predict(x)
    for i in range(x.shape[0]):                    # decode: one token at a time
        idx = cold[mask[i]]                        # the only cold rows the CPU reads
        h = torch.relu(x[i] @ W_up[idx].T + b_up[idx])
        y[i] += h @ W_dn[:, idx].T                 # partial sums are added on the GPU
    return y, mask.float().mean().item()

x = torch.randn(64, d)
y_ref = ffn_dense(x)
for name, pred in [("oracle", oracle_pred), ("rank-16", lowrank_pred)]:
    y, touched = ffn_hybrid(x, pred)
    recall = (pred(x) & oracle_pred(x)).sum() / oracle_pred(x).sum()
    err = ((y - y_ref).norm() / y_ref.norm()).item()
    print(f"{name:8s} cold rows read {touched:.1%}  recall {recall:.1%}  rel. error {err:.1e}")

# 4) FlexGen-style group-wise 4-bit quantization (min-max, groups of 64), formula (130.8)
def quant4(w, g=64):
    w = w.reshape(-1, g)
    lo, hi = w.min(1, keepdim=True).values, w.max(1, keepdim=True).values
    q = torch.round((w - lo) / (hi - lo) * 15).to(torch.uint8)       # integers 0..15
    return q, lo, hi
def dequant4(q, lo, hi, shape):
    return (q.float() / 15 * (hi - lo) + lo).reshape(shape)
q, lo, hi = quant4(W_dn)
err = (dequant4(q, lo, hi, W_dn.shape) - W_dn).norm() / W_dn.norm()
bits = (q.numel() * 4 + (lo.numel() + hi.numel()) * 16) / W_dn.numel()
print(f"4-bit group-wise: {bits:.2f} bits per weight, relative error {err:.3f}")

Output (PyTorch 2.8 on CPU):

active per token: 9.6%; hot 20% cover 56.6% of activations
oracle   cold rows read 4.8%  recall 100.0%  rel. error 4.9e-07
rank-16  cold rows read 5.6%  recall 93.9%  rel. error 2.1e-02
4-bit group-wise: 4.50 bits per weight, relative error 0.089

Read it like this. The CPU reads only about 5% of the cold rows per token instead of 100%. With the oracle predictor the split is exact (the error is only float rounding): splitting an FFN by neurons and adding the partial sums is just a re-ordering of the same sum. The cheap predictor misses 6% of the active cold neurons and the output error is 2%; real systems train better predictors and use a margin so that misses are rare. The toy’s skew (20% of neurons cover 57%) is weaker than PowerInfer’s measured skew (17% to 26% cover 80%).

A cost model that picks -ngl for llama.cpp, in plain Python (equations 130.1 and 130.3; same numbers as Figure 2):

def best_ngl(L=80, layer_gb=0.4813, head_gb=0.591, kv_gb_per_layer_tok=4096e-9,
             ctx=4096, gpu_gb=24.0, buffers_gb=1.5, bw_gpu=1008, bw_cpu=67.2):
    best = None
    for ngl in range(L + 2):                       # L+1 = layers and the output head
        n_l, head_on_gpu = min(ngl, L), ngl > L
        gpu_w = n_l * layer_gb + (head_gb if head_on_gpu else 0)
        cpu_w = (L - n_l) * layer_gb + (0 if head_on_gpu else head_gb)
        mem = gpu_w + n_l * kv_gb_per_layer_tok * ctx + buffers_gb
        if mem > gpu_gb:
            break
        t = gpu_w / bw_gpu + cpu_w / bw_cpu          # seconds per token, a lower bound
        best = (ngl, round(mem, 2), round(1 / t, 2))
    return best
print(best_ngl())            # (45, 23.91, 3.56): 45 layers, 23.91 GB, at most 3.56 tokens/s

Choosing a strategy

SituationUseWhy
Model fits on the GPU after 4-bit quantizationNo offloading; quantize insteadAny offloading is many times slower than GPU memory (Figure 1)
Chat on one PC, dense model a bit too largellama.cpp layer split, as many -ngl as fitComputing in place uses DRAM bandwidth, not the slower PCIe (equation 130.3)
Chat on one PC, MoE modelExperts on CPU (--n-cpu-moe), everything else on GPUOnly a few experts are read per token, so the CPU side reads few bytes
Chat on one PC, ReLU-sparse modelPowerInfer-style hot/cold neuronsSparsity and the power law cut the CPU bytes per token (equation 130.9)
Offline batch jobs, one GPU, huge modelFlexGen-style block schedule, large effective batch, 4-bit, CPU attentionWeight transfer is shared by the batch (equation 130.4)
Phone or laptop, model larger than DRAMFlash loading with windowing and bundlingLoads only new, predicted rows, in large chunks

Where this is used

ApplicationOffloading choiceNotes
Local chat assistants on PCs and Macsllama.cpp and apps built on it, GGUF models, partial -nglLatency for one user; batch size 1 [Likely]
Large MoE models (Mixtral, DeepSeek, gpt-oss) on a single consumer GPUExpert weights in DRAM, attention and shared parts on the GPUThe --cpu-moe family of options exists for this [Certain]; speed depends on how many expert bytes each token reads
Offline evaluation and data labeling (benchmarks, synthetic data, document classification)FlexGen-style throughput offloadingFlexGen ran the HELM sub-scenarios for a 30B model on one GPU in 21 hours [Certain]
Captioning or tagging a video archive with a VLM, overnightLarge effective batch, weights streamed once per step for all frames or clipsSame trade as FlexGen: minutes of latency per clip are fine; frame tokens make the KV cache large, so CPU attention helps
On-device video assistant (key-moment search, Video-RAG over home videos)llama.cpp-style split or flash loading on laptops and phonesBatch size 1; DRAM or flash bandwidth sets the speed; llama.cpp supports some VLMs [Likely]
Serving very long contextsKV cache in CPU memory, moved or attended on CPUThe same bandwidth arithmetic as equation 130.7; see session 110 for the compression side

Common mistakes

  • Streaming weights over PCIe at batch size 1. If the weights are already in DRAM and DRAM is faster than the bus, compute on the CPU (equation 130.3). Generic “offload to CPU” in training-style libraries often streams. [Likely]
  • Expecting offloading to be “almost as fast”. Even a good split of a 70B model on a 24 GB GPU is bounded at about 3.6 tokens/s, against 25.8 if it all fit. The slowest level decides the speed.
  • Forgetting the KV cache and buffers when counting GPU memory. A long context takes GPU memory from the layers. Plan with equation 130.1 plus the KV cache, then leave headroom.
  • Using a FlexGen-style big block for interactive use. A large effective batch gives high throughput but each request waits for the whole block (minutes). Throughput and latency are different goals (Figure 4).
  • Expecting PowerInfer-style gains on a SwiGLU model. Without real activation sparsity, the cold part is not cold. The paper itself shows smaller gains for SiLU models.
  • Ignoring prefill. Offloading analyses usually focus on decode. With long prompts, prefill is compute-bound and a CPU is slow at it; in Figure 4, prefill dominates at large batches.
  • Letting the model page from the SSD by accident. If a memory-mapped model is larger than free DRAM, the operating system reads pages from disk repeatedly. Speed drops to SSD speed (seconds per token).
  • Comparing systems with different quantization. A 4-bit llama.cpp run against an FP16 run of another system says more about bytes than about the engine. Compare bytes per token first.

Interview questions

Your model does not fit on the GPU. List your options and when you would pick each.basic
(1) Quantize (8-bit, 4-bit) until it fits: usually the best for latency. (2) Use more GPUs with tensor or pipeline parallelism: best if you have them. (3) Offload part of the model or KV cache to CPU memory or SSD. For one interactive user, use a layer split that computes CPU layers in place (llama.cpp), or hot/cold neurons if the model is sparse (PowerInfer). For offline throughput, use a large batch and a schedule that reuses each loaded weight many times (FlexGen). Often you combine quantization and offloading.
Why does llama.cpp compute the offloaded layers on the CPU instead of copying them to the GPU?core
At batch size 1, decode reads each weight once and does about 2 FLOPs per weight, so it is limited by memory bandwidth on any device. If the weights are in DRAM, the CPU reads them at DRAM bandwidth (about 60–100 GB/s on desktops), while copying them to the GPU goes over PCIe (about 25 GB/s for 4.0 ×16). So computing in place is faster, and only a few KB of activations cross the bus. For 17.44 GB of CPU-side weights: 0.26 s versus 0.70 s per token. The picture changes for large batches or prefill, where the CPU’s low compute becomes the limit.
How does FlexGen reach high throughput with weights on CPU and disk?core
It shares each weight transfer among many sequences. It uses a zig-zag block schedule: for each layer, load the weights once and run them on all GPU batches of a block, instead of loading them again for every batch (row-by-row). It overlaps weight loads, cache and activation loads and stores with compute. It chooses where weights, activations and KV cache live (GPU, CPU, disk fractions) with a linear program over a cost model. It computes attention on the CPU when the KV cache is there, and compresses weights and KV cache to 4 bits so that more fits in DRAM. Result: OPT-175B on one T4 at 0.69 tokens/s (1.12 with compression), against 0.01 for the baselines.
Why does FlexGen compute attention on the CPU? Give the ratio.core
In decode, attention reads the whole KV cache to do little arithmetic. If the cache is in DRAM, moving it to the GPU costs \(b \cdot s \cdot h_1 \cdot 4\) bytes per layer (FP16 K and V), while moving the activations to the CPU costs \(b \cdot h_1 \cdot 4\). The ratio is \(s\), the sequence length: 512× less traffic at \(s = 512\). FlexGen uses this for \(s \ge 512\) and turns it off with compression, because decompression on the CPU is slow.
Explain the hot/cold neuron idea of PowerInfer, and when it does not help.core
In ReLU-family LLMs, most FFN neurons output 0 for a given token, and over many tokens a small set fires very often: about 17% of OPT-30B’s neurons produce 80% of activations. PowerInfer profiles frequencies offline, uses an integer linear program to place the most frequent neurons on the GPU within its memory, and keeps the rest in DRAM. Online, predictors guess which neurons will fire; the GPU computes active hot neurons, the CPU computes active cold ones, and the partial results are added on the GPU. It helps little for dense activations (SwiGLU without ReLU-fication), when the predictor is bad, or when the whole model fits on the GPU anyway.
Write the time per token for a layer split and use it to choose the number of GPU layers.core
\(t = W_g / B_{\text{gpu}} + W_c / B_{\text{dram}}\), with \(W_g\) and \(W_c\) the bytes on each side. Each layer moved to the GPU changes \(t\) by \(w_\ell (1/B_{\text{gpu}} - 1/B_{\text{dram}})\), which is always negative, so use as many GPU layers as fit after reserving memory for the KV cache of those layers and working buffers. For 70B at 4.5 bits on 24 GB with 4,096 tokens of context: 45 layers, about 3.56 tokens/s as an upper bound.
What is the zig-zag block schedule and why is it not used for interactive serving?advanced
The work is a grid of (batch, layer) squares per token. Row-by-row finishes one batch through all layers and reloads weights for every square. The block schedule runs a column (one layer) over a block of GPU batches before moving to the next layer, so each weight load serves the whole block; it stops when the parked activations and KV cache fill memory. Its I/O is within 2× of optimal. For interactive use, each request must wait until the whole block finishes every token, so latency is very high; the schedule only makes sense when throughput is the goal.
How does LLM in a flash make flash storage usable for inference?advanced
It keeps attention weights (about a third of the model) in DRAM and loads only the FFN rows predicted to be active. Windowing keeps neurons used by the last few tokens in DRAM, so each new token loads only the new ones. Row-column bundling stores the up-projection row and down-projection column of a neuron together, doubling read size, because flash is fast only for large reads. It reports 4–5× (CPU) and 20–25× (GPU) faster than naive loading for models up to twice the DRAM size.
Offloading an MoE model: what goes where, and why does it work better than for a dense model?advanced
Put attention, shared experts, router and KV cache on the GPU, and the routed experts in DRAM, computed by the CPU (llama.cpp’s --cpu-moe, --n-cpu-moe). Each token uses only the top-k experts, so the CPU reads only a fraction of the expert bytes, while total parameter count can be huge. A dense model must read all CPU-side bytes for every token. The speed is set by active expert bytes per token over DRAM bandwidth.
Could speculative decoding help an offloaded model? How?advanced
Yes, in principle. With offloading, each pass over the weights is very expensive and almost all of the cost is data movement, so verifying several drafted tokens in one pass amortizes the transfer over several tokens, like a batch does in FlexGen. A small draft model can stay entirely on the GPU. The acceptance rate sets the gain (session 128). Systems that combine speculation with offloading exist. [Likely]
A team reports “our offloading engine is 10× faster than llama.cpp”. What do you check?core
Same model, same quantization (bytes per token), same context length, same hardware (GPU memory, DRAM bandwidth, PCIe version), the same -ngl tuned to fill the GPU, batch size 1 versus large batch, prefill versus decode speed, and output quality (sparsity predictors can change outputs). For sparsity-based engines, also check whether the model was ReLU-fied and how its accuracy compares to the original.

Exercises

  1. How much memory does Llama 3.1 70B need for its weights in Q8_0 (8.5 bits per weight)? Will it fit in a 192 GB DRAM desktop together with a 24 GB GPU?
  2. Using Figure 2’s model at 4.5 bits, compute the bound on tokens/s with 20 GPU layers, for both “CPU computes” and “stream”.
  3. In Figure 3, how many weight loads does row-by-row need for 8 GPU batches and 96 layers (OPT-175B) per token? And the block schedule?
  4. FlexGen at \(s = 1024\): by what factor does CPU delegation reduce the bytes moved for attention?
  5. Group-wise 4-bit quantization with groups of 128 and FP16 min and max: how many bits per value?
  6. In Figure 5, set \(\rho = 40\%\). What happens to the gain over the dense split? Explain with equation 130.10.
  7. Bundling in LLM in a flash for a model with \(d_{\text{model}} = 8192\) in FP16: what is the chunk size per neuron, before and after bundling?
Answerscheck
1. \(70.55 \times 10^9 \times 8.5 / 8 = 75.0\) GB. Yes: it fits in 192 GB DRAM easily; about 22 GB of it can go to the GPU. 2. GPU: \(20 \times 0.481 = 9.63\) GB; CPU: 29.47 GB. CPU computes: \(9.63/1008 + 29.47/67.2 = 0.448\) s, 2.23 tokens/s. Stream: \(9.63/1008 + 29.47/25 = 1.188\) s, 0.84 tokens/s. 3. Row-by-row: \(8 \times 96 = 768\) loads. Block: 96 loads, one per layer (8× fewer). 4. A factor of \(s = 1024\). 5. \(4 + 32/128 = 4.25\) bits. 6. The gain falls sharply (from about 14× to about 3.6× over the dense split in the figure): many more neurons are active, so both sums in 130.10 grow, and the CPU sum grows most because more of the probability lies in the cold tail; at very low sparsity the method approaches a dense split. 7. Before: \(8192 \times 2 = 16{,}384\) bytes (16 KiB); after: 32 KiB.

Key takeaways

  • Offloading trades speed for capacity. The time per token is set by bytes divided by bandwidth at each level (130.2), and the slow levels dominate: PCIe and DRAM are 15–40× slower than GPU memory.
  • At batch size 1, compute where the weights live (130.3): a CPU reading DRAM beats copying weights over PCIe. That is llama.cpp’s layer split, controlled by -ngl.
  • For throughput, share each weight transfer across a huge batch (130.4). FlexGen adds a zig-zag block schedule, a cost model (130.5), a linear program for placement (130.6), CPU attention (130.7) and 4-bit compression (130.8): OPT-175B on one T4 at about 1 token/s.
  • With sparse (ReLU) activations, split inside the FFN: hot neurons on the GPU, cold neurons on the CPU computed only when predicted (130.9, 130.10). PowerInfer reports up to 11.69× over llama.cpp.
  • When DRAM is too small, load only predicted rows from flash, reuse recent ones (130.11) and read in large bundled chunks (LLM in a flash).

Key formulas

\[ t_{\text{cpu}} = \frac{W_g}{B_{\text{gpu}}} + \frac{W_c}{B_{\text{dram}}}, \quad \frac{t_{\text{transfer}}}{\text{token}} = \frac{W_c}{B \cdot B_{\text{pcie}}}, \quad \frac{\text{KV bytes}}{\text{act. bytes}} = s, \quad c(g) = \frac{\sum_{i \le gn} p_i}{\sum_i p_i} \]

References for this session

  1. Sheng et al. (2023). FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  2. Song et al. (2023). PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
  3. Alizadeh et al. (2023). LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  4. ggml-org (code). llama.cpp
  5. Sheng et al. (2023). FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  6. Song et al. (2023). PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
  7. Alizadeh et al. (2023). LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  8. ggml-org (code). llama.cpp
  9. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  10. Liu, Wang, Dao et al. (2023). Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
  11. Aminabadi, Rajbhandari, Zhang et al. (2022). DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
  12. He, Zhai (2024). FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines