LLM inference and serving
The efficient-inference survey end to end: KV cache, quantization, pruning, FlashAttention, speculative decoding, vLLM, TensorRT-LLM.
Inference foundations and the efficiency problem
How generation runs, how to measure it, the three root causes of slow inference, and GPU kernel basics. Survey §2–3.
The generation loop: prefill and decodesurvey §2.2
- Prefill: read the whole prompt in parallel and build the KV cache (compute-bound)
- Decode: one token per step, reusing the KV cache (memory-bound)
- Why decode at batch 1 reads all weights to make about one token
- Example from the survey: LLaMA-2-70B in FP16 needs 140 GB, and one token takes about 100 ms on 2 A100s
Decoding strategies
- Greedy and beam search: where they work (translation) and fail (open text)
- Temperature, top-k, top-p (nucleus), min-p
- Repetition penalties and contrastive decoding
- Constrained decoding with grammars and JSON schemas
The KV cachesurvey §2.2
- Why we cache K and V, and not Q
- The size formula, with worked examples
- KV cache with MHA vs GQA vs MLA
- Peak memory ≈ weights + KV cache: the main limit for long context and big batches
Inference metrics: latency, memory and throughputsurvey §2.2
- Latency: first-token latency (TTFT), per-output-token latency (TPOT), generation latency
- Memory: model size, KV-cache size, peak memory
- Throughput: tokens per second and requests per second; per user vs for the whole system
- Goodput under a latency target (SLO); Little’s law
Why inference is slow: three root causes and the rooflinesurvey §2.3
- Three costs: compute, memory access, memory size
- Three root causes: model size, quadratic attention, token-by-token decoding
- Roofline: prefill sits near the compute roof, decode sits on the bandwidth slope
- Worked example: one 8B model on an NVIDIA L4 vs an H100
The map: data-level, model-level and system-level optimizationsurvey §3
- Data level: change the input or the shape of the output; the model stays the same
- Model level: change the architecture or compress the weights; often lossy, may need training
- System level: change the engine or the server; usually lossless
- How the next 11 weeks follow this map, one branch at a time
Batching basics and the latency–throughput trade-off
- Why batching helps decode: weights are read once for B tokens
- Static and dynamic batching; padding waste
- Plot: tokens/s and TPOT vs batch size; where the curve bends
- Batch limits from KV-cache memory
GPU kernels 1: CUDA basics for ML engineers
- Threads, warps, blocks, grids; occupancy
- Global, shared and register memory; coalesced access; bank conflicts
- Why fusion helps: fewer trips to HBM
- Placing a kernel on the roofline
GPU kernels 2: write kernels in the Triton language
- The programming model: blocks of data, not single threads
- Write a fused softmax and an RMSNorm kernel; autotune block sizes
- Compare with PyTorch eager and torch.compile
- Name trap: the Triton language is not Triton Inference Server
KV-cached generation, metrics and your GPU’s roofline
- Write generate() with and without a KV cache; measure the speedup
- Measure TTFT and TPOT vs prompt length and batch size
- Measure the memory bandwidth of your GPU and draw its roofline
- Place prefill and decode on the roofline
Data-level optimization
Make the input shorter and the output more parallel, without changing the model. Survey §4.
Prompt pruning: Selective Context, LLMLingua, LongLLMLinguasurvey §4.1.1
- Remove tokens, sentences or examples that carry little information
- Self-information from a small LM as the importance score
- LLMLingua: budget controller, coarse-to-fine pruning, iterative token pruning
- LongLLMLingua: question-aware scores and reordering for RAG prompts
Prompt summary and soft-prompt compression: RECOMP, Gisting, AutoCompressors, ICAEsurvey §4.1.2 §4.1.3
- Prompt summary: a small model rewrites retrieved text into a short summary (RECOMP)
- Soft prompts: learn a few vectors that act like a long prompt
- Gisting: compress task instructions into gist tokens
- AutoCompressors and ICAE: compress any context into memory slots
RAG as input compression: FLARE, REPLUG, Self-RAGsurvey §4.1.4
- Instead of a huge prompt, retrieve only the relevant pieces
- FLARE: retrieve when the model is about to be unsure
- REPLUG: improve the retriever using the frozen LLM as the teacher
- Cost math: long context vs retrieval (prefill tokens and KV cache)
Output organization: Skeleton-of-Thought, APAR, SGLang; model cascadessurvey §4.2
- Skeleton-of-Thought: write a short outline, then expand all points in parallel as a batch
- Graph-based and fork-based parallel decoding (SGD, APAR)
- SGLang programs: share the KV cache of common prefixes automatically
- Model cascades (FrugalGPT): try a small model first, call the big one only when needed
compress a long RAG promptsurvey §4.1.1 §4.1.4
- Build a long-context QA test (many retrieved chunks)
- Apply LLMLingua-style pruning at 2×, 5× and 10× compression
- Measure tokens, TTFT and answer accuracy at each ratio
- Compare with better retrieval (fewer, better chunks)
Model-level 1: efficient structure design
Cheaper FFNs (MoE, sharing), cheaper attention (GQA, low-rank, kernels), and alternatives to the transformer. Survey §5.1.
Where the cost is: FFN and attention, by the numberssurvey §5.1
- The survey reports the FFN holds 63% of LLaMA-7B parameters and 72% of LLaMA-70B parameters
- FFN cost grows with model size; attention cost grows with context length
- Compute your own model’s split: params, FLOPs and bytes per token
- Which part to optimize for your workload (short chat vs long documents vs video)
Efficient FFN 1: SwiGLU and parameter sharing (ALBERT)survey §5.1.1
- SwiGLU: better quality per parameter; why d_{ff} becomes about 8d/3
- ALBERT: share one layer’s weights across all layers
- ALBERT: factorized embeddings (|V|×E plus E×d instead of |V|×d)
- What sharing saves (memory) and what it does not save (compute)
Efficient FFN 2: MoE routing in depthsurvey §5.1.1
- Top-k token-choice routing and the capacity factor
- Load imbalance: why it wastes compute and padding
- Fixes: load-balancing loss (Switch), router z-loss, BASE layers, expert-choice routing
- Shared experts and fine-grained experts (DeepSeekMoE style)
Efficient FFN 3: MoE from dense models, and MoE inferencesurvey §5.1.1
- MoEfication: split a dense FFN into experts
- Sparse upcycling: copy a dense FFN into many experts, then train
- Inference: total vs active parameters; memory holds all experts
- Batching and expert parallelism at serving time; Mixtral and DeepSeek as examples
Efficient attention 1: MQA, GQA and MLA, from the KV-cache viewsurvey §5.1.2
- Share K and V across heads (MQA) or groups of heads (GQA)
- Uptraining: turn an MHA checkpoint into GQA by mean-pooling heads
- MLA: store one small latent vector instead of full K and V
- KV bytes per token and quality, side by side
Efficient attention 2: low-rank attention (Linformer, Luna, Set Transformer)survey §5.1.2
- The attention matrix is often close to low rank
- Linformer: project K and V from length n to a fixed k
- Luna and Set Transformer: a small set of learned queries summarizes the sequence (pack, then unpack)
- Limits: causal masking and variable lengths are hard
Efficient attention 3: kernel-based attention (Linear Transformer, Performer, RFA)survey §5.1.2
- Replace softmax(qᵀk) with φ(q)ᵀφ(k)
- Use associativity: compute φ(K)ᵀV once (d×d), then multiply by φ(Q)
- Performer (FAVOR+) and RFA: random features that approximate softmax
- Why quality drops: no sharp, peaked attention
Transformer alternatives 1: state space models from HiPPO to S4survey §5.1.3
- The linear state space recurrence
- HiPPO: compress history into polynomial coefficients
- S4, DSS, S4D, S5: diagonal matrices and fast convolution
- Two views of one model: recurrence (fast decode) and convolution (parallel training)
Transformer alternatives 2: Mamba, RWKV, RetNet, Hyena and hybridssurvey §5.1.3
- Mamba: input-dependent (selective) A, B, C with a parallel scan
- RWKV and RetNet: attention-like training, RNN-like decoding
- Hyena: long convolutions with data-dependent kernels
- Weakness at exact recall; hybrids such as Jamba mix attention and SSM layers
Linformer, Performer and GQA against full attentionsurvey §5.1.2
- Implement the three in one small GPT
- Measure speed and memory for sequence lengths 512 to 16k
- Measure quality (validation loss) and a copy task that needs exact recall
- Explain each result with the formulas
Model-level 2: quantization in depth
The math of quantization, weight-only and weight–activation methods, rotations, KV-cache quantization and QAT. Survey §5.2.1.
Quantization math: scale, zero point, granularity and errorsurvey §5.2.1
- Uniform affine quantization; symmetric vs asymmetric
- Granularity: per-tensor, per-channel, per-group (for example 128), per-token
- Rounding error vs clipping error; choosing the range (min-max, percentile, MSE)
- Why LLMs are hard: outliers and wide ranges
Where quantization helps: weight-only vs weight–activationsurvey §5.2.1
- Decode is GEMV and memory-bound: weight-only (W4A16, W8A16) helps
- Prefill is GEMM and compute-bound: you need W8A8 or FP8 to use faster tensor cores
- Survey test (A100, AWQ W4A16): decode about 2.0–2.4× faster at batch 1 and 1.2–1.4× at batch 16; prefill about 0.8–1.1×, often slower
- The gain shrinks as batch and context grow; bigger models gain more
Number formats: INT8, INT4, FP8, FP4, NF4 and microscaling
- FP8 E4M3 vs E5M2; per-tensor scaling
- FP4 and block-scaled formats (MX): a shared scale per small block
- NF4 for normally distributed weights (QLoRA)
- Hardware: which formats each GPU generation runs on tensor cores (for example, FP8 on L4 and H100) [check per GPU]
Weight PTQ 1: from OBQ to GPTQsurvey §5.2.1
- Layer-wise objective: keep WX close to ŴX on calibration data
- The Hessian H = 2XXᵀ tells which weights matter
- Quantize one column, then fix the remaining columns to cancel the error
- GPTQ’s speed tricks: fixed order, lazy batch updates, Cholesky
Weight PTQ 2: AWQ, outliers and mixed precisionsurvey §5.2.1
- AWQ: protect weights connected to large activations by scaling them first
- Search the scale per channel to minimize output error
- Keep outliers in high precision: SpQR, OWQ; non-uniform levels: SqueezeLLM
- Look-up-table kernels for non-uniform formats (LUT-GEMM)
Weight–activation quantization: LLM.int8(), SmoothQuant, ZeroQuant, OmniQuantsurvey §5.2.1
- Activation outliers live in a few channels
- LLM.int8(): run outlier channels in FP16, the rest in INT8
- SmoothQuant: move difficulty from activations to weights with a per-channel scale
- ZeroQuant (group and token scales), OmniQuant (learned clipping), Atom (mixed precision)
Rotations against outliers: QuIP, QuIP#, QuaRot, SpinQuantsurvey §5.2.1
- Rotate weights and activations; the output does not change
- A rotation spreads one big outlier over many channels
- QuaRot: random Hadamard rotations, enables 4-bit weights, activations and KV
- SpinQuant: learn the rotation on a small dataset
KV-cache quantization: KIVI and KVQuantsurvey §5.2.1
- The KV cache can be bigger than the weights at long context or big batch
- Keys have channel outliers: quantize keys per channel, values per token (KIVI)
- Non-uniform types and pre-RoPE keys (KVQuant)
- Measure: long-context accuracy, not only perplexity
Quantization-aware training: fake quant, STE, LLM-QAT, QLoRA, LoftQsurvey §5.2.1
- Fake quantization in the forward pass; straight-through estimator in the backward pass
- LLM-QAT: data-free QAT by distilling from the FP16 model on its own generated text
- QLoRA, QA-LoRA and LoftQ: low-bit base weights plus trainable low-rank adapters
- When PTQ is enough and when QAT is worth the cost
compare RTN, GPTQ, AWQ, SmoothQuant and FP8 on one modelsurvey §5.2.1
- Quantize a 7B–8B model with each method
- Measure perplexity, two tasks, prefill latency and decode latency on an L4
- Repeat at batch 1 and batch 16; compare with the survey’s table
- Write a one-page recommendation for your production model
Model-level 3: sparsity, factorization, distillation and dynamic inference
Pruning weights and attention, low-rank factorization, knowledge distillation and early exit. Survey §5.2.2–5.2.5.
Pruning basics: magnitude, OBD and OBS; structured vs unstructuredsurvey §5.2.2
- Magnitude pruning and its limits
- Second-order pruning: OBD and OBS use the Hessian to choose and fix weights
- Unstructured sparsity: high sparsity, little real speedup on GPUs
- Structured sparsity (channels, heads, layers): real speedup, bigger accuracy loss
Unstructured and 2:4 pruning: SparseGPT, Wanda, OWLsurvey §5.2.2
- SparseGPT: OBS-style pruning that scales to large models
- Wanda: score = |weight| × norm of its input activation; no weight update
- OWL: different sparsity per layer, based on outliers
- 2:4 semi-structured sparsity: NVIDIA tensor cores skip half the weights
Structured pruning: LLM-Pruner, Sheared LLaMA, SliceGPT, Minitronsurvey §5.2.2
- Find coupled structures (heads, channels) and remove them together
- Sheared LLaMA: prune to a target shape, then continue pretraining
- SliceGPT: rotate, then delete whole rows and columns
- Prune + distill (Minitron style); pruning experts in MoE models
Sparse attention 1 (static): Sparse Transformer, Longformer, BigBird, StreamingLLMsurvey §5.2.2
- Local (window), global, random and dilated patterns
- BigBird: local + global + random keeps the model’s power
- StreamingLLM: a window plus the first few tokens (attention sinks)
- Static patterns also shrink the KV cache during decode
Sparse attention 2 (dynamic): Reformer, Routing Transformer, SpAtten, H2Osurvey §5.2.2
- Bucket tokens by hashing (Reformer) or clustering (Routing Transformer)
- Prune unimportant tokens and heads on the fly (SpAtten)
- H2O: keep recent tokens plus heavy hitters; evict the rest from the KV cache
- Risk: once evicted, information is gone
Structure optimization: NAS and low-rank factorizationsurvey §5.2.3
- Neural architecture search for transformers, and why it is costly for LLMs
- Low-rank factorization with SVD
- Activation-aware SVD (ASVD, SVD-LLM)
- Mixing factorization with quantization and pruning
Distillation for LLMs 1: white-box (logits and features)survey §5.2.4
- Forward KL (mode-covering) vs reverse KL (mode-seeking)
- MiniLLM: reverse KL for generation
- GKD: train on the student’s own outputs (on-policy distillation)
- Feature and layer distillation; the capacity gap
Distillation for LLMs 2: black-box (outputs only)survey §5.2.4
- Distilling from API models: you only see the text
- Distilling in-context learning and instruction following
- Chain-of-thought distillation: Distilling step-by-step
- Reasoning distillation into small models (DeepSeek-R1 style); filtering bad rationales
Dynamic inference: early exit and layer skippingsurvey §5.2.5
- Sample-level early exit for classification: DeeBERT, FastBERT, PABEE
- Token-level early exit for generation: CALM, SkipDecode
- The KV-cache problem: skipped layers have no K and V for later tokens
- Self-speculation with early-exit layers (LayerSkip)
prune with Wanda (2:4), recover with distillationsurvey §5.2.2 §5.2.4
- Prune a small LLM to 50% (unstructured) and to 2:4
- Measure perplexity and real speed with sparse kernels
- Recover quality with a short distillation run
- Report the trade-off curve
System-level 1: the inference engine
Operator and graph optimization, FlashAttention and FlashDecoding, speculative decoding, offloading, TensorRT-LLM. Survey §6.1.
Runtime profiling: where the time goessurvey §6.1.1
- The survey’s profiling: attention and linear operators often take more than 75% of the time
- Many tiny kernels: launch overhead and CPU gaps
- Read a PyTorch profiler and an Nsight Systems trace
- Find the top 3 time sinks of a real model
Attention operator 1: FlashAttention 1, 2 and 3survey §6.1.1
- Standard attention writes the T×T matrix to HBM; that is the slow part
- Tiling + online softmax: attention block by block, inside SRAM
- Recompute in backward instead of storing; FA2 work split; FA3 on Hopper (asynchrony, FP8)
- Numerics: small differences from standard attention
Attention operator 2: decode attention, FlashDecoding and FlashDecoding++survey §6.1.1
- Decode has one query row: not enough parallel work for the GPU
- FlashDecoding: split the KV sequence, compute parts in parallel, merge with log-sum-exp
- FlashDecoding++: unified max value for softmax, flat-GEMM tuning
- Paged-KV attention kernels
Linear operators: GEMM, GEMV and flat GEMMsurvey §6.1.1
- Prefill = GEMM (big matrices); decode = GEMV or flat GEMM (very thin matrices)
- Tiling, split-K and padding waste for thin shapes
- Quantized GEMM: dequantize inside the kernel (W4A16)
- MoE grouped GEMM
Graph-level optimization: fusion, CUDA graphs, torch.compilesurvey §6.1.1
- Fuse small ops (bias, activation, norm, residual) into one kernel
- CUDA graphs remove CPU launch overhead in decode
- torch.compile: graph capture, fusion, graph breaks
- How TensorRT and inference engines fuse whole transformer layers
Speculative decoding 1: the mathsurvey §6.1
- A cheap draft proposes γ tokens; the target model checks them all in one pass
- The acceptance rule keeps the target model’s exact distribution
- Expected tokens per step from the acceptance rate α
- When speculation does not pay: big batches, weak drafts
Speculative decoding 2: drafts and verification (Medusa, EAGLE, Lookahead, SpecInfer)survey §6.1
- Separate draft model vs self-drafting heads (Medusa) vs feature-level drafting (EAGLE)
- Lookahead and prompt-lookup decoding: no draft model
- Tree verification: check many candidate branches at once (SpecInfer)
- Multi-token prediction heads used as drafters (MTP)
Offloading: FlexGen, llama.cpp, PowerInfersurvey §6.1
- When the model does not fit on the GPU: move weights or KV to CPU or SSD
- FlexGen: plan where each tensor lives to maximize throughput
- llama.cpp: split layers between CPU and GPU
- PowerInfer: keep “hot” neurons on the GPU, “cold” ones on the CPU
TensorRT and TensorRT-LLM in depth
- TensorRT: ONNX or network → optimized engine; layer fusion, kernel tuning, FP16, INT8, FP8
- Optimization profiles for dynamic shapes; an engine is tied to a GPU and a version
- TensorRT-LLM: paged KV, in-flight batching, quantization, speculative decoding, plugins
- When TensorRT-LLM beats vLLM, and when it is not worth the effort
profile, fuse and speculatesurvey §6.1.1 §6.1
- Profile one decode step with Nsight Systems
- Fuse RMSNorm + residual in Triton and measure
- Run speculative decoding with a small draft and with EAGLE-style heads
- Measure acceptance rate and tokens/s at batch 1, 4 and 16
System-level 2: serving systems
Memory management, batching, scheduling, distributed and disaggregated serving, engines, hardware and scenarios. Survey §6.2–7.
The serving stack and the life of a request
- Engine (vLLM, TensorRT-LLM, SGLang) → server (OpenAI API, Triton) → router → autoscaler
- Life of a request: tokenize, queue, schedule, prefill, decode, detokenize, stream
- Streaming, cancellation, timeouts
- Serving many models and many LoRA adapters
Memory management: PagedAttention, prefix caching, RadixAttentionsurvey §6.2
- Fragmentation with naive KV allocation
- Paging: fixed-size blocks and a block table, like virtual memory
- Sharing blocks across requests; copy-on-write for parallel sampling
- Prefix caching and RadixAttention for repeated system prompts and documents
Batching: continuous batching, chunked prefill, split-fusesurvey §6.2
- Iteration-level (continuous) batching: requests join and leave every step (ORCA)
- Chunked prefill: mix prefill chunks with decode steps (Sarathi, Sarathi-Serve)
- Split-fuse in DeepSpeed-FastGen
- Effect on TTFT, TPOT and throughput
Scheduling: FCFS, preemption, fairness and SLOssurvey §6.2
- Head-of-line blocking with first-come-first-served
- FastServe: preemptive multi-level feedback queue
- Fairness between users: Virtual Token Counter (VTC)
- SLO-aware scheduling and output-length prediction
Distributed serving and prefill–decode disaggregationsurvey §6.2
- Tensor and pipeline parallel serving
- Disaggregation: prefill GPUs and decode GPUs (Splitwise, DistServe, TetriInfer)
- Moving the KV cache between GPUs; cost of transfer
- Serving on cheap spot instances (SpotServe)
vLLM internals
- Scheduler and block manager; preemption by recompute or swap
- CUDA graphs for decode; tensor-parallel and pipeline-parallel serving
- Prefix caching, chunked prefill and speculative decoding inside vLLM
- Key settings: max_num_seqs, max_model_len, gpu_memory_utilization
Triton Inference Server, SGLang, LMDeploy, TGI, llama.cppsurvey §6.2
- Triton Inference Server: model repository, config.pbtxt, backends (TensorRT, TensorRT-LLM, ONNX, PyTorch, Python), ensembles
- SGLang and LMDeploy: what each does best
- The survey compares inference engines and serving systems; how to read such tables
- How to benchmark engines fairly on your own traffic
Hardware accelerators and edge deploymentsurvey §6.3
- GPUs vs FPGAs (FlightLLM) vs NPUs and other accelerators
- Edge: memory, power and bandwidth limits
- llama.cpp, MLX and mobile runtimes
- Which survey techniques matter most at the edge
Scenarios and capacity planning: agents, long context, edge, costsurvey §7
- Agents and multi-model pipelines: many calls, shared prefixes
- Long context: prefill cost, KV memory, sparse attention
- From traffic to GPUs: prompt and output lengths, latency target
- Worked example: one model on an L4 vs an H100; cost per million tokens
benchmark a serving systemsurvey §6.2 §6.2
- Serve a model with vLLM (optional: TensorRT-LLM on an L4)
- Plot TTFT, TPOT and throughput vs load and context length
- Turn on prefix caching, chunked prefill, FP8 or INT4, and speculative decoding, one at a time
- Explain each change with the survey’s taxonomy