LLM & VLM courseBlog
Track 5 · sessions 78–142

LLM inference and serving

The efficient-inference survey end to end: KV cache, quantization, pruning, FlashAttention, speculative decoding, vLLM, TensorRT-LLM.

Inference foundations and the efficiency problem

How generation runs, how to measure it, the three root causes of slow inference, and GPU kernel basics. Survey §2–3.

78

The generation loop: prefill and decodesurvey §2.2

  • Prefill: read the whole prompt in parallel and build the KV cache (compute-bound)
  • Decode: one token per step, reusing the KV cache (memory-bound)
  • Why decode at batch 1 reads all weights to make about one token
  • Example from the survey: LLaMA-2-70B in FP16 needs 140 GB, and one token takes about 100 ms on 2 A100s
Ready
79

Decoding strategies

  • Greedy and beam search: where they work (translation) and fail (open text)
  • Temperature, top-k, top-p (nucleus), min-p
  • Repetition penalties and contrastive decoding
  • Constrained decoding with grammars and JSON schemas
Ready
80

The KV cachesurvey §2.2

  • Why we cache K and V, and not Q
  • The size formula, with worked examples
  • KV cache with MHA vs GQA vs MLA
  • Peak memory ≈ weights + KV cache: the main limit for long context and big batches
Ready
81

Inference metrics: latency, memory and throughputsurvey §2.2

  • Latency: first-token latency (TTFT), per-output-token latency (TPOT), generation latency
  • Memory: model size, KV-cache size, peak memory
  • Throughput: tokens per second and requests per second; per user vs for the whole system
  • Goodput under a latency target (SLO); Little’s law
Ready
82

Why inference is slow: three root causes and the rooflinesurvey §2.3

  • Three costs: compute, memory access, memory size
  • Three root causes: model size, quadratic attention, token-by-token decoding
  • Roofline: prefill sits near the compute roof, decode sits on the bandwidth slope
  • Worked example: one 8B model on an NVIDIA L4 vs an H100
Ready
83

The map: data-level, model-level and system-level optimizationsurvey §3

  • Data level: change the input or the shape of the output; the model stays the same
  • Model level: change the architecture or compress the weights; often lossy, may need training
  • System level: change the engine or the server; usually lossless
  • How the next 11 weeks follow this map, one branch at a time
Ready
84

Batching basics and the latency–throughput trade-off

  • Why batching helps decode: weights are read once for B tokens
  • Static and dynamic batching; padding waste
  • Plot: tokens/s and TPOT vs batch size; where the curve bends
  • Batch limits from KV-cache memory
Ready
85

GPU kernels 1: CUDA basics for ML engineers

  • Threads, warps, blocks, grids; occupancy
  • Global, shared and register memory; coalesced access; bank conflicts
  • Why fusion helps: fewer trips to HBM
  • Placing a kernel on the roofline
Ready
86

GPU kernels 2: write kernels in the Triton language

  • The programming model: blocks of data, not single threads
  • Write a fused softmax and an RMSNorm kernel; autotune block sizes
  • Compare with PyTorch eager and torch.compile
  • Name trap: the Triton language is not Triton Inference Server
Ready
87

KV-cached generation, metrics and your GPU’s roofline

  • Write generate() with and without a KV cache; measure the speedup
  • Measure TTFT and TPOT vs prompt length and batch size
  • Measure the memory bandwidth of your GPU and draw its roofline
  • Place prefill and decode on the roofline
Ready

Data-level optimization

Make the input shorter and the output more parallel, without changing the model. Survey §4.

88

Prompt pruning: Selective Context, LLMLingua, LongLLMLinguasurvey §4.1.1

  • Remove tokens, sentences or examples that carry little information
  • Self-information from a small LM as the importance score
  • LLMLingua: budget controller, coarse-to-fine pruning, iterative token pruning
  • LongLLMLingua: question-aware scores and reordering for RAG prompts
Ready
89

Prompt summary and soft-prompt compression: RECOMP, Gisting, AutoCompressors, ICAEsurvey §4.1.2 §4.1.3

  • Prompt summary: a small model rewrites retrieved text into a short summary (RECOMP)
  • Soft prompts: learn a few vectors that act like a long prompt
  • Gisting: compress task instructions into gist tokens
  • AutoCompressors and ICAE: compress any context into memory slots
Ready
90

RAG as input compression: FLARE, REPLUG, Self-RAGsurvey §4.1.4

  • Instead of a huge prompt, retrieve only the relevant pieces
  • FLARE: retrieve when the model is about to be unsure
  • REPLUG: improve the retriever using the frozen LLM as the teacher
  • Cost math: long context vs retrieval (prefill tokens and KV cache)
Ready
91

Output organization: Skeleton-of-Thought, APAR, SGLang; model cascadessurvey §4.2

  • Skeleton-of-Thought: write a short outline, then expand all points in parallel as a batch
  • Graph-based and fork-based parallel decoding (SGD, APAR)
  • SGLang programs: share the KV cache of common prefixes automatically
  • Model cascades (FrugalGPT): try a small model first, call the big one only when needed
Ready
92

compress a long RAG promptsurvey §4.1.1 §4.1.4

  • Build a long-context QA test (many retrieved chunks)
  • Apply LLMLingua-style pruning at 2×, 5× and 10× compression
  • Measure tokens, TTFT and answer accuracy at each ratio
  • Compare with better retrieval (fewer, better chunks)
Ready

Model-level 1: efficient structure design

Cheaper FFNs (MoE, sharing), cheaper attention (GQA, low-rank, kernels), and alternatives to the transformer. Survey §5.1.

93

Where the cost is: FFN and attention, by the numberssurvey §5.1

  • The survey reports the FFN holds 63% of LLaMA-7B parameters and 72% of LLaMA-70B parameters
  • FFN cost grows with model size; attention cost grows with context length
  • Compute your own model’s split: params, FLOPs and bytes per token
  • Which part to optimize for your workload (short chat vs long documents vs video)
Ready
94

Efficient FFN 1: SwiGLU and parameter sharing (ALBERT)survey §5.1.1

  • SwiGLU: better quality per parameter; why d_{ff} becomes about 8d/3
  • ALBERT: share one layer’s weights across all layers
  • ALBERT: factorized embeddings (|V|×E plus E×d instead of |V|×d)
  • What sharing saves (memory) and what it does not save (compute)
Ready
95

Efficient FFN 2: MoE routing in depthsurvey §5.1.1

  • Top-k token-choice routing and the capacity factor
  • Load imbalance: why it wastes compute and padding
  • Fixes: load-balancing loss (Switch), router z-loss, BASE layers, expert-choice routing
  • Shared experts and fine-grained experts (DeepSeekMoE style)
Ready
96

Efficient FFN 3: MoE from dense models, and MoE inferencesurvey §5.1.1

  • MoEfication: split a dense FFN into experts
  • Sparse upcycling: copy a dense FFN into many experts, then train
  • Inference: total vs active parameters; memory holds all experts
  • Batching and expert parallelism at serving time; Mixtral and DeepSeek as examples
Ready
97

Efficient attention 1: MQA, GQA and MLA, from the KV-cache viewsurvey §5.1.2

  • Share K and V across heads (MQA) or groups of heads (GQA)
  • Uptraining: turn an MHA checkpoint into GQA by mean-pooling heads
  • MLA: store one small latent vector instead of full K and V
  • KV bytes per token and quality, side by side
Ready
98

Efficient attention 2: low-rank attention (Linformer, Luna, Set Transformer)survey §5.1.2

  • The attention matrix is often close to low rank
  • Linformer: project K and V from length n to a fixed k
  • Luna and Set Transformer: a small set of learned queries summarizes the sequence (pack, then unpack)
  • Limits: causal masking and variable lengths are hard
Ready
99

Efficient attention 3: kernel-based attention (Linear Transformer, Performer, RFA)survey §5.1.2

  • Replace softmax(qᵀk) with φ(q)ᵀφ(k)
  • Use associativity: compute φ(K)ᵀV once (d×d), then multiply by φ(Q)
  • Performer (FAVOR+) and RFA: random features that approximate softmax
  • Why quality drops: no sharp, peaked attention
Ready
100

Transformer alternatives 1: state space models from HiPPO to S4survey §5.1.3

  • The linear state space recurrence
  • HiPPO: compress history into polynomial coefficients
  • S4, DSS, S4D, S5: diagonal matrices and fast convolution
  • Two views of one model: recurrence (fast decode) and convolution (parallel training)
Ready
101

Transformer alternatives 2: Mamba, RWKV, RetNet, Hyena and hybridssurvey §5.1.3

  • Mamba: input-dependent (selective) A, B, C with a parallel scan
  • RWKV and RetNet: attention-like training, RNN-like decoding
  • Hyena: long convolutions with data-dependent kernels
  • Weakness at exact recall; hybrids such as Jamba mix attention and SSM layers
Ready
102

Linformer, Performer and GQA against full attentionsurvey §5.1.2

  • Implement the three in one small GPT
  • Measure speed and memory for sequence lengths 512 to 16k
  • Measure quality (validation loss) and a copy task that needs exact recall
  • Explain each result with the formulas
Ready

Model-level 2: quantization in depth

The math of quantization, weight-only and weight–activation methods, rotations, KV-cache quantization and QAT. Survey §5.2.1.

103

Quantization math: scale, zero point, granularity and errorsurvey §5.2.1

  • Uniform affine quantization; symmetric vs asymmetric
  • Granularity: per-tensor, per-channel, per-group (for example 128), per-token
  • Rounding error vs clipping error; choosing the range (min-max, percentile, MSE)
  • Why LLMs are hard: outliers and wide ranges
Ready
104

Where quantization helps: weight-only vs weight–activationsurvey §5.2.1

  • Decode is GEMV and memory-bound: weight-only (W4A16, W8A16) helps
  • Prefill is GEMM and compute-bound: you need W8A8 or FP8 to use faster tensor cores
  • Survey test (A100, AWQ W4A16): decode about 2.0–2.4× faster at batch 1 and 1.2–1.4× at batch 16; prefill about 0.8–1.1×, often slower
  • The gain shrinks as batch and context grow; bigger models gain more
Ready
105

Number formats: INT8, INT4, FP8, FP4, NF4 and microscaling

  • FP8 E4M3 vs E5M2; per-tensor scaling
  • FP4 and block-scaled formats (MX): a shared scale per small block
  • NF4 for normally distributed weights (QLoRA)
  • Hardware: which formats each GPU generation runs on tensor cores (for example, FP8 on L4 and H100) [check per GPU]
Ready
106

Weight PTQ 1: from OBQ to GPTQsurvey §5.2.1

  • Layer-wise objective: keep WX close to ŴX on calibration data
  • The Hessian H = 2XXᵀ tells which weights matter
  • Quantize one column, then fix the remaining columns to cancel the error
  • GPTQ’s speed tricks: fixed order, lazy batch updates, Cholesky
Ready
107

Weight PTQ 2: AWQ, outliers and mixed precisionsurvey §5.2.1

  • AWQ: protect weights connected to large activations by scaling them first
  • Search the scale per channel to minimize output error
  • Keep outliers in high precision: SpQR, OWQ; non-uniform levels: SqueezeLLM
  • Look-up-table kernels for non-uniform formats (LUT-GEMM)
Ready
108

Weight–activation quantization: LLM.int8(), SmoothQuant, ZeroQuant, OmniQuantsurvey §5.2.1

  • Activation outliers live in a few channels
  • LLM.int8(): run outlier channels in FP16, the rest in INT8
  • SmoothQuant: move difficulty from activations to weights with a per-channel scale
  • ZeroQuant (group and token scales), OmniQuant (learned clipping), Atom (mixed precision)
Ready
109

Rotations against outliers: QuIP, QuIP#, QuaRot, SpinQuantsurvey §5.2.1

  • Rotate weights and activations; the output does not change
  • A rotation spreads one big outlier over many channels
  • QuaRot: random Hadamard rotations, enables 4-bit weights, activations and KV
  • SpinQuant: learn the rotation on a small dataset
Ready
110

KV-cache quantization: KIVI and KVQuantsurvey §5.2.1

  • The KV cache can be bigger than the weights at long context or big batch
  • Keys have channel outliers: quantize keys per channel, values per token (KIVI)
  • Non-uniform types and pre-RoPE keys (KVQuant)
  • Measure: long-context accuracy, not only perplexity
Ready
111

Quantization-aware training: fake quant, STE, LLM-QAT, QLoRA, LoftQsurvey §5.2.1

  • Fake quantization in the forward pass; straight-through estimator in the backward pass
  • LLM-QAT: data-free QAT by distilling from the FP16 model on its own generated text
  • QLoRA, QA-LoRA and LoftQ: low-bit base weights plus trainable low-rank adapters
  • When PTQ is enough and when QAT is worth the cost
Ready
112

compare RTN, GPTQ, AWQ, SmoothQuant and FP8 on one modelsurvey §5.2.1

  • Quantize a 7B–8B model with each method
  • Measure perplexity, two tasks, prefill latency and decode latency on an L4
  • Repeat at batch 1 and batch 16; compare with the survey’s table
  • Write a one-page recommendation for your production model
Ready

Model-level 3: sparsity, factorization, distillation and dynamic inference

Pruning weights and attention, low-rank factorization, knowledge distillation and early exit. Survey §5.2.2–5.2.5.

113

Pruning basics: magnitude, OBD and OBS; structured vs unstructuredsurvey §5.2.2

  • Magnitude pruning and its limits
  • Second-order pruning: OBD and OBS use the Hessian to choose and fix weights
  • Unstructured sparsity: high sparsity, little real speedup on GPUs
  • Structured sparsity (channels, heads, layers): real speedup, bigger accuracy loss
Ready
114

Unstructured and 2:4 pruning: SparseGPT, Wanda, OWLsurvey §5.2.2

  • SparseGPT: OBS-style pruning that scales to large models
  • Wanda: score = |weight| × norm of its input activation; no weight update
  • OWL: different sparsity per layer, based on outliers
  • 2:4 semi-structured sparsity: NVIDIA tensor cores skip half the weights
Ready
115

Structured pruning: LLM-Pruner, Sheared LLaMA, SliceGPT, Minitronsurvey §5.2.2

  • Find coupled structures (heads, channels) and remove them together
  • Sheared LLaMA: prune to a target shape, then continue pretraining
  • SliceGPT: rotate, then delete whole rows and columns
  • Prune + distill (Minitron style); pruning experts in MoE models
Ready
116

Sparse attention 1 (static): Sparse Transformer, Longformer, BigBird, StreamingLLMsurvey §5.2.2

  • Local (window), global, random and dilated patterns
  • BigBird: local + global + random keeps the model’s power
  • StreamingLLM: a window plus the first few tokens (attention sinks)
  • Static patterns also shrink the KV cache during decode
Ready
117

Sparse attention 2 (dynamic): Reformer, Routing Transformer, SpAtten, H2Osurvey §5.2.2

  • Bucket tokens by hashing (Reformer) or clustering (Routing Transformer)
  • Prune unimportant tokens and heads on the fly (SpAtten)
  • H2O: keep recent tokens plus heavy hitters; evict the rest from the KV cache
  • Risk: once evicted, information is gone
Ready
118

Structure optimization: NAS and low-rank factorizationsurvey §5.2.3

  • Neural architecture search for transformers, and why it is costly for LLMs
  • Low-rank factorization with SVD
  • Activation-aware SVD (ASVD, SVD-LLM)
  • Mixing factorization with quantization and pruning
Ready
119

Distillation for LLMs 1: white-box (logits and features)survey §5.2.4

  • Forward KL (mode-covering) vs reverse KL (mode-seeking)
  • MiniLLM: reverse KL for generation
  • GKD: train on the student’s own outputs (on-policy distillation)
  • Feature and layer distillation; the capacity gap
Ready
120

Distillation for LLMs 2: black-box (outputs only)survey §5.2.4

  • Distilling from API models: you only see the text
  • Distilling in-context learning and instruction following
  • Chain-of-thought distillation: Distilling step-by-step
  • Reasoning distillation into small models (DeepSeek-R1 style); filtering bad rationales
Ready
121

Dynamic inference: early exit and layer skippingsurvey §5.2.5

  • Sample-level early exit for classification: DeeBERT, FastBERT, PABEE
  • Token-level early exit for generation: CALM, SkipDecode
  • The KV-cache problem: skipped layers have no K and V for later tokens
  • Self-speculation with early-exit layers (LayerSkip)
Ready
122

prune with Wanda (2:4), recover with distillationsurvey §5.2.2 §5.2.4

  • Prune a small LLM to 50% (unstructured) and to 2:4
  • Measure perplexity and real speed with sparse kernels
  • Recover quality with a short distillation run
  • Report the trade-off curve
Ready

System-level 1: the inference engine

Operator and graph optimization, FlashAttention and FlashDecoding, speculative decoding, offloading, TensorRT-LLM. Survey §6.1.

123

Runtime profiling: where the time goessurvey §6.1.1

  • The survey’s profiling: attention and linear operators often take more than 75% of the time
  • Many tiny kernels: launch overhead and CPU gaps
  • Read a PyTorch profiler and an Nsight Systems trace
  • Find the top 3 time sinks of a real model
Ready
124

Attention operator 1: FlashAttention 1, 2 and 3survey §6.1.1

  • Standard attention writes the T×T matrix to HBM; that is the slow part
  • Tiling + online softmax: attention block by block, inside SRAM
  • Recompute in backward instead of storing; FA2 work split; FA3 on Hopper (asynchrony, FP8)
  • Numerics: small differences from standard attention
Ready
125

Attention operator 2: decode attention, FlashDecoding and FlashDecoding++survey §6.1.1

  • Decode has one query row: not enough parallel work for the GPU
  • FlashDecoding: split the KV sequence, compute parts in parallel, merge with log-sum-exp
  • FlashDecoding++: unified max value for softmax, flat-GEMM tuning
  • Paged-KV attention kernels
Ready
126

Linear operators: GEMM, GEMV and flat GEMMsurvey §6.1.1

  • Prefill = GEMM (big matrices); decode = GEMV or flat GEMM (very thin matrices)
  • Tiling, split-K and padding waste for thin shapes
  • Quantized GEMM: dequantize inside the kernel (W4A16)
  • MoE grouped GEMM
Ready
127

Graph-level optimization: fusion, CUDA graphs, torch.compilesurvey §6.1.1

  • Fuse small ops (bias, activation, norm, residual) into one kernel
  • CUDA graphs remove CPU launch overhead in decode
  • torch.compile: graph capture, fusion, graph breaks
  • How TensorRT and inference engines fuse whole transformer layers
Ready
128

Speculative decoding 1: the mathsurvey §6.1

  • A cheap draft proposes γ tokens; the target model checks them all in one pass
  • The acceptance rule keeps the target model’s exact distribution
  • Expected tokens per step from the acceptance rate α
  • When speculation does not pay: big batches, weak drafts
Ready
129

Speculative decoding 2: drafts and verification (Medusa, EAGLE, Lookahead, SpecInfer)survey §6.1

  • Separate draft model vs self-drafting heads (Medusa) vs feature-level drafting (EAGLE)
  • Lookahead and prompt-lookup decoding: no draft model
  • Tree verification: check many candidate branches at once (SpecInfer)
  • Multi-token prediction heads used as drafters (MTP)
Ready
130

Offloading: FlexGen, llama.cpp, PowerInfersurvey §6.1

  • When the model does not fit on the GPU: move weights or KV to CPU or SSD
  • FlexGen: plan where each tensor lives to maximize throughput
  • llama.cpp: split layers between CPU and GPU
  • PowerInfer: keep “hot” neurons on the GPU, “cold” ones on the CPU
Ready
131

TensorRT and TensorRT-LLM in depth

  • TensorRT: ONNX or network → optimized engine; layer fusion, kernel tuning, FP16, INT8, FP8
  • Optimization profiles for dynamic shapes; an engine is tied to a GPU and a version
  • TensorRT-LLM: paged KV, in-flight batching, quantization, speculative decoding, plugins
  • When TensorRT-LLM beats vLLM, and when it is not worth the effort
Ready
132

profile, fuse and speculatesurvey §6.1.1 §6.1

  • Profile one decode step with Nsight Systems
  • Fuse RMSNorm + residual in Triton and measure
  • Run speculative decoding with a small draft and with EAGLE-style heads
  • Measure acceptance rate and tokens/s at batch 1, 4 and 16
Ready

System-level 2: serving systems

Memory management, batching, scheduling, distributed and disaggregated serving, engines, hardware and scenarios. Survey §6.2–7.

133

The serving stack and the life of a request

  • Engine (vLLM, TensorRT-LLM, SGLang) → server (OpenAI API, Triton) → router → autoscaler
  • Life of a request: tokenize, queue, schedule, prefill, decode, detokenize, stream
  • Streaming, cancellation, timeouts
  • Serving many models and many LoRA adapters
Ready
134

Memory management: PagedAttention, prefix caching, RadixAttentionsurvey §6.2

  • Fragmentation with naive KV allocation
  • Paging: fixed-size blocks and a block table, like virtual memory
  • Sharing blocks across requests; copy-on-write for parallel sampling
  • Prefix caching and RadixAttention for repeated system prompts and documents
Ready
135

Batching: continuous batching, chunked prefill, split-fusesurvey §6.2

  • Iteration-level (continuous) batching: requests join and leave every step (ORCA)
  • Chunked prefill: mix prefill chunks with decode steps (Sarathi, Sarathi-Serve)
  • Split-fuse in DeepSpeed-FastGen
  • Effect on TTFT, TPOT and throughput
Ready
136

Scheduling: FCFS, preemption, fairness and SLOssurvey §6.2

  • Head-of-line blocking with first-come-first-served
  • FastServe: preemptive multi-level feedback queue
  • Fairness between users: Virtual Token Counter (VTC)
  • SLO-aware scheduling and output-length prediction
Ready
137

Distributed serving and prefill–decode disaggregationsurvey §6.2

  • Tensor and pipeline parallel serving
  • Disaggregation: prefill GPUs and decode GPUs (Splitwise, DistServe, TetriInfer)
  • Moving the KV cache between GPUs; cost of transfer
  • Serving on cheap spot instances (SpotServe)
Ready
138

vLLM internals

  • Scheduler and block manager; preemption by recompute or swap
  • CUDA graphs for decode; tensor-parallel and pipeline-parallel serving
  • Prefix caching, chunked prefill and speculative decoding inside vLLM
  • Key settings: max_num_seqs, max_model_len, gpu_memory_utilization
Ready
139

Triton Inference Server, SGLang, LMDeploy, TGI, llama.cppsurvey §6.2

  • Triton Inference Server: model repository, config.pbtxt, backends (TensorRT, TensorRT-LLM, ONNX, PyTorch, Python), ensembles
  • SGLang and LMDeploy: what each does best
  • The survey compares inference engines and serving systems; how to read such tables
  • How to benchmark engines fairly on your own traffic
Ready
140

Hardware accelerators and edge deploymentsurvey §6.3

  • GPUs vs FPGAs (FlightLLM) vs NPUs and other accelerators
  • Edge: memory, power and bandwidth limits
  • llama.cpp, MLX and mobile runtimes
  • Which survey techniques matter most at the edge
Ready
141

Scenarios and capacity planning: agents, long context, edge, costsurvey §7

  • Agents and multi-model pipelines: many calls, shared prefixes
  • Long context: prefill cost, KV memory, sparse attention
  • From traffic to GPUs: prompt and output lengths, latency target
  • Worked example: one model on an L4 vs an H100; cost per million tokens
Ready
142

benchmark a serving systemsurvey §6.2 §6.2

  • Serve a model with vLLM (optional: TensorRT-LLM on an L4)
  • Plot TTFT, TPOT and throughput vs load and context length
  • Turn on prefix caching, chunked prefill, FP8 or INT4, and speculative decoding, one at a time
  • Explain each change with the survey’s taxonomy
Ready