LLM & VLM courseBlog
139Track 5: LLM inference and serving · session 62 of 65

Triton Inference Server, SGLang, LMDeploy, TGI, llama.cpp

Five serving tools you will meet in real systems, each seen through the problem it solves best: Triton Inference Server for many models behind one API, SGLang for shared prefixes and structured output, LMDeploy for 4-bit weights and small KV caches, TGI as a clean design now in maintenance, and llama.cpp for laptops and CPUs. Then two skills that outlast any tool: reading a comparison table critically, and benchmarking engines fairly by goodput on your own traffic.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: IntermediateBefore this: Sessions 133 (the serving stack), 134 (memory management and RadixAttention), 135 (continuous batching), 131 (TensorRT-LLM) and 81 (inference metrics)

What you will learn

  • What a Triton model repository contains (config.pbtxt, version folders, backends, instance groups) and how ensembles and BLS build pipelines inside the server
  • How Triton's dynamic batcher forms batches, the batch size it settles at (b* = λα/(1 − λβ)), and why it is the wrong tool for an LLM
  • What SGLang does best: RadixAttention prefix reuse, constrained decoding with a compressed state machine, and an overlap scheduler
  • What LMDeploy does best: TurboMind, 4-bit weight-only inference, and online INT8/INT4 KV caches; and what TGI's design was and why it is now in maintenance mode
  • How llama.cpp's GGUF block types set the bits per weight, the file size and the bandwidth bound on decode speed
  • How to read the survey's comparison of inference engines and serving systems, and the five questions to ask of any such table
  • How to benchmark engines fairly: same setup, your own trace, open-loop load, and goodput at your SLO

Why this matters

You know how one LLM engine works inside: the KV cache, paging, continuous batching, schedulers. In practice you must also choose an engine, and many exist. This session looks at five that you will meet in real systems, each from the point of view “what problem does it solve best?”: NVIDIA Triton Inference Server, SGLang, LMDeploy, Hugging Face TGI and llama.cpp. Then it teaches two skills that matter more than any single tool: how to read a comparison table, and how to benchmark engines fairly on your own traffic.

In session 133 you split a serving system into layers: the API gateway, the router, the engine and the model runtime. In session 131 you studied one engine from NVIDIA, and session 138 is the reference design most others copy. The tools here sit at different layers. Triton is a general model server: it serves any model (a ViT, a tokenizer, an LLM) behind one API and lets an LLM engine do the batching. SGLang, LMDeploy and TGI are LLM serving engines like vLLM. llama.cpp is an engine for local and CPU-heavy inference, with its own file format.

The efficient-inference survey (Zhou et al., Section 6, our source for the system-level map) compares inference engines and serving systems in one table (its Table VI). We read that table carefully later, because the way it measures things is typical of every comparison you will see, including vendor blog posts.

Two kinds of system: model servers and LLM engines

Two words first. An inference engine runs a model fast: kernels, KV cache, batching inside one forward pass. A serving system takes requests from many users over the network, queues them, schedules them on the engine, and returns results. The survey uses exactly this split: its table has an “inference” half (attention, linear and graph optimizations, speculative decoding, measured in tokens/s for one request) and a “serving” half (memory management, batching, scheduling, measured in requests/s). [Certain]

SystemLayerWritten inBest at
Triton Inference Server (now “NVIDIA Dynamo-Triton”)general model serverC++ core, many backendsserving many different models (CNN, ViT, ONNX, Python logic, LLMs) behind one API, with pipelines (ensembles) and a dynamic batcher
SGLangLLM engine + serverPython scheduler, CUDA/Triton kernelsprefix reuse (RadixAttention), structured output, multi-call programs, large MoE models
LMDeployLLM engine + serverC++/CUDA engine (TurboMind) and a PyTorch enginefast 4-bit weight-only (AWQ) inference and online KV-cache quantization
Text Generation Inference (TGI)LLM serverRust router, Python model server, gRPCwas Hugging Face’s production server; now in maintenance mode
llama.cpplocal engine + small serverC/C++ (ggml), no Pythonlaptops, Apple silicon, CPUs, consumer GPUs; quantized GGUF files; CPU+GPU split

Two status notes, because tutorials online are often out of date. On 18 March 2025 NVIDIA renamed Triton Inference Server to NVIDIA Dynamo-Triton and put it under the Dynamo platform; NVIDIA says it continues to support it. Dynamo itself is a separate, newer framework for distributed LLM serving (with vLLM, SGLang or TensorRT-LLM as engines). [Certain] The TGI repository says it “is now in maintenance mode”, was archived on 21 March 2026, and recommends vLLM, SGLang, llama.cpp or MLX instead. [Certain] Do not confuse Triton Inference Server with the Triton language for writing GPU kernels (session 86): same name, different projects.

Triton Inference Server

The model repository

Triton serves what it finds in a model repository: a directory with one folder per model. Each model folder holds a config.pbtxt (the model configuration, in protobuf text format) and at least one numbered version folder with the model file. Folders that are not numbers, or that start with 0, are ignored. [Certain] The default file name depends on the backend: model.plan for TensorRT, model.onnx for ONNX Runtime, model.pt for PyTorch (TorchScript), model.py for the Python backend, and model.xml plus model.bin for OpenVINO. [Certain]

A minimal configuration gives the platform or backend, max_batch_size, and the input and output tensors. [Certain] A real one for a ViT encoder:

# model_repository/vit_encoder/config.pbtxt
name: "vit_encoder"
platform: "tensorrt_plan"            # or backend: "onnxruntime", "pytorch", "python", "tensorrtllm", "vllm"
max_batch_size: 32                   # 0 means: the model does not support batching
input  [ { name: "pixels"    data_type: TYPE_FP16 dims: [ 3, 224, 224 ] } ]
output [ { name: "embedding" data_type: TYPE_FP16 dims: [ 768 ] } ]
instance_group [ { count: 2 kind: KIND_GPU } ]   # two copies of the model on each GPU
dynamic_batching { max_queue_delay_microseconds: 2000 }
version_policy: { latest: { num_versions: 1 } }  # the default

The dims do not include the batch dimension: with max_batch_size > 0, Triton adds it in front. The rules for the other fields, from the Triton docs: [Certain]

  • Version policy. latest (with \(n\) versions), all, or specific. With no policy, the latest single version is served. So you can deploy version 3 by adding a folder, and roll back by removing it.
  • Instance groups. How many copies (instances) of the model run, and where (KIND_GPU or KIND_CPU). The default for GPUs is one instance on each GPU. Two instances on one GPU let one copy compute while the other copies inputs or runs CPU-side work.
  • Schedulers. The default scheduler sends each request as it is. The dynamic batcher combines requests into a batch on the server. The sequence batcher is for stateful models, where all requests of one sequence must go to the same instance. The ensemble scheduler runs a pipeline of models.

Backends

A backend is the library that actually executes a model. Triton’s core does networking, queueing, batching and metrics; backends do the math. The list includes TensorRT, ONNX Runtime, PyTorch (TorchScript and PyTorch 2), TensorFlow, OpenVINO, Python, DALI (data preprocessing on GPU), FIL (tree models such as XGBoost), TensorRT-LLM and vLLM. [Certain] The Python backend is the glue: you write a class TritonPythonModel with initialize, execute(requests) and finalize, and you can run any Python there, for example a tokenizer. With BLS (business logic scripting), a Python model can call other models in the same server, with loops and conditions an ensemble cannot express. [Certain]

Ensembles: a pipeline inside the server

An ensemble is a model with platform: "ensemble" and no weights. Its configuration lists steps; each step names a model and maps tensor names between steps. A request to the ensemble runs the whole pipeline inside the server, so intermediate tensors never go back over the network. [Certain] The standard LLM pipeline of the TensorRT-LLM backend is exactly this: preprocessing (Python, tokenizer) → tensorrt_llm → postprocessing (Python, detokenizer), wrapped in ensemble. For token streaming, the LLM model is decoupled: it may send zero, one or many responses for one request. [Certain for decoupled models; Likely for the exact folder names in your version]

Figure 1. A Triton model repository with a TensorRT ViT encoder and an LLM pipeline (an ensemble of preprocessing, TensorRT-LLM and postprocessing), and the path of one streamed chat request through the server.
How to read Figure 1

Step through. The left side is what you put on disk; the highlighted lines are the files that matter at each step. The right side is the server at run time. Boxes in the dashed frame belong to the ensemble: one request from the client runs all three models in order. The ViT encoder is a separate model that a VLM pipeline could call in another ensemble step; it uses Triton’s own dynamic batcher (Figure 2), while the LLM step batches inside its backend.

The dynamic batcher, with numbers

For a fixed-shape model like a ViT encoder or an embedding model, batching on the server is the main way to use the GPU well. The dynamic batcher keeps a queue per model. When an instance is free, it forms a batch from the oldest requests: if it can reach a preferred batch size it takes the largest one, otherwise the largest batch up to max_batch_size; if max_queue_delay_microseconds is set, it may wait that long for more requests, and “if the delay expires the dynamic batcher sends the batch as is”. The default configuration does not wait. [Certain]

Model one instance with a linear cost per batch. A batch of \(b\) requests takes \(t(b) = \alpha + \beta b\): \(\alpha\) is the fixed part (reading weights, launching kernels) and \(\beta\) the cost of one more input. Then

\[ X(b) = \frac{b}{\alpha + \beta b}, \qquad X_{\max} = \frac{B_{\max}}{\alpha + \beta B_{\max}} \;\lt\; \frac{1}{\beta} \tag{139.1} \]
Read the symbols
\(X(b)\)
throughput of one instance in requests per second when every batch has \(b\) requests
\(\alpha, \beta\)
fixed cost of one batch, and the extra cost per request; we use \(\alpha = 4\) ms, \(\beta = 0.5\) ms (our assumption for a ViT encoder) [Likely]
\(B_{\max}\)
max_batch_size in the configuration

With zero delay, the batch size adjusts itself to the load. While a batch of size \(b\) runs, \(\lambda\, t(b)\) new requests arrive, and they form the next batch. The steady state is the fixed point \(b = \lambda(\alpha + \beta b)\):

\[ b^{*} = \frac{\lambda \alpha}{1 - \lambda \beta} \quad (\text{if } \lambda\beta \lt 1 \text{ and } b^{*} \le B_{\max}), \qquad \text{stable only if } \lambda \lt X_{\max} \tag{139.2} \]
Read the symbols
\(\lambda\)
arrival rate in requests per millisecond (1,000 req/s is \(\lambda = 1\))
\(b^{*}\)
the batch size the system settles at, with no delay configured
Worked example: one ViT encoder instance, \(\alpha = 4\) ms, \(\beta = 0.5\) ms

Without batching (max_batch_size: 1) the capacity is \(1/4.5\ \text{ms} = 222\) req/s. With \(B_{\max} = 32\): \(32 / (4 + 16)\ \text{ms} = 1{,}600\) req/s, 7.2× more. At 1,000 req/s the fixed point is \(b^{*} = 1 \cdot 4 / (1 - 0.5) = 8\) and one batch takes 8 ms; our simulation of 3,000 Poisson arrivals gives a mean batch of 8.00, mean latency 12.78 ms, p99 20.6 ms. The same traffic with no batching is unstable: the queue grows for the whole run (mean latency 5.3 s after 3,000 requests). At 100 req/s, a 5 ms delay raises the mean batch from 1.09 to 1.49 and lowers GPU busy time from 42% to 32%, but the mean latency rises from 5.49 to 9.06 ms.

So, in this model, the delay does not raise the maximum throughput; it trades latency for fewer, larger batches. That is useful when the GPU is shared with other models (Triton often hosts many) or when a larger batch is much faster per item than the linear model says (for example a TensorRT engine tuned for batch 8). It is why the Triton docs recommend: start with dynamic_batching { }, measure, then raise the delay only while latency stays inside the budget. [Certain for the procedure]

arrival / latencya batch running on the GPUbatch size, GPU busy
Figure 2. A simulation of Triton’s dynamic batcher for one model instance: 3,000 requests with Poisson arrivals (seed 139), batch cost \(t(b) = 4 + 0.5\,b\) ms, formulas (139.1) and (139.2). Latency is queue time plus compute time.
How to read Figure 2

The ticks are arrivals, the bars are batches on the GPU with their size. Move the rate up: batches grow by themselves, even with zero delay, and the readout shows that the mean batch matches \(b^{*}\). Choose max batch 1 and a rate above 222 req/s: the system is unstable and latency explodes. Add a delay at low load: batches grow, GPU busy time falls, latency rises. At high load the delay changes almost nothing, because the queue is never empty.

For measurement, Triton ships perf_analyzer (load generator), Model Analyzer (searches configurations such as instance count and batch delay) and an LLM benchmarking tool (GenAI-Perf, whose role is being taken over by NVIDIA’s newer AIPerf). [Certain for perf_analyzer and Model Analyzer; Likely for the AIPerf transition] Metrics are on port 8002 in Prometheus format, next to HTTP on 8000 and gRPC on 8001. [Certain]

SGLang: reuse, structure and programs

SGLang (Zheng et al., 2023) started as two things together: a small frontend language embedded in Python for programs that make many LLM calls, and a runtime that executes them fast. The paper reports up to 6.4× higher throughput than other systems on agent, reasoning, few-shot, JSON, RAG and multi-turn chat workloads. [Certain that the paper reports it; the number is for their setup and versions] Today most people use the runtime alone, as an OpenAI-compatible server, like vLLM. You met its key idea, RadixAttention, in session 134 and its programming model in session 91; here we look at what makes it the right choice.

1. RadixAttention: the KV cache is a tree

SGLang keeps the KV cache of finished and running requests in a radix tree keyed by token ids. A new request walks the tree, reuses the longest cached prefix, and only computes the rest. When memory is short, leaves are evicted in LRU order. This is on by default (disable_radix_cache = False in the current server arguments). [Certain] The default scheduling policy is now fcfs; lpm (longest prefix match first, the paper’s cache-aware order) is an option. [Certain, from the source of the current main branch] If \(n\) requests share a prefix of \(S\) tokens and each adds \(u\) own tokens, the prefill work is

\[ P_{\text{no cache}} = n(S + u), \qquad P_{\text{radix}} = S + n u, \qquad \frac{P_{\text{no cache}}}{P_{\text{radix}}} \to \frac{S+u}{u} \ \text{ for large } n \tag{139.3} \]
Worked example: few-shot classification

A 2,000-token prompt (instructions and examples) is shared by 64 requests, each with a 100-token question. Without reuse: \(64 \cdot 2{,}100 = 134{,}400\) prefill tokens. With the radix cache: \(2{,}000 + 64 \cdot 100 = 8{,}400\), which is 16× less prefill. The same structure appears in agents (a long system prompt and tool list), multi-turn chat (the history), and tree search (many branches of one reasoning trace).

2. Structured output with a compressed state machine

To force JSON that follows a schema, an engine turns the schema into a finite state machine (FSM) over characters and, at each step, masks every token that would leave the FSM. Much of a JSON answer is fixed text: braces, quotes, key names. SGLang’s paper compresses chains of states that have only one possible next string into one edge, and when decoding reaches such an edge it appends the whole string at once and continues (“jump-forward”). [Certain for the paper] If the answer has \(F\) forced tokens in \(J\) runs and \(G\) free tokens:

\[ \text{passes}_{\text{token by token}} = F + G, \qquad \text{passes}_{\text{jump-forward}} \approx G + J \tag{139.4} \]

For example, \(F = 12\) forced tokens in \(J = 3\) runs and \(G = 8\) free tokens need 20 passes token by token, about 11 with jump-forward (an illustration, not a measurement). The appended text is processed like a short prefill, which is cheap next to a decode step for every token. One detail matters: the forced string must be re-tokenized together with the text before it, because the tokenizer may merge characters across the boundary differently. [Likely] Grammar backends change between versions (xgrammar is a common default today), so check what your version enables. [Likely]

Algorithm 1: one decode step with grammar-constrained decoding and jump-forward
  1. input: the request state \(s\) in the compressed FSM, the text so far, logits of the model
  2. if the edge out of \(s\) is a single forced string \(c\) then
  3.   append \(c\) to the text, re-tokenize the end of the text, and schedule the new tokens as an extend (prefill) step // formula (139.4)
  4.   move \(s\) to the end of the edge; continue
  5. mask the logits: keep only tokens whose characters stay inside the FSM from \(s\)
  6. sample a token \(x\) from the masked distribution; append \(x\); move \(s\) along the characters of \(x\)
  7. if \(s\) is an accepting state and \(x\) is end-of-sequence then finish
  8. output: a string that is guaranteed to match the schema

3. A scheduler that hides its own cost

At small models and large batches, the CPU work of scheduling (building the batch, updating the radix tree, preparing inputs) can take a large part of each step. SGLang’s overlap scheduler prepares batch \(k+1\) on the CPU while the GPU runs batch \(k\); it is on by default (disable_overlap_schedule = False). [Certain] Other features you will see in its docs: chunked prefill, CUDA graphs for decode, speculative decoding (EAGLE), prefill-decode disaggregation, data-parallel attention and expert parallelism for large MoE models such as DeepSeek, and a router that sends requests to the replica most likely to have their prefix cached. [Likely for the details of each in your version]

When to pick SGLang. When traffic shares long prefixes (agents, few-shot, multi-turn chat, RL rollouts with many samples per prompt), when you need structured output at high speed, or when you serve very large MoE models on many GPUs. When traffic has no shared prefixes, vLLM and SGLang implement much the same ideas, and your benchmark should decide.

LMDeploy: 4-bit weights and a small KV cache

LMDeploy (from the InternLM team at Shanghai AI Lab) has two engines. TurboMind is a C++/CUDA engine, which “strives for ultimate optimization of inference performance”; the PyTorch engine is written in Python to make new models and hardware (for example Huawei Ascend) easy to add. They support different models and data types. [Certain] Its README lists persistent batch (its name for continuous batching), a blocked KV cache (paging), dynamic split&fuse (chunked prefill), tensor parallelism and its own CUDA kernels, and claims up to 1.8× the request throughput of vLLM and 4-bit inference 2.4× faster than FP16. [Certain that the README claims it; not verified by us, and versions have moved since]

The independent numbers in the survey (Table IV, LLaMA-2 on one A100, W4A16 weight-only quantization) show where the 4-bit gain comes from. For LLaMA-2-7B in LMDeploy at batch 1 and input 128, the speedups over FP16 are 1.30 for prefill, 2.11 for decode and 2.09 end to end. At batch 16 with input 2,048, decode gains only 1.28. [Certain, from the table] This is exactly the rule from session 104: weight-only quantization helps memory-bound decode at small batch, and helps little in compute-bound prefill or at large batch.

LMDeploy’s other strength is online KV-cache quantization to INT8 or INT4, computed while serving with no calibration step. [Certain that it exists] The KV bytes per token (ignoring the small scales) are

\[ s_{\text{tok}} = 2\, n_{\text{layers}}\, n_{\text{kv}}\, d_{\text{head}}\, b_{\text{kv}}, \qquad N_{\text{tok}} = \frac{M_{\text{KV}}}{s_{\text{tok}}} \tag{139.5} \]
Read the symbols
\(n_{\text{layers}}, n_{\text{kv}}, d_{\text{head}}\)
number of layers, KV heads and head size; the 2 counts K and V
\(b_{\text{kv}}\)
bytes per stored number: 2 for BF16, 1 for INT8, 0.5 for INT4
\(M_{\text{KV}}, N_{\text{tok}}\)
memory given to the KV cache, and how many tokens fit in it
Worked example: Llama 3.1 8B, 40 GB of KV memory

\(s_{\text{tok}} = 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes in BF16, so 40 GB holds 305,175 tokens. INT8 halves it to 65,536 bytes (610,351 tokens), INT4 to 32,768 bytes (1,220,703 tokens), 4× more concurrent context than BF16, before scales. Whether accuracy holds at INT4 depends on the model and task: see session 110 and measure it.

When to pick LMDeploy. 4-bit AWQ models on NVIDIA GPUs (including older ones), long contexts where KV memory is the limit, and InternLM or InternVL models, which it supports early. [Likely for the last point]

TGI: an important design, now in maintenance

Text Generation Inference was Hugging Face’s production server and powered its inference endpoints. Its design is worth knowing because it shows a clean split of roles: a Rust launcher starts everything, a Rust router receives HTTP requests, queues them and forms continuous batches under a token budget, and one Python model server per GPU shard runs the model, connected to the router by gRPC. [Certain for the components] Its features: continuous batching, tensor parallelism, FlashAttention and PagedAttention kernels, quantization (bitsandbytes, GPTQ, AWQ, Marlin, FP8, EETQ), JSON and grammar constraints, speculation, token streaming with server-sent events, and an OpenAI-compatible Messages API. It also had backends for TensorRT-LLM, llama.cpp and AWS Neuron, and ran on NVIDIA, AMD, Intel, Gaudi, Inferentia and TPU. [Certain]

The repository is now in maintenance mode (archived 21 March 2026, only small fixes) and points users to vLLM, SGLang, llama.cpp or MLX. [Certain] So: you will still meet TGI in existing deployments and in older papers and benchmarks; for a new service, plan for a maintained engine.

llama.cpp: one file, any laptop

llama.cpp is a C/C++ inference engine built on the ggml tensor library, with no Python at run time. It runs on CPUs (with SIMD kernels), Apple silicon (Metal), NVIDIA and AMD GPUs, Vulkan and more. [Certain] You saw its CPU+GPU layer split (-ngl) in session 130. Here we look at the two other pieces you need: the GGUF file and the server.

GGUF: the model in one file

A GGUF file holds everything needed to run a model: a header, metadata as key-value pairs (architecture, hyper-parameters, the tokenizer and often the chat template), a list of tensors with their types and shapes, and the aligned tensor data, which can be memory-mapped. [Certain] Each tensor has its own quantization type. Types are built from blocks: a block stores a few scales plus packed low-bit codes for 32 or 256 weights. From the block sizes in ggml-common.h: [Certain]

\[ \text{bpw} = \frac{8 \cdot \text{bytes per block}}{\text{weights per block}}, \qquad \text{size} \approx \frac{P \cdot \overline{\text{bpw}}}{8}, \qquad r_{\text{decode}} \le \frac{\mathrm{BW}}{\text{size}} \tag{139.6} \]
Read the symbols
bpw
bits per weight, including the scales
\(P, \overline{\text{bpw}}\)
number of parameters, and the average bpw over all tensors
\(r_{\text{decode}}, \mathrm{BW}\)
tokens per second at batch 1, and memory bandwidth: each new token must read (almost) all the weights once
TypeWeights per blockBlock contentBytesbpw
Q8_032FP16 scale + 32 int8348.5
Q4_032FP16 scale + \(32 \times 4\) bits184.5
Q6_K256FP16 scale + 16 int8 sub-scales + \(256 \times 6\) bits2106.5625
Q5_K2562 FP16 + 12 bytes of 6-bit scales and mins + \(256 \times 5\) bits1765.5
Q4_K2562 FP16 + 12 bytes of 6-bit scales and mins + \(256 \times 4\) bits1444.5
Q3_K256FP16 scale + 12 bytes of scales + \(256 \times 3\) bits (\(2 + 1\) bit planes)1103.4375
Q2_K2562 FP16 + 16 bytes of 4-bit scales and mins + \(256 \times 2\) bits842.625

The “K” types use a 256-weight super-block with sub-blocks that have their own small scales, a two-level version of the group quantization in session 103. Names like Q4_K_M and Q4_K_S are mixes: most tensors in Q4_K, some sensitive ones (for example parts of the attention value and FFN down projections, and the output layer) in a larger type, so the average bpw is a bit above 4.5. [Likely for the exact tensor rules, which change between versions]

Worked example: Llama 3.1 8B (8,030,261,248 parameters)

F16: 16.06 GB. Q8_0: \(8.03 \cdot 8.5/8 = 8.53\) GB. Q4_K for every tensor: 4.52 GB. On a laptop with dual-channel DDR5-5600, bandwidth is \(5{,}600 \cdot 10^6 \cdot 8\ \text{bytes} \cdot 2 = 89.6\) GB/s, so decode is at most \(89.6/4.52 = 19.8\) tokens/s at Q4_K and 5.6 tokens/s at F16. On an Apple M2 Ultra (800 GB/s) the bound is 177.1 tokens/s at Q4_K. These are upper limits: real speeds are lower, and the embedding table (6.5% of the weights) is not read in full each step. The KV cache in F16 adds 128 KiB per token: 4.29 GB for 32,768 tokens.

weights (GGUF file)KV cache, F16
Figure 3. GGUF size and the batch-1 decode bound (139.6) for three Llama models, with every tensor in one type (real _M and _S files mix types). Device memory and bandwidth are nominal; for discrete GPUs that are too small, the readout splits layers with -ngl and assumes 89.6 GB/s host memory.
How to read Figure 3

The top bar is what must be in memory: weights plus the KV cache for the chosen context; the dashed line is the device memory. The lower bars give the speed limit for each type on the chosen device: halve the bits and the bound doubles. Choose the 70B model on the RTX 4090: it does not fit even at Q2_K, and the readout shows how slow the split becomes, because the host part runs at laptop bandwidth. Raise the context to 131,072 tokens for 70B: the F16 KV cache alone is 42.9 GB.

llama-server

llama-server exposes an OpenAI-compatible HTTP API. Requests run in slots (-np, the number of parallel sequences; the default is automatic), with continuous batching on by default, and a unified KV buffer shared by all slots. -ngl sets how many layers live on the GPU (default auto), and -c the context size (0 means: take it from the model). [Certain, from the current server README] This is enough for a few users on one machine, an edge box, or an offline batch job; it is not a replacement for a GPU cluster engine with paged attention and large batches. For measuring raw speed, llama-bench reports prompt processing and token generation rates separately.

How to read a comparison table

The survey’s Table VI is a good example to practice on. It compares seven systems on one NVIDIA A100 80 GB: inference throughput with Llama2-7B (batch 1, input 1k, output 128 tokens, speculative decoding off) and serving throughput on the ShareGPT dataset. [Certain] Some numbers from it: HuggingFace 38.963 tokens/s, vLLM 90.052, TensorRT-LLM 92.512, FlashDecoding++ 106.636; for serving, vLLM 7.11 req/s, TensorRT-LLM 5.87, LightLLM 10.29, and no serving number for HuggingFace, OpenPPL or FlashDecoding++.

Figure 4. The numbers of Table VI of the efficient-inference survey (Zhou et al., 2024), one A100 80 GB. The ticks on the right count the inference optimizations the table marks for each system (attention, linear, graph, speculative decoding).
How to read Figure 4

Switch the sort. The leader changes: FlashDecoding++ by tokens/s, LightLLM by requests/s, and TensorRT-LLM, with the most ticks, is last among the systems with a serving number. The two columns measure different things (kernels for one request, against memory management and scheduling under load), and neither is goodput under an SLO.

Five questions to ask of any such table, this one included:

  1. What exactly is measured? Batch-1 tokens/s measures kernel speed for one user. Maximum requests/s measures behaviour under overload. Neither says what happens at your load with your latency target.
  2. Is the gap larger than the noise? TensorRT-LLM’s 92.512 tokens/s is 2.7% above vLLM’s 90.052. One run, five significant digits, no error bars: this gap means “about the same”.
  3. Are the columns consistent? A system can win one column and lose the next (LightLLM is 1.75× TensorRT-LLM in requests/s, but 20% slower in tokens/s). Ask why: here, token-wise KV memory and the scheduler matter more than kernels under load.
  4. When, and which versions? Engines change monthly. A 2024 table says little about 2026 releases, and the table itself notes that TensorRT-LLM’s scheduler was not open source.
  5. Does it look right? In the survey’s Table IV (W4A16 speedups), the LLaMA-2-7B rows for batch 8 and batch 16 are identical in both the TensorRT-LLM and the LMDeploy half, cell by cell. That looks like a copy error. [Likely] Careful readers notice such things before quoting a number.

How to benchmark engines fairly on your own traffic

The goal is not “which engine is fastest” but “which engine serves my traffic at my latency target with the fewest GPUs”. The right number is goodput (DistServe’s term): the highest request rate at which a chosen fraction of requests meets every SLO (service-level objective), such as time to first token (TTFT) and time per output token (TPOT), defined in session 81.

\[ \text{goodput} = \max\Big\{ \lambda \ :\ \Pr\big[\text{TTFT} \le T_1 \ \wedge\ \text{TPOT} \le T_2 \,\big|\, \lambda\big] \ge q \Big\}, \qquad C = \lambda\, W \tag{139.7} \]
Read the symbols
\(\lambda\)
request arrival rate (requests per second)
\(T_1, T_2, q\)
the SLO targets for TTFT and TPOT, and the fraction of requests that must meet both (for example \(q = 0.9\))
\(C, W\)
Little’s law: average number of requests in the system equals rate times average time in the system

Little’s law explains a common trap. A closed-loop benchmark keeps \(C\) users busy: each sends its next request when the last one ends. If each request takes \(W = 8\) s, 64 users produce only \(\lambda = 64/8 = 8\) req/s, and if the server slows down, the load falls with it, so queueing is hidden. Real users do not wait for each other. Use an open-loop load (Poisson arrivals at a fixed rate) and sweep the rate.

Algorithm 2: a fair benchmark of serving engines
  1. input: a trace from production (prompt lengths, output lengths, shared prefixes, arrival pattern), SLOs \(T_1, T_2, q\), the candidate engines
  2. Fix everything that is not the engine: same model weights and revision, same precision (BF16 vs BF16, FP8 vs FP8), same GPUs and count, same parallelism, same tokenizer and chat template
  3. Fix the work: same max output tokens, same stop rules, same sampling; check that output lengths match (forcing them with ignore_eos changes the workload)
  4. Tune each engine fairly: max batch, max tokens per step, chunked prefill, KV memory fraction, CUDA graphs; give each the same tuning time
  5. Decide prefix caching on purpose: on for both if your traffic shares prefixes, and use a realistic hit rate, not the same prompt repeated
  6. for each rate \(\lambda\) in a sweep: warm up, send open-loop Poisson arrivals for several minutes, record TTFT and TPOT per request, check output quality on a sample
  7. Compute goodput (139.7) for each engine; repeat runs and report the spread
  8. output: goodput per GPU for your SLO, the latency curves, and the configuration of each engine

Figure 5 shows why the SLO matters. Two made-up engines run the same seeded trace with continuous batching. Engine A has a cheaper decode step at small batch (6.0 ms fixed cost) but a maximum batch of 64; engine B has a higher fixed cost (7.5 ms) but cheaper per-request cost and a batch of 256. These constants are our assumptions, not measurements of any real engine.

Worked example: the winner depends on the SLO (chat trace, \(\text{TTFT} \le 1\) s, \(q = 0.9\))

With \(\text{TPOT} \le 10\) ms, engine A’s goodput is 8 req/s and B’s is 5 req/s: A wins by 1.6×. With \(\text{TPOT} \le 30\) ms, A gets 14 req/s and B 22 req/s: B wins by 1.57×. With \(\text{TPOT} \le 12\) ms they tie at 10. A “maximum throughput” benchmark would always crown B, and a batch-1 benchmark would always crown A. Only goodput at your SLO gives the answer that matters for your bill.

engine A (fixed cost 6.0 ms, max batch 64)engine B (fixed cost 7.5 ms, max batch 256)SLO
Figure 5. Continuous-batching simulation of two hypothetical engines on the same 500-request Poisson trace per rate (seed 139). A step costs \(\text{fixed} + \text{per-request} \times \text{decode batch} + \text{per-token} \times \text{new prefill tokens}\). Goodput (139.7) with \(q = 0.9\) is marked by the thin vertical lines.
How to read Figure 5

Each point is one benchmark run at one request rate. Left: P90 TPOT; right: P90 TTFT on a log scale. Move the TPOT SLO from 10 to 30 ms: the winner changes from A to B. Switch to the RAG trace: long prompts make prefill dominate, both engines fall to a few requests per second, and TTFT becomes the binding SLO. This is why a benchmark must use your traffic shape, not a generic dataset.

Tools: vllm bench serve, SGLang’s bench_serving, NVIDIA’s GenAI-Perf/AIPerf and llama-bench all report TTFT and inter-token latency; use one load generator for all engines so that differences come from the engines, not the clients. The full workflow is the lab session 142.

PyTorch code

Three small programs that reproduce the core ideas of this session on CPU: (1) a Q4_0-style GGUF block (one FP16 scale and 32 four-bit codes in 18 bytes), (2) a Triton-style dynamic batcher, (3) goodput from per-request measurements. The batcher here draws arrivals with PyTorch’s generator, so its numbers differ slightly from Figure 2, which uses another seeded generator. The output is the same on every run.

import math, torch
torch.manual_seed(0)

# 1. A Q4_0-style GGUF block: 32 weights -> one FP16 scale + 16 bytes of 4-bit codes
def q4_0_pack(w):                       # w: (n,) float32, n divisible by 32
    b = w.view(-1, 32)
    amax_idx = b.abs().argmax(dim=1, keepdim=True)
    m = b.gather(1, amax_idx)               # signed value with the largest magnitude
    d = (m / -8).to(torch.float16)          # scale, as in llama.cpp's Q4_0
    inv = torch.where(d == 0, torch.zeros_like(d), 1.0 / d.float())
    q = (b * inv + 8.5).floor().clamp(0, 15).to(torch.uint8)
    packed = q[:, :16] | (q[:, 16:] << 4)  # two codes per byte
    return d, packed
def q4_0_unpack(d, packed):
    q = torch.cat([packed & 15, packed >> 4], dim=1).float()
    return ((q - 8) * d.float()).view(-1)
w = torch.randn(4096 * 32) * 0.02
d, p = q4_0_pack(w)
nbytes = d.numel() * 2 + p.numel()
print(f"bits per weight: {8 * nbytes / w.numel():.3f}")
err = (q4_0_unpack(d, p) - w).pow(2).mean().sqrt() / w.pow(2).mean().sqrt()
print(f"relative RMS error: {err:.3f}")

# 2. A Triton-style dynamic batcher: one instance, t(b) = alpha + beta * b
def batcher(lam, delay_ms, bmax, n=3000, alpha=4.0, beta=0.5, seed=139):
    g = torch.Generator().manual_seed(seed)
    arr = torch.cumsum(torch.empty(n).exponential_(generator=g) / lam * 1000, 0).tolist()
    i, free, lat, nb = 0, 0.0, [], 0
    while i < n:
        start = max(free, arr[i])                  # instance free, oldest request waiting
        deadline = max(start, arr[i] + delay_ms)   # send when full or when the delay expires
        k = i + bmax - 1
        send = max(start, arr[k]) if k < n and arr[k] <= deadline else deadline
        j = i
        while j < n and j - i < bmax and arr[j] <= send: j += 1
        done = send + alpha + beta * (j - i)
        lat += [done - a for a in arr[i:j]]; nb += 1; free = done; i = j
    lat = torch.tensor(lat)
    return n / nb, lat.mean().item(), lat.quantile(0.99).item()
for lam, dl, bm in [(100, 0, 32), (100, 5, 32), (1000, 0, 32), (1000, 0, 1)]:
    b, mean, p99 = batcher(lam, dl, bm)
    print(f"rate {lam:5d}/s delay {dl} ms max_batch {bm:2d}: mean batch {b:5.2f}, "
          f"mean {mean:8.2f} ms, p99 {p99:8.2f} ms")

# 3. Goodput: requests/s that meet both SLOs, from per-request measurements
def goodput(rate, ttft_ms, tpot_ms, slo_ttft, slo_tpot, need=0.9):
    ok = ((ttft_ms <= slo_ttft) & (tpot_ms <= slo_tpot)).float().mean().item()
    return ok, (rate if ok >= need else 0.0)
g = torch.Generator().manual_seed(1)
ttft = 200 + 300 * torch.rand(1000, generator=g)      # stand-in measurements
tpot = 8 + 4 * torch.rand(1000, generator=g)
for slo in (10.0, 12.0):
    ok, gp = goodput(20, ttft, tpot, 1000, slo)
    print(f"TPOT SLO {slo} ms: {100*ok:.1f}% of requests meet it, goodput at 20 req/s = {gp}")

Output with PyTorch 2.8 on CPU:

bits per weight: 4.500
relative RMS error: 0.086
rate   100/s delay 0 ms max_batch 32: mean batch  1.09, mean     5.57 ms, p99     9.42 ms
rate   100/s delay 5 ms max_batch 32: mean batch  1.52, mean     9.07 ms, p99    11.00 ms
rate  1000/s delay 0 ms max_batch 32: mean batch  8.02, mean    12.81 ms, p99    21.37 ms
rate  1000/s delay 0 ms max_batch  1: mean batch  1.00, mean  5256.61 ms, p99 10411.28 ms
TPOT SLO 10.0 ms: 49.7% of requests meet it, goodput at 20 req/s = 0.0
TPOT SLO 12.0 ms: 100.0% of requests meet it, goodput at 20 req/s = 20

Read it: the block format costs exactly 4.5 bits per weight with an 8.6% relative error on Gaussian weights; batching turns an unstable service at 1,000 req/s into one with a 21 ms p99; and the same measurements give a goodput of 20 or 0 depending only on the SLO.

The real commands (not run here; check flags against your versions):

# Triton: serve a model repository; HTTP 8000, gRPC 8001, metrics 8002
tritonserver --model-repository=/models
perf_analyzer -m vit_encoder --concurrency-range 1:64:8

# SGLang: an OpenAI-compatible server with the radix cache (on by default)
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 30000

# LMDeploy: TurboMind with a 4-bit AWQ model and an INT8 KV cache (quant_policy 8; 4 for INT4)
lmdeploy serve api_server ./llama-3.1-8b-instruct-awq --model-format awq --quant-policy 8

# llama.cpp: a GGUF model, all layers on the GPU, 4 slots, 16k context shared by the slots
llama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl all -np 4 -c 16384
llama-bench  -m Llama-3.1-8B-Instruct-Q4_K_M.gguf

Choosing an engine

SituationStart withWhy
Many non-LLM models (detectors, ViTs, embedding models, tree models) plus an LLM, one platform teamTriton (Dynamo-Triton) with an LLM backendone API, one metrics system, ensembles, a dynamic batcher for fixed-shape models
Agents, few-shot prompts, multi-turn chat, RL rollouts: lots of shared prefixesSGLang (or vLLM with prefix caching), then benchmarkRadixAttention reuse (139.3), cache-aware routing
Strict JSON or grammar output at high throughputSGLang or vLLM with a fast grammar backendcompressed FSM and jump-forward (139.4) [Likely for the size of the gain]
4-bit AWQ models, long contexts limited by KV memory, InternLM/InternVL modelsLMDeploy (TurboMind)W4A16 kernels, online INT8/INT4 KV cache (139.5)
A laptop, a Mac, a CPU server, an edge box, one or a few usersllama.cppGGUF quant types, CPU+GPU split, no Python (139.6)
An existing TGI deploymentkeep it running, plan a migrationmaintenance mode since 2026; no new features
Large NVIDIA fleet, a few stable models, cost per token firstTensorRT-LLM or vLLM or SGLang, decided by goodputsee session 131 and Algorithm 2

Where this is used

ApplicationTypical choiceWhat matters most
Video retrieval and Video-RAG indexing (frame and clip embeddings)Triton with TensorRT or ONNX Runtime encoders, dynamic batcherlarge batches, throughput per GPU, a DALI or Python step for frame decoding
Action recognition and key-moment detection on live streamsTriton ensembles (decode → detector → classifier), or TensorRT on the edgelow p99 latency at small batch; sequence batcher if the model keeps state across clips
Video fingerprinting and near-duplicate detectionTriton with an embedding modelsteady throughput; batch delay tuned to the latency budget
Video question answering with a VLMSGLang or vLLM for the language model; vision encoder in the same engine or as a Triton modelmany image tokens in prefill (TTFT), prefix caching of system prompt and frames
Agents and tool useSGLangradix-tree prefix reuse across turns and branches; structured tool-call output
Long-document chat and RAG with long contextsLMDeploy or any engine with KV quantizationKV memory per token; TTFT under long prefill
On-device assistants, privacy-sensitive or offline use, video segmentation tools on a laptopllama.cpp (LLM part) with GGUFmemory size of the quant type, bandwidth-bound decode speed

Common mistakes

  • Using Triton’s dynamic batcher for an LLM. Request-level batching waits for the longest request. LLMs need the iteration-level batching that their backend does.
  • Setting a large max_queue_delay “for throughput”. In the batcher model (139.2) the delay does not raise capacity; at low load it only adds latency (5.49 → 9.06 ms mean at 100 req/s with 5 ms delay). Raise it only while measuring the latency budget.
  • Choosing an engine from a single table or blog post. Batch-1 tokens/s and maximum requests/s rank systems differently (Figure 4), and neither is goodput at your SLO.
  • Benchmarking with a closed loop or a fixed prompt. Closed-loop load hides queueing (Little’s law), and the same prompt repeated gives a 100% prefix-cache hit rate that your traffic will never see.
  • Comparing at different precision or output lengths. FP8 against BF16, or ignore_eos in one engine and natural stopping in another, measures the setup, not the engine.
  • Thinking Q4_K_M is exactly 4 bits. Q4_K is 4.5 bits per weight with its scales, and the _M mix uses larger types for some tensors. Budget memory from the real file size plus the KV cache.
  • Ignoring the KV cache in llama.cpp. At long contexts the F16 KV cache can be larger than the weights (70B at 131,072 tokens: 42.9 GB of KV).
  • Starting a new service on an archived project. TGI is in maintenance mode; Triton is now Dynamo-Triton. Check the status and release date of every tool and tutorial.

Interview questions

What is in a Triton model repository, and how does Triton choose which version to serve?basic
A directory with one folder per model. Each folder has a config.pbtxt (backend or platform, max_batch_size, input and output tensors, instance groups, scheduler settings) and numbered version folders holding the model file, such as 1/model.plan for TensorRT, model.onnx for ONNX Runtime, model.pt for TorchScript or model.py for the Python backend. Non-numeric folders and folders starting with 0 are ignored. The version policy decides what is served: latest n versions (default: latest 1), all, or a specific list. So you deploy a new version by adding a folder and roll back by removing it.
How does Triton’s dynamic batcher form a batch, and what does max_queue_delay do?core
It keeps a queue per model. When an instance is free it takes the oldest requests: the largest preferred batch size it can form, otherwise the largest batch up to max_batch_size. With a delay set, it may wait up to that time for more requests, and sends what it has when the delay expires; by default it does not wait. With zero delay the batch size already follows the load: with \(t(b) = \alpha + \beta b\) the steady state is \(b^{*} = \lambda\alpha/(1 - \lambda\beta)\), 8 at 1,000 req/s for \(\alpha = 4\) ms, \(\beta = 0.5\) ms. The delay trades latency for larger batches and less GPU busy time; it is worth it when the GPU is shared or bigger batches are much faster per item, and you raise it only while the latency budget holds.
Why should you not use Triton’s dynamic batcher in front of an LLM, and how is an LLM served in Triton?core
The dynamic batcher batches whole requests. An LLM request runs for a different number of decode steps each, so a request-level batch waits for its longest member and new requests cannot join. LLMs need continuous (iteration-level) batching with a paged KV cache. In Triton this is done by the LLM backend (TensorRT-LLM or vLLM), which takes requests one by one and batches every step. A typical setup is an ensemble: Python preprocessing (tokenizer), the LLM backend, Python postprocessing (detokenizer), with the LLM model decoupled so it can stream many responses for one request.
What is an ensemble in Triton, and when would you use BLS instead?basic
An ensemble is a model with platform "ensemble" and no weights: a fixed graph of steps, each calling one model, with tensors connected by name. The whole pipeline runs inside the server, so intermediate tensors do not travel over the network, and each step keeps its own instances and batching. BLS (business logic scripting) is a Python-backend model that calls other models from code, so it can use loops, conditions and early exits, for example calling a classifier only if a detector found something. Use an ensemble for a fixed pipeline, BLS for control flow.
What makes SGLang fast for agent and few-shot workloads?core
RadixAttention: the KV cache of past requests is kept in a radix tree keyed by token ids, so a new request reuses the longest cached prefix and only prefills the rest; leaves are evicted LRU. With \(n\) requests sharing a prefix of \(S\) tokens and \(u\) own tokens each, prefill drops from \(n(S + u)\) to \(S + nu\): 16× for 64 requests with \(S = 2{,}000\) and \(u = 100\). Agents, few-shot prompts, multi-turn chat and sampling many answers to one prompt all share long prefixes. A cache-aware schedule (longest prefix first, an option) and a cache-aware router across replicas keep the hit rate high. The overlap scheduler hides CPU scheduling time behind GPU work.
How does constrained decoding with a compressed FSM and jump-forward work, and what can go wrong?advanced
The schema or regex is compiled into a finite state machine over characters. At each step the engine masks tokens that would leave the FSM, then samples. Many states have only one possible continuation (braces, quotes, key names); the compressed FSM merges those chains into single edges, and when decoding reaches one, the engine appends the whole string at once and processes it like a short prefill instead of one decode step per token: passes drop from \(F + G\) to about \(G + J\). Pitfalls: the appended text must be re-tokenized with the text before it, because token boundaries can differ, otherwise the model sees an unnatural tokenization; masks have a CPU cost that must be overlapped with the GPU; and forcing a format can lower answer quality if the schema fights the model.
Where does LMDeploy’s 4-bit speedup come from, and when does it disappear?core
W4A16 stores weights in 4 bits and computes in FP16, so a decode step reads about 4× fewer weight bytes. Decode at small batch is memory-bound, so it gets faster: in the survey’s Table IV, LLaMA-2-7B in LMDeploy at batch 1 and input 128 decodes 2.11× faster than FP16, while prefill gains only 1.30× because it is compute-bound and the weights must be dequantized. At batch 16 with input 2,048, decode gains only 1.28×, because the step becomes more compute-bound and the KV cache reads (not quantized here) grow. The speedup is largest for one user and short contexts.
How big is Llama 3.1 8B in Q4_K, and how fast can it decode on a laptop?core
Q4_K stores 256 weights in 144 bytes (two FP16 super-scales, 12 bytes of 6-bit sub-scales and mins, 128 bytes of 4-bit codes): 4.5 bits per weight. \(8.03\text{ B parameters} \times 4.5 / 8 = 4.52\) GB (a real Q4_K_M file is a bit larger because some tensors use 6-bit types). Decode at batch 1 reads the weights once per token, so with dual-channel DDR5-5600 (89.6 GB/s) the bound is \(89.6 / 4.52 = 19.8\) tokens/s; real speed is lower. In F16 the same machine is limited to 5.6 tokens/s. Add the KV cache: 128 KiB per token in F16, 4.29 GB for a 32k context.
Table VI of the survey says LightLLM serves 10.29 req/s and TensorRT-LLM 5.87, but TensorRT-LLM has higher tokens/s. How do you explain this, and what do you conclude?advanced
The two columns measure different things. Tokens/s at batch 1 measures kernels and graph optimizations for one request; TensorRT-LLM has the most of them. Requests/s on ShareGPT at maximum load measures memory management and scheduling: how many requests fit in the KV cache and how prefill and decode are mixed. LightLLM uses token-wise KV memory and split-and-fuse batching, which can pack more requests; the table also notes that TensorRT-LLM’s scheduler was not open source, so its serving setup may not have been tuned. Conclusion: neither column predicts goodput at my SLO, the numbers are from one GPU, one model and 2024 versions, and small gaps (2.7% between TensorRT-LLM and vLLM in tokens/s) are within noise. I would run my own benchmark.
A blog post says engine X is 2× faster than your current engine. How do you decide whether to switch?core
First, read what was measured: model, precision, GPU, versions, batch-1 speed or maximum throughput, prompt and output lengths, prefix caching, closed or open loop. Then replay my own trace on both engines with everything else fixed (same weights, precision, GPUs, parallelism, chat template, output lengths, sampling), tune both with equal effort, sweep open-loop Poisson rates, and compute goodput: the highest rate at which, say, 90% of requests meet both the TTFT and TPOT SLOs. The winner can change with the SLO: in our two-engine model, A wins by 1.6× at \(\text{TPOT} \le 10\) ms and B by 1.57× at 30 ms. Finally, weigh the saved GPU-hours against migration, testing, on-call and the project’s maintenance status.
Why can a closed-loop benchmark make a slow server look fine?advanced
In a closed loop, \(C\) virtual users each send a new request only after the previous one ends. By Little’s law the offered rate is \(\lambda = C / W\). If the server slows down and \(W\) grows, \(\lambda\) falls automatically, so the queue never builds the way it would with real users who arrive independently. Latency at overload is under-reported (a form of coordinated omission). An open-loop generator sends Poisson arrivals at a fixed rate regardless of responses, so queueing delay appears in TTFT as soon as the rate exceeds capacity. Use open loop and sweep the rate; use closed loop only to model a fixed pool of clients, such as a batch job with a fixed number of workers.
What was TGI’s architecture, and what is its status?basic
A Rust launcher starts the processes; a Rust router receives HTTP requests, queues them and builds continuous batches under a token budget; one Python model server per GPU shard runs the model, connected to the router by gRPC. It had FlashAttention and PagedAttention kernels, tensor parallelism, several quantization formats, grammar-constrained output, speculation, SSE streaming and an OpenAI-compatible API, plus backends for TensorRT-LLM, llama.cpp and Neuron. The repository is in maintenance mode and was archived in March 2026; Hugging Face recommends vLLM, SGLang, llama.cpp or MLX for new work.

Exercises

  1. A model has \(t(b) = 3 + 0.25\,b\) ms and max_batch_size 16. Give its capacity with and without batching, and the batch size it settles at with zero delay at 1,600 req/s.
  2. An agent sends 20 turns; the system prompt and tools are 3,000 tokens, and each turn adds 200 tokens of history. With full prefix reuse, how many prefill tokens are computed over the 20 turns, and how many without reuse?
  3. Compute the bits per weight of Q5_0 (block of 32: an FP16 scale, 4 bytes of high bits, 16 bytes of low nibbles) and the size of Llama 3.1 70B in that type.
  4. Llama 3.1 70B in Q4_K on an Apple M2 Ultra: does it fit with a 32k context in F16? What is the batch-1 decode bound?
  5. Write down three settings that must be equal before you compare SGLang and LMDeploy on your traffic, and one that you should deliberately set to match production.
  6. A closed-loop test with 32 users gives a mean latency of 4 s. What request rate did it test? If production traffic is 12 req/s, what does this test tell you?
Answerscheck
1. Without batching: \(1/3.25\ \text{ms} = 307.7\) req/s. With batch 16: \(16/(3 + 4)\ \text{ms} = 2{,}285.7\) req/s. At \(\lambda = 1.6\)/ms: \(b^{*} = 1.6 \cdot 3/(1 - 1.6 \cdot 0.25) = 4.8/0.6 = 8\). 2. Turn \(k\) (from 1 to 20) has a prompt of \(3{,}000 + 200k\) tokens. Without reuse: \(\sum_k (3{,}000 + 200k) = 60{,}000 + 200 \cdot 210 = 102{,}000\) tokens. With reuse only the new 200 tokens per turn, plus the 3,000 once: \(3{,}000 + 20 \cdot 200 = 7{,}000\) tokens, 14.6× fewer. (Generated tokens of each turn are already in the cache.) 3. \(2 + 4 + 16 = 22\) bytes per 32 weights: 5.5 bpw; \(70.55 \cdot 5.5 / 8 = 48.51\) GB. 4. Weights: \(70.55 \cdot 4.5/8 = 39.69\) GB; KV: \(320\text{ KiB} \times 32{,}768 = 10.74\) GB; total 50.4 GB, which fits in 192 GB. Bound: \(800/39.69 = 20.2\) tokens/s. 5. Equal: model weights and precision; GPUs and parallelism; chat template, output lengths and sampling (any three). Match production: the prefix-cache hit rate (or the arrival pattern, open-loop at production rates). 6. \(\lambda = C/W = 32/4 = 8\) req/s. It says nothing about 12 req/s: the test never offered that load, and at 12 req/s queueing may push latency much higher. Run an open-loop test at 12 req/s and above.

Key takeaways

  • Triton (now Dynamo-Triton) is a general model server: a repository of models with config.pbtxt and version folders, many backends, instance groups, a dynamic batcher for fixed-shape models, and ensembles or BLS for pipelines. LLMs are batched by their own backend, not by the dynamic batcher.
  • With zero delay the dynamic batcher already adapts: \(b^{*} = \lambda\alpha/(1-\lambda\beta)\) (139.2); batching raises capacity (7.2× in our example), the delay mostly trades latency for GPU time.
  • SGLang is the engine for shared prefixes and structure: RadixAttention cuts prefill to \(S + nu\) (139.3), and a compressed FSM with jump-forward speeds up constrained output (139.4).
  • LMDeploy’s TurboMind is strong at 4-bit weight-only inference (a decode-time, small-batch gain) and online INT8/INT4 KV caches (139.5). TGI’s router/model-server design is instructive, but the project is in maintenance mode.
  • llama.cpp runs GGUF files whose block types fix the bits per weight (Q4_K = 4.5); decode speed is bounded by bandwidth divided by file size (139.6).
  • Read comparison tables by asking what is measured, how large the gap is relative to noise, and when. Choose engines by goodput at your SLO on your own open-loop traffic (139.7); the winner can change with the SLO.

Key formulas

\[ X_{\max} = \frac{B_{\max}}{\alpha + \beta B_{\max}}, \qquad b^{*} = \frac{\lambda\alpha}{1-\lambda\beta}, \qquad \frac{P_{\text{no cache}}}{P_{\text{radix}}} = \frac{n(S+u)}{S+nu} \]
\[ s_{\text{tok}} = 2 n_{\text{layers}} n_{\text{kv}} d_{\text{head}} b_{\text{kv}}, \qquad \text{bpw} = \frac{8 \cdot \text{bytes}}{\text{weights}}, \qquad r_{\text{decode}} \le \frac{\mathrm{BW}}{\text{size}} \]
\[ \text{goodput} = \max\{\lambda : \Pr[\text{TTFT} \le T_1 \wedge \text{TPOT} \le T_2] \ge q\}, \qquad C = \lambda W \]

References for this session

  1. NVIDIA (code). Triton Inference Server
  2. Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
  3. ggml-org (code). llama.cpp
  4. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  5. NVIDIA (code). Triton Inference Server
  6. NVIDIA (docs). Triton Inference Server user guide: Model Repository, Model Configuration, Dynamic Batcher
  7. NVIDIA (2025). NVIDIA Dynamo-Triton (formerly Triton Inference Server), product page
  8. Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
  9. SGLang project (code). SGLang server arguments: python/sglang/srt/arg_groups/fields (schedule_policy, disable_radix_cache, disable_overlap_schedule defaults)
  10. InternLM team (code). LMDeploy: a toolkit for compressing, deploying and serving LLMs
  11. Hugging Face (code). Text Generation Inference (maintenance mode, archived March 2026)
  12. ggml-org (code). llama.cpp
  13. ggml-org (code). llama.cpp: ggml/src/ggml-common.h (quantization block layouts)
  14. ggml-org (docs). llama.cpp: llama-server README (slots, continuous batching, -ngl, -c)
  15. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  16. vLLM project (docs). vLLM documentation
  17. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  18. Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
  19. Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
  20. Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve