Triton Inference Server, SGLang, LMDeploy, TGI, llama.cpp
Five serving tools you will meet in real systems, each seen through the problem it solves best: Triton Inference Server for many models behind one API, SGLang for shared prefixes and structured output, LMDeploy for 4-bit weights and small KV caches, TGI as a clean design now in maintenance, and llama.cpp for laptops and CPUs. Then two skills that outlast any tool: reading a comparison table critically, and benchmarking engines fairly by goodput on your own traffic.
What you will learn
- What a Triton model repository contains (config.pbtxt, version folders, backends, instance groups) and how ensembles and BLS build pipelines inside the server
- How Triton's dynamic batcher forms batches, the batch size it settles at (b* = λα/(1 − λβ)), and why it is the wrong tool for an LLM
- What SGLang does best: RadixAttention prefix reuse, constrained decoding with a compressed state machine, and an overlap scheduler
- What LMDeploy does best: TurboMind, 4-bit weight-only inference, and online INT8/INT4 KV caches; and what TGI's design was and why it is now in maintenance mode
- How llama.cpp's GGUF block types set the bits per weight, the file size and the bandwidth bound on decode speed
- How to read the survey's comparison of inference engines and serving systems, and the five questions to ask of any such table
- How to benchmark engines fairly: same setup, your own trace, open-loop load, and goodput at your SLO
Why this matters
You know how one LLM engine works inside: the KV cache, paging, continuous batching, schedulers. In practice you must also choose an engine, and many exist. This session looks at five that you will meet in real systems, each from the point of view “what problem does it solve best?”: NVIDIA Triton Inference Server, SGLang, LMDeploy, Hugging Face TGI and llama.cpp. Then it teaches two skills that matter more than any single tool: how to read a comparison table, and how to benchmark engines fairly on your own traffic.
In session 133 you split a serving system into layers: the API gateway, the router, the engine and the model runtime. In session 131 you studied one engine from NVIDIA, and session 138 is the reference design most others copy. The tools here sit at different layers. Triton is a general model server: it serves any model (a ViT, a tokenizer, an LLM) behind one API and lets an LLM engine do the batching. SGLang, LMDeploy and TGI are LLM serving engines like vLLM. llama.cpp is an engine for local and CPU-heavy inference, with its own file format.
The efficient-inference survey (Zhou et al., Section 6, our source for the system-level map) compares inference engines and serving systems in one table (its Table VI). We read that table carefully later, because the way it measures things is typical of every comparison you will see, including vendor blog posts.
Two kinds of system: model servers and LLM engines
Two words first. An inference engine runs a model fast: kernels, KV cache, batching inside one forward pass. A serving system takes requests from many users over the network, queues them, schedules them on the engine, and returns results. The survey uses exactly this split: its table has an “inference” half (attention, linear and graph optimizations, speculative decoding, measured in tokens/s for one request) and a “serving” half (memory management, batching, scheduling, measured in requests/s). [Certain]
| System | Layer | Written in | Best at |
|---|---|---|---|
| Triton Inference Server (now “NVIDIA Dynamo-Triton”) | general model server | C++ core, many backends | serving many different models (CNN, ViT, ONNX, Python logic, LLMs) behind one API, with pipelines (ensembles) and a dynamic batcher |
| SGLang | LLM engine + server | Python scheduler, CUDA/Triton kernels | prefix reuse (RadixAttention), structured output, multi-call programs, large MoE models |
| LMDeploy | LLM engine + server | C++/CUDA engine (TurboMind) and a PyTorch engine | fast 4-bit weight-only (AWQ) inference and online KV-cache quantization |
| Text Generation Inference (TGI) | LLM server | Rust router, Python model server, gRPC | was Hugging Face’s production server; now in maintenance mode |
| llama.cpp | local engine + small server | C/C++ (ggml), no Python | laptops, Apple silicon, CPUs, consumer GPUs; quantized GGUF files; CPU+GPU split |
Two status notes, because tutorials online are often out of date. On 18 March 2025 NVIDIA renamed Triton Inference Server to NVIDIA Dynamo-Triton and put it under the Dynamo platform; NVIDIA says it continues to support it. Dynamo itself is a separate, newer framework for distributed LLM serving (with vLLM, SGLang or TensorRT-LLM as engines). [Certain] The TGI repository says it “is now in maintenance mode”, was archived on 21 March 2026, and recommends vLLM, SGLang, llama.cpp or MLX instead. [Certain] Do not confuse Triton Inference Server with the Triton language for writing GPU kernels (session 86): same name, different projects.
Triton Inference Server
The model repository
Triton serves what it finds in a model repository: a directory with one folder per model. Each model folder holds a config.pbtxt (the model configuration, in protobuf text format) and at least one numbered version folder with the model file. Folders that are not numbers, or that start with 0, are ignored. [Certain] The default file name depends on the backend: model.plan for TensorRT, model.onnx for ONNX Runtime, model.pt for PyTorch (TorchScript), model.py for the Python backend, and model.xml plus model.bin for OpenVINO. [Certain]
A minimal configuration gives the platform or backend, max_batch_size, and the input and output tensors. [Certain] A real one for a ViT encoder:
# model_repository/vit_encoder/config.pbtxt name: "vit_encoder" platform: "tensorrt_plan" # or backend: "onnxruntime", "pytorch", "python", "tensorrtllm", "vllm" max_batch_size: 32 # 0 means: the model does not support batching input [ { name: "pixels" data_type: TYPE_FP16 dims: [ 3, 224, 224 ] } ] output [ { name: "embedding" data_type: TYPE_FP16 dims: [ 768 ] } ] instance_group [ { count: 2 kind: KIND_GPU } ] # two copies of the model on each GPU dynamic_batching { max_queue_delay_microseconds: 2000 } version_policy: { latest: { num_versions: 1 } } # the default
The dims do not include the batch dimension: with max_batch_size > 0, Triton adds it in front. The rules for the other fields, from the Triton docs: [Certain]
- Version policy.
latest(with \(n\) versions),all, orspecific. With no policy, the latest single version is served. So you can deploy version 3 by adding a folder, and roll back by removing it. - Instance groups. How many copies (instances) of the model run, and where (
KIND_GPUorKIND_CPU). The default for GPUs is one instance on each GPU. Two instances on one GPU let one copy compute while the other copies inputs or runs CPU-side work. - Schedulers. The default scheduler sends each request as it is. The dynamic batcher combines requests into a batch on the server. The sequence batcher is for stateful models, where all requests of one sequence must go to the same instance. The ensemble scheduler runs a pipeline of models.
Backends
A backend is the library that actually executes a model. Triton’s core does networking, queueing, batching and metrics; backends do the math. The list includes TensorRT, ONNX Runtime, PyTorch (TorchScript and PyTorch 2), TensorFlow, OpenVINO, Python, DALI (data preprocessing on GPU), FIL (tree models such as XGBoost), TensorRT-LLM and vLLM. [Certain] The Python backend is the glue: you write a class TritonPythonModel with initialize, execute(requests) and finalize, and you can run any Python there, for example a tokenizer. With BLS (business logic scripting), a Python model can call other models in the same server, with loops and conditions an ensemble cannot express. [Certain]
Ensembles: a pipeline inside the server
An ensemble is a model with platform: "ensemble" and no weights. Its configuration lists steps; each step names a model and maps tensor names between steps. A request to the ensemble runs the whole pipeline inside the server, so intermediate tensors never go back over the network. [Certain] The standard LLM pipeline of the TensorRT-LLM backend is exactly this: preprocessing (Python, tokenizer) → tensorrt_llm → postprocessing (Python, detokenizer), wrapped in ensemble. For token streaming, the LLM model is decoupled: it may send zero, one or many responses for one request. [Certain for decoupled models; Likely for the exact folder names in your version]
How to read Figure 1
Step through. The left side is what you put on disk; the highlighted lines are the files that matter at each step. The right side is the server at run time. Boxes in the dashed frame belong to the ensemble: one request from the client runs all three models in order. The ViT encoder is a separate model that a VLM pipeline could call in another ensemble step; it uses Triton’s own dynamic batcher (Figure 2), while the LLM step batches inside its backend.
The dynamic batcher, with numbers
For a fixed-shape model like a ViT encoder or an embedding model, batching on the server is the main way to use the GPU well. The dynamic batcher keeps a queue per model. When an instance is free, it forms a batch from the oldest requests: if it can reach a preferred batch size it takes the largest one, otherwise the largest batch up to max_batch_size; if max_queue_delay_microseconds is set, it may wait that long for more requests, and “if the delay expires the dynamic batcher sends the batch as is”. The default configuration does not wait. [Certain]
Model one instance with a linear cost per batch. A batch of \(b\) requests takes \(t(b) = \alpha + \beta b\): \(\alpha\) is the fixed part (reading weights, launching kernels) and \(\beta\) the cost of one more input. Then
- Read the symbols
- \(X(b)\)
- throughput of one instance in requests per second when every batch has \(b\) requests
- \(\alpha, \beta\)
- fixed cost of one batch, and the extra cost per request; we use \(\alpha = 4\) ms, \(\beta = 0.5\) ms (our assumption for a ViT encoder) [Likely]
- \(B_{\max}\)
max_batch_sizein the configuration
With zero delay, the batch size adjusts itself to the load. While a batch of size \(b\) runs, \(\lambda\, t(b)\) new requests arrive, and they form the next batch. The steady state is the fixed point \(b = \lambda(\alpha + \beta b)\):
- Read the symbols
- \(\lambda\)
- arrival rate in requests per millisecond (1,000 req/s is \(\lambda = 1\))
- \(b^{*}\)
- the batch size the system settles at, with no delay configured
Without batching (max_batch_size: 1) the capacity is \(1/4.5\ \text{ms} = 222\) req/s. With \(B_{\max} = 32\): \(32 / (4 + 16)\ \text{ms} = 1{,}600\) req/s, 7.2× more. At 1,000 req/s the fixed point is \(b^{*} = 1 \cdot 4 / (1 - 0.5) = 8\) and one batch takes 8 ms; our simulation of 3,000 Poisson arrivals gives a mean batch of 8.00, mean latency 12.78 ms, p99 20.6 ms. The same traffic with no batching is unstable: the queue grows for the whole run (mean latency 5.3 s after 3,000 requests). At 100 req/s, a 5 ms delay raises the mean batch from 1.09 to 1.49 and lowers GPU busy time from 42% to 32%, but the mean latency rises from 5.49 to 9.06 ms.
So, in this model, the delay does not raise the maximum throughput; it trades latency for fewer, larger batches. That is useful when the GPU is shared with other models (Triton often hosts many) or when a larger batch is much faster per item than the linear model says (for example a TensorRT engine tuned for batch 8). It is why the Triton docs recommend: start with dynamic_batching { }, measure, then raise the delay only while latency stays inside the budget. [Certain for the procedure]
How to read Figure 2
The ticks are arrivals, the bars are batches on the GPU with their size. Move the rate up: batches grow by themselves, even with zero delay, and the readout shows that the mean batch matches \(b^{*}\). Choose max batch 1 and a rate above 222 req/s: the system is unstable and latency explodes. Add a delay at low load: batches grow, GPU busy time falls, latency rises. At high load the delay changes almost nothing, because the queue is never empty.
For measurement, Triton ships perf_analyzer (load generator), Model Analyzer (searches configurations such as instance count and batch delay) and an LLM benchmarking tool (GenAI-Perf, whose role is being taken over by NVIDIA’s newer AIPerf). [Certain for perf_analyzer and Model Analyzer; Likely for the AIPerf transition] Metrics are on port 8002 in Prometheus format, next to HTTP on 8000 and gRPC on 8001. [Certain]
SGLang: reuse, structure and programs
SGLang (Zheng et al., 2023) started as two things together: a small frontend language embedded in Python for programs that make many LLM calls, and a runtime that executes them fast. The paper reports up to 6.4× higher throughput than other systems on agent, reasoning, few-shot, JSON, RAG and multi-turn chat workloads. [Certain that the paper reports it; the number is for their setup and versions] Today most people use the runtime alone, as an OpenAI-compatible server, like vLLM. You met its key idea, RadixAttention, in session 134 and its programming model in session 91; here we look at what makes it the right choice.
1. RadixAttention: the KV cache is a tree
SGLang keeps the KV cache of finished and running requests in a radix tree keyed by token ids. A new request walks the tree, reuses the longest cached prefix, and only computes the rest. When memory is short, leaves are evicted in LRU order. This is on by default (disable_radix_cache = False in the current server arguments). [Certain] The default scheduling policy is now fcfs; lpm (longest prefix match first, the paper’s cache-aware order) is an option. [Certain, from the source of the current main branch] If \(n\) requests share a prefix of \(S\) tokens and each adds \(u\) own tokens, the prefill work is
A 2,000-token prompt (instructions and examples) is shared by 64 requests, each with a 100-token question. Without reuse: \(64 \cdot 2{,}100 = 134{,}400\) prefill tokens. With the radix cache: \(2{,}000 + 64 \cdot 100 = 8{,}400\), which is 16× less prefill. The same structure appears in agents (a long system prompt and tool list), multi-turn chat (the history), and tree search (many branches of one reasoning trace).
2. Structured output with a compressed state machine
To force JSON that follows a schema, an engine turns the schema into a finite state machine (FSM) over characters and, at each step, masks every token that would leave the FSM. Much of a JSON answer is fixed text: braces, quotes, key names. SGLang’s paper compresses chains of states that have only one possible next string into one edge, and when decoding reaches such an edge it appends the whole string at once and continues (“jump-forward”). [Certain for the paper] If the answer has \(F\) forced tokens in \(J\) runs and \(G\) free tokens:
For example, \(F = 12\) forced tokens in \(J = 3\) runs and \(G = 8\) free tokens need 20 passes token by token, about 11 with jump-forward (an illustration, not a measurement). The appended text is processed like a short prefill, which is cheap next to a decode step for every token. One detail matters: the forced string must be re-tokenized together with the text before it, because the tokenizer may merge characters across the boundary differently. [Likely] Grammar backends change between versions (xgrammar is a common default today), so check what your version enables. [Likely]
- input: the request state \(s\) in the compressed FSM, the text so far, logits of the model
- if the edge out of \(s\) is a single forced string \(c\) then
- append \(c\) to the text, re-tokenize the end of the text, and schedule the new tokens as an extend (prefill) step // formula (139.4)
- move \(s\) to the end of the edge; continue
- mask the logits: keep only tokens whose characters stay inside the FSM from \(s\)
- sample a token \(x\) from the masked distribution; append \(x\); move \(s\) along the characters of \(x\)
- if \(s\) is an accepting state and \(x\) is end-of-sequence then finish
- output: a string that is guaranteed to match the schema
3. A scheduler that hides its own cost
At small models and large batches, the CPU work of scheduling (building the batch, updating the radix tree, preparing inputs) can take a large part of each step. SGLang’s overlap scheduler prepares batch \(k+1\) on the CPU while the GPU runs batch \(k\); it is on by default (disable_overlap_schedule = False). [Certain] Other features you will see in its docs: chunked prefill, CUDA graphs for decode, speculative decoding (EAGLE), prefill-decode disaggregation, data-parallel attention and expert parallelism for large MoE models such as DeepSeek, and a router that sends requests to the replica most likely to have their prefix cached. [Likely for the details of each in your version]
When to pick SGLang. When traffic shares long prefixes (agents, few-shot, multi-turn chat, RL rollouts with many samples per prompt), when you need structured output at high speed, or when you serve very large MoE models on many GPUs. When traffic has no shared prefixes, vLLM and SGLang implement much the same ideas, and your benchmark should decide.
LMDeploy: 4-bit weights and a small KV cache
LMDeploy (from the InternLM team at Shanghai AI Lab) has two engines. TurboMind is a C++/CUDA engine, which “strives for ultimate optimization of inference performance”; the PyTorch engine is written in Python to make new models and hardware (for example Huawei Ascend) easy to add. They support different models and data types. [Certain] Its README lists persistent batch (its name for continuous batching), a blocked KV cache (paging), dynamic split&fuse (chunked prefill), tensor parallelism and its own CUDA kernels, and claims up to 1.8× the request throughput of vLLM and 4-bit inference 2.4× faster than FP16. [Certain that the README claims it; not verified by us, and versions have moved since]
The independent numbers in the survey (Table IV, LLaMA-2 on one A100, W4A16 weight-only quantization) show where the 4-bit gain comes from. For LLaMA-2-7B in LMDeploy at batch 1 and input 128, the speedups over FP16 are 1.30 for prefill, 2.11 for decode and 2.09 end to end. At batch 16 with input 2,048, decode gains only 1.28. [Certain, from the table] This is exactly the rule from session 104: weight-only quantization helps memory-bound decode at small batch, and helps little in compute-bound prefill or at large batch.
LMDeploy’s other strength is online KV-cache quantization to INT8 or INT4, computed while serving with no calibration step. [Certain that it exists] The KV bytes per token (ignoring the small scales) are
- Read the symbols
- \(n_{\text{layers}}, n_{\text{kv}}, d_{\text{head}}\)
- number of layers, KV heads and head size; the 2 counts K and V
- \(b_{\text{kv}}\)
- bytes per stored number: 2 for BF16, 1 for INT8, 0.5 for INT4
- \(M_{\text{KV}}, N_{\text{tok}}\)
- memory given to the KV cache, and how many tokens fit in it
\(s_{\text{tok}} = 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes in BF16, so 40 GB holds 305,175 tokens. INT8 halves it to 65,536 bytes (610,351 tokens), INT4 to 32,768 bytes (1,220,703 tokens), 4× more concurrent context than BF16, before scales. Whether accuracy holds at INT4 depends on the model and task: see session 110 and measure it.
When to pick LMDeploy. 4-bit AWQ models on NVIDIA GPUs (including older ones), long contexts where KV memory is the limit, and InternLM or InternVL models, which it supports early. [Likely for the last point]
TGI: an important design, now in maintenance
Text Generation Inference was Hugging Face’s production server and powered its inference endpoints. Its design is worth knowing because it shows a clean split of roles: a Rust launcher starts everything, a Rust router receives HTTP requests, queues them and forms continuous batches under a token budget, and one Python model server per GPU shard runs the model, connected to the router by gRPC. [Certain for the components] Its features: continuous batching, tensor parallelism, FlashAttention and PagedAttention kernels, quantization (bitsandbytes, GPTQ, AWQ, Marlin, FP8, EETQ), JSON and grammar constraints, speculation, token streaming with server-sent events, and an OpenAI-compatible Messages API. It also had backends for TensorRT-LLM, llama.cpp and AWS Neuron, and ran on NVIDIA, AMD, Intel, Gaudi, Inferentia and TPU. [Certain]
The repository is now in maintenance mode (archived 21 March 2026, only small fixes) and points users to vLLM, SGLang, llama.cpp or MLX. [Certain] So: you will still meet TGI in existing deployments and in older papers and benchmarks; for a new service, plan for a maintained engine.
llama.cpp: one file, any laptop
llama.cpp is a C/C++ inference engine built on the ggml tensor library, with no Python at run time. It runs on CPUs (with SIMD kernels), Apple silicon (Metal), NVIDIA and AMD GPUs, Vulkan and more. [Certain] You saw its CPU+GPU layer split (-ngl) in session 130. Here we look at the two other pieces you need: the GGUF file and the server.
GGUF: the model in one file
A GGUF file holds everything needed to run a model: a header, metadata as key-value pairs (architecture, hyper-parameters, the tokenizer and often the chat template), a list of tensors with their types and shapes, and the aligned tensor data, which can be memory-mapped. [Certain] Each tensor has its own quantization type. Types are built from blocks: a block stores a few scales plus packed low-bit codes for 32 or 256 weights. From the block sizes in ggml-common.h: [Certain]
- Read the symbols
- bpw
- bits per weight, including the scales
- \(P, \overline{\text{bpw}}\)
- number of parameters, and the average bpw over all tensors
- \(r_{\text{decode}}, \mathrm{BW}\)
- tokens per second at batch 1, and memory bandwidth: each new token must read (almost) all the weights once
| Type | Weights per block | Block content | Bytes | bpw |
|---|---|---|---|---|
Q8_0 | 32 | FP16 scale + 32 int8 | 34 | 8.5 |
Q4_0 | 32 | FP16 scale + \(32 \times 4\) bits | 18 | 4.5 |
Q6_K | 256 | FP16 scale + 16 int8 sub-scales + \(256 \times 6\) bits | 210 | 6.5625 |
Q5_K | 256 | 2 FP16 + 12 bytes of 6-bit scales and mins + \(256 \times 5\) bits | 176 | 5.5 |
Q4_K | 256 | 2 FP16 + 12 bytes of 6-bit scales and mins + \(256 \times 4\) bits | 144 | 4.5 |
Q3_K | 256 | FP16 scale + 12 bytes of scales + \(256 \times 3\) bits (\(2 + 1\) bit planes) | 110 | 3.4375 |
Q2_K | 256 | 2 FP16 + 16 bytes of 4-bit scales and mins + \(256 \times 2\) bits | 84 | 2.625 |
The “K” types use a 256-weight super-block with sub-blocks that have their own small scales, a two-level version of the group quantization in session 103. Names like Q4_K_M and Q4_K_S are mixes: most tensors in Q4_K, some sensitive ones (for example parts of the attention value and FFN down projections, and the output layer) in a larger type, so the average bpw is a bit above 4.5. [Likely for the exact tensor rules, which change between versions]
F16: 16.06 GB. Q8_0: \(8.03 \cdot 8.5/8 = 8.53\) GB. Q4_K for every tensor: 4.52 GB. On a laptop with dual-channel DDR5-5600, bandwidth is \(5{,}600 \cdot 10^6 \cdot 8\ \text{bytes} \cdot 2 = 89.6\) GB/s, so decode is at most \(89.6/4.52 = 19.8\) tokens/s at Q4_K and 5.6 tokens/s at F16. On an Apple M2 Ultra (800 GB/s) the bound is 177.1 tokens/s at Q4_K. These are upper limits: real speeds are lower, and the embedding table (6.5% of the weights) is not read in full each step. The KV cache in F16 adds 128 KiB per token: 4.29 GB for 32,768 tokens.
_M and _S files mix types). Device memory and bandwidth are nominal; for discrete GPUs that are too small, the readout splits layers with -ngl and assumes 89.6 GB/s host memory.How to read Figure 3
The top bar is what must be in memory: weights plus the KV cache for the chosen context; the dashed line is the device memory. The lower bars give the speed limit for each type on the chosen device: halve the bits and the bound doubles. Choose the 70B model on the RTX 4090: it does not fit even at Q2_K, and the readout shows how slow the split becomes, because the host part runs at laptop bandwidth. Raise the context to 131,072 tokens for 70B: the F16 KV cache alone is 42.9 GB.
llama-server
llama-server exposes an OpenAI-compatible HTTP API. Requests run in slots (-np, the number of parallel sequences; the default is automatic), with continuous batching on by default, and a unified KV buffer shared by all slots. -ngl sets how many layers live on the GPU (default auto), and -c the context size (0 means: take it from the model). [Certain, from the current server README] This is enough for a few users on one machine, an edge box, or an offline batch job; it is not a replacement for a GPU cluster engine with paged attention and large batches. For measuring raw speed, llama-bench reports prompt processing and token generation rates separately.
How to read a comparison table
The survey’s Table VI is a good example to practice on. It compares seven systems on one NVIDIA A100 80 GB: inference throughput with Llama2-7B (batch 1, input 1k, output 128 tokens, speculative decoding off) and serving throughput on the ShareGPT dataset. [Certain] Some numbers from it: HuggingFace 38.963 tokens/s, vLLM 90.052, TensorRT-LLM 92.512, FlashDecoding++ 106.636; for serving, vLLM 7.11 req/s, TensorRT-LLM 5.87, LightLLM 10.29, and no serving number for HuggingFace, OpenPPL or FlashDecoding++.
How to read Figure 4
Switch the sort. The leader changes: FlashDecoding++ by tokens/s, LightLLM by requests/s, and TensorRT-LLM, with the most ticks, is last among the systems with a serving number. The two columns measure different things (kernels for one request, against memory management and scheduling under load), and neither is goodput under an SLO.
Five questions to ask of any such table, this one included:
- What exactly is measured? Batch-1 tokens/s measures kernel speed for one user. Maximum requests/s measures behaviour under overload. Neither says what happens at your load with your latency target.
- Is the gap larger than the noise? TensorRT-LLM’s 92.512 tokens/s is 2.7% above vLLM’s 90.052. One run, five significant digits, no error bars: this gap means “about the same”.
- Are the columns consistent? A system can win one column and lose the next (LightLLM is 1.75× TensorRT-LLM in requests/s, but 20% slower in tokens/s). Ask why: here, token-wise KV memory and the scheduler matter more than kernels under load.
- When, and which versions? Engines change monthly. A 2024 table says little about 2026 releases, and the table itself notes that TensorRT-LLM’s scheduler was not open source.
- Does it look right? In the survey’s Table IV (W4A16 speedups), the LLaMA-2-7B rows for batch 8 and batch 16 are identical in both the TensorRT-LLM and the LMDeploy half, cell by cell. That looks like a copy error. [Likely] Careful readers notice such things before quoting a number.
How to benchmark engines fairly on your own traffic
The goal is not “which engine is fastest” but “which engine serves my traffic at my latency target with the fewest GPUs”. The right number is goodput (DistServe’s term): the highest request rate at which a chosen fraction of requests meets every SLO (service-level objective), such as time to first token (TTFT) and time per output token (TPOT), defined in session 81.
- Read the symbols
- \(\lambda\)
- request arrival rate (requests per second)
- \(T_1, T_2, q\)
- the SLO targets for TTFT and TPOT, and the fraction of requests that must meet both (for example \(q = 0.9\))
- \(C, W\)
- Little’s law: average number of requests in the system equals rate times average time in the system
Little’s law explains a common trap. A closed-loop benchmark keeps \(C\) users busy: each sends its next request when the last one ends. If each request takes \(W = 8\) s, 64 users produce only \(\lambda = 64/8 = 8\) req/s, and if the server slows down, the load falls with it, so queueing is hidden. Real users do not wait for each other. Use an open-loop load (Poisson arrivals at a fixed rate) and sweep the rate.
- input: a trace from production (prompt lengths, output lengths, shared prefixes, arrival pattern), SLOs \(T_1, T_2, q\), the candidate engines
- Fix everything that is not the engine: same model weights and revision, same precision (BF16 vs BF16, FP8 vs FP8), same GPUs and count, same parallelism, same tokenizer and chat template
- Fix the work: same max output tokens, same stop rules, same sampling; check that output lengths match (forcing them with
ignore_eoschanges the workload) - Tune each engine fairly: max batch, max tokens per step, chunked prefill, KV memory fraction, CUDA graphs; give each the same tuning time
- Decide prefix caching on purpose: on for both if your traffic shares prefixes, and use a realistic hit rate, not the same prompt repeated
- for each rate \(\lambda\) in a sweep: warm up, send open-loop Poisson arrivals for several minutes, record TTFT and TPOT per request, check output quality on a sample
- Compute goodput (139.7) for each engine; repeat runs and report the spread
- output: goodput per GPU for your SLO, the latency curves, and the configuration of each engine
Figure 5 shows why the SLO matters. Two made-up engines run the same seeded trace with continuous batching. Engine A has a cheaper decode step at small batch (6.0 ms fixed cost) but a maximum batch of 64; engine B has a higher fixed cost (7.5 ms) but cheaper per-request cost and a batch of 256. These constants are our assumptions, not measurements of any real engine.
With \(\text{TPOT} \le 10\) ms, engine A’s goodput is 8 req/s and B’s is 5 req/s: A wins by 1.6×. With \(\text{TPOT} \le 30\) ms, A gets 14 req/s and B 22 req/s: B wins by 1.57×. With \(\text{TPOT} \le 12\) ms they tie at 10. A “maximum throughput” benchmark would always crown B, and a batch-1 benchmark would always crown A. Only goodput at your SLO gives the answer that matters for your bill.
How to read Figure 5
Each point is one benchmark run at one request rate. Left: P90 TPOT; right: P90 TTFT on a log scale. Move the TPOT SLO from 10 to 30 ms: the winner changes from A to B. Switch to the RAG trace: long prompts make prefill dominate, both engines fall to a few requests per second, and TTFT becomes the binding SLO. This is why a benchmark must use your traffic shape, not a generic dataset.
Tools: vllm bench serve, SGLang’s bench_serving, NVIDIA’s GenAI-Perf/AIPerf and llama-bench all report TTFT and inter-token latency; use one load generator for all engines so that differences come from the engines, not the clients. The full workflow is the lab session 142.
PyTorch code
Three small programs that reproduce the core ideas of this session on CPU: (1) a Q4_0-style GGUF block (one FP16 scale and 32 four-bit codes in 18 bytes), (2) a Triton-style dynamic batcher, (3) goodput from per-request measurements. The batcher here draws arrivals with PyTorch’s generator, so its numbers differ slightly from Figure 2, which uses another seeded generator. The output is the same on every run.
import math, torch torch.manual_seed(0) # 1. A Q4_0-style GGUF block: 32 weights -> one FP16 scale + 16 bytes of 4-bit codes def q4_0_pack(w): # w: (n,) float32, n divisible by 32 b = w.view(-1, 32) amax_idx = b.abs().argmax(dim=1, keepdim=True) m = b.gather(1, amax_idx) # signed value with the largest magnitude d = (m / -8).to(torch.float16) # scale, as in llama.cpp's Q4_0 inv = torch.where(d == 0, torch.zeros_like(d), 1.0 / d.float()) q = (b * inv + 8.5).floor().clamp(0, 15).to(torch.uint8) packed = q[:, :16] | (q[:, 16:] << 4) # two codes per byte return d, packed def q4_0_unpack(d, packed): q = torch.cat([packed & 15, packed >> 4], dim=1).float() return ((q - 8) * d.float()).view(-1) w = torch.randn(4096 * 32) * 0.02 d, p = q4_0_pack(w) nbytes = d.numel() * 2 + p.numel() print(f"bits per weight: {8 * nbytes / w.numel():.3f}") err = (q4_0_unpack(d, p) - w).pow(2).mean().sqrt() / w.pow(2).mean().sqrt() print(f"relative RMS error: {err:.3f}") # 2. A Triton-style dynamic batcher: one instance, t(b) = alpha + beta * b def batcher(lam, delay_ms, bmax, n=3000, alpha=4.0, beta=0.5, seed=139): g = torch.Generator().manual_seed(seed) arr = torch.cumsum(torch.empty(n).exponential_(generator=g) / lam * 1000, 0).tolist() i, free, lat, nb = 0, 0.0, [], 0 while i < n: start = max(free, arr[i]) # instance free, oldest request waiting deadline = max(start, arr[i] + delay_ms) # send when full or when the delay expires k = i + bmax - 1 send = max(start, arr[k]) if k < n and arr[k] <= deadline else deadline j = i while j < n and j - i < bmax and arr[j] <= send: j += 1 done = send + alpha + beta * (j - i) lat += [done - a for a in arr[i:j]]; nb += 1; free = done; i = j lat = torch.tensor(lat) return n / nb, lat.mean().item(), lat.quantile(0.99).item() for lam, dl, bm in [(100, 0, 32), (100, 5, 32), (1000, 0, 32), (1000, 0, 1)]: b, mean, p99 = batcher(lam, dl, bm) print(f"rate {lam:5d}/s delay {dl} ms max_batch {bm:2d}: mean batch {b:5.2f}, " f"mean {mean:8.2f} ms, p99 {p99:8.2f} ms") # 3. Goodput: requests/s that meet both SLOs, from per-request measurements def goodput(rate, ttft_ms, tpot_ms, slo_ttft, slo_tpot, need=0.9): ok = ((ttft_ms <= slo_ttft) & (tpot_ms <= slo_tpot)).float().mean().item() return ok, (rate if ok >= need else 0.0) g = torch.Generator().manual_seed(1) ttft = 200 + 300 * torch.rand(1000, generator=g) # stand-in measurements tpot = 8 + 4 * torch.rand(1000, generator=g) for slo in (10.0, 12.0): ok, gp = goodput(20, ttft, tpot, 1000, slo) print(f"TPOT SLO {slo} ms: {100*ok:.1f}% of requests meet it, goodput at 20 req/s = {gp}")
Output with PyTorch 2.8 on CPU:
bits per weight: 4.500 relative RMS error: 0.086 rate 100/s delay 0 ms max_batch 32: mean batch 1.09, mean 5.57 ms, p99 9.42 ms rate 100/s delay 5 ms max_batch 32: mean batch 1.52, mean 9.07 ms, p99 11.00 ms rate 1000/s delay 0 ms max_batch 32: mean batch 8.02, mean 12.81 ms, p99 21.37 ms rate 1000/s delay 0 ms max_batch 1: mean batch 1.00, mean 5256.61 ms, p99 10411.28 ms TPOT SLO 10.0 ms: 49.7% of requests meet it, goodput at 20 req/s = 0.0 TPOT SLO 12.0 ms: 100.0% of requests meet it, goodput at 20 req/s = 20
Read it: the block format costs exactly 4.5 bits per weight with an 8.6% relative error on Gaussian weights; batching turns an unstable service at 1,000 req/s into one with a 21 ms p99; and the same measurements give a goodput of 20 or 0 depending only on the SLO.
The real commands (not run here; check flags against your versions):
# Triton: serve a model repository; HTTP 8000, gRPC 8001, metrics 8002 tritonserver --model-repository=/models perf_analyzer -m vit_encoder --concurrency-range 1:64:8 # SGLang: an OpenAI-compatible server with the radix cache (on by default) python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 30000 # LMDeploy: TurboMind with a 4-bit AWQ model and an INT8 KV cache (quant_policy 8; 4 for INT4) lmdeploy serve api_server ./llama-3.1-8b-instruct-awq --model-format awq --quant-policy 8 # llama.cpp: a GGUF model, all layers on the GPU, 4 slots, 16k context shared by the slots llama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl all -np 4 -c 16384 llama-bench -m Llama-3.1-8B-Instruct-Q4_K_M.gguf
Choosing an engine
| Situation | Start with | Why |
|---|---|---|
| Many non-LLM models (detectors, ViTs, embedding models, tree models) plus an LLM, one platform team | Triton (Dynamo-Triton) with an LLM backend | one API, one metrics system, ensembles, a dynamic batcher for fixed-shape models |
| Agents, few-shot prompts, multi-turn chat, RL rollouts: lots of shared prefixes | SGLang (or vLLM with prefix caching), then benchmark | RadixAttention reuse (139.3), cache-aware routing |
| Strict JSON or grammar output at high throughput | SGLang or vLLM with a fast grammar backend | compressed FSM and jump-forward (139.4) [Likely for the size of the gain] |
| 4-bit AWQ models, long contexts limited by KV memory, InternLM/InternVL models | LMDeploy (TurboMind) | W4A16 kernels, online INT8/INT4 KV cache (139.5) |
| A laptop, a Mac, a CPU server, an edge box, one or a few users | llama.cpp | GGUF quant types, CPU+GPU split, no Python (139.6) |
| An existing TGI deployment | keep it running, plan a migration | maintenance mode since 2026; no new features |
| Large NVIDIA fleet, a few stable models, cost per token first | TensorRT-LLM or vLLM or SGLang, decided by goodput | see session 131 and Algorithm 2 |
Where this is used
| Application | Typical choice | What matters most |
|---|---|---|
| Video retrieval and Video-RAG indexing (frame and clip embeddings) | Triton with TensorRT or ONNX Runtime encoders, dynamic batcher | large batches, throughput per GPU, a DALI or Python step for frame decoding |
| Action recognition and key-moment detection on live streams | Triton ensembles (decode → detector → classifier), or TensorRT on the edge | low p99 latency at small batch; sequence batcher if the model keeps state across clips |
| Video fingerprinting and near-duplicate detection | Triton with an embedding model | steady throughput; batch delay tuned to the latency budget |
| Video question answering with a VLM | SGLang or vLLM for the language model; vision encoder in the same engine or as a Triton model | many image tokens in prefill (TTFT), prefix caching of system prompt and frames |
| Agents and tool use | SGLang | radix-tree prefix reuse across turns and branches; structured tool-call output |
| Long-document chat and RAG with long contexts | LMDeploy or any engine with KV quantization | KV memory per token; TTFT under long prefill |
| On-device assistants, privacy-sensitive or offline use, video segmentation tools on a laptop | llama.cpp (LLM part) with GGUF | memory size of the quant type, bandwidth-bound decode speed |
Common mistakes
- Using Triton’s dynamic batcher for an LLM. Request-level batching waits for the longest request. LLMs need the iteration-level batching that their backend does.
- Setting a large
max_queue_delay“for throughput”. In the batcher model (139.2) the delay does not raise capacity; at low load it only adds latency (5.49 → 9.06 ms mean at 100 req/s with 5 ms delay). Raise it only while measuring the latency budget. - Choosing an engine from a single table or blog post. Batch-1 tokens/s and maximum requests/s rank systems differently (Figure 4), and neither is goodput at your SLO.
- Benchmarking with a closed loop or a fixed prompt. Closed-loop load hides queueing (Little’s law), and the same prompt repeated gives a 100% prefix-cache hit rate that your traffic will never see.
- Comparing at different precision or output lengths. FP8 against BF16, or
ignore_eosin one engine and natural stopping in another, measures the setup, not the engine. - Thinking Q4_K_M is exactly 4 bits. Q4_K is 4.5 bits per weight with its scales, and the _M mix uses larger types for some tensors. Budget memory from the real file size plus the KV cache.
- Ignoring the KV cache in llama.cpp. At long contexts the F16 KV cache can be larger than the weights (70B at 131,072 tokens: 42.9 GB of KV).
- Starting a new service on an archived project. TGI is in maintenance mode; Triton is now Dynamo-Triton. Check the status and release date of every tool and tutorial.
Interview questions
What is in a Triton model repository, and how does Triton choose which version to serve?basic
How does Triton’s dynamic batcher form a batch, and what does max_queue_delay do?core
Why should you not use Triton’s dynamic batcher in front of an LLM, and how is an LLM served in Triton?core
What is an ensemble in Triton, and when would you use BLS instead?basic
What makes SGLang fast for agent and few-shot workloads?core
How does constrained decoding with a compressed FSM and jump-forward work, and what can go wrong?advanced
Where does LMDeploy’s 4-bit speedup come from, and when does it disappear?core
How big is Llama 3.1 8B in Q4_K, and how fast can it decode on a laptop?core
Table VI of the survey says LightLLM serves 10.29 req/s and TensorRT-LLM 5.87, but TensorRT-LLM has higher tokens/s. How do you explain this, and what do you conclude?advanced
A blog post says engine X is 2× faster than your current engine. How do you decide whether to switch?core
Why can a closed-loop benchmark make a slow server look fine?advanced
What was TGI’s architecture, and what is its status?basic
Exercises
- A model has \(t(b) = 3 + 0.25\,b\) ms and
max_batch_size16. Give its capacity with and without batching, and the batch size it settles at with zero delay at 1,600 req/s. - An agent sends 20 turns; the system prompt and tools are 3,000 tokens, and each turn adds 200 tokens of history. With full prefix reuse, how many prefill tokens are computed over the 20 turns, and how many without reuse?
- Compute the bits per weight of
Q5_0(block of 32: an FP16 scale, 4 bytes of high bits, 16 bytes of low nibbles) and the size of Llama 3.1 70B in that type. - Llama 3.1 70B in Q4_K on an Apple M2 Ultra: does it fit with a 32k context in F16? What is the batch-1 decode bound?
- Write down three settings that must be equal before you compare SGLang and LMDeploy on your traffic, and one that you should deliberately set to match production.
- A closed-loop test with 32 users gives a mean latency of 4 s. What request rate did it test? If production traffic is 12 req/s, what does this test tell you?
Answerscheck
Key takeaways
- Triton (now Dynamo-Triton) is a general model server: a repository of models with config.pbtxt and version folders, many backends, instance groups, a dynamic batcher for fixed-shape models, and ensembles or BLS for pipelines. LLMs are batched by their own backend, not by the dynamic batcher.
- With zero delay the dynamic batcher already adapts: \(b^{*} = \lambda\alpha/(1-\lambda\beta)\) (139.2); batching raises capacity (7.2× in our example), the delay mostly trades latency for GPU time.
- SGLang is the engine for shared prefixes and structure: RadixAttention cuts prefill to \(S + nu\) (139.3), and a compressed FSM with jump-forward speeds up constrained output (139.4).
- LMDeploy’s TurboMind is strong at 4-bit weight-only inference (a decode-time, small-batch gain) and online INT8/INT4 KV caches (139.5). TGI’s router/model-server design is instructive, but the project is in maintenance mode.
- llama.cpp runs GGUF files whose block types fix the bits per weight (Q4_K = 4.5); decode speed is bounded by bandwidth divided by file size (139.6).
- Read comparison tables by asking what is measured, how large the gap is relative to noise, and when. Choose engines by goodput at your SLO on your own open-loop traffic (139.7); the winner can change with the SLO.
Key formulas
References for this session
- NVIDIA (code). Triton Inference Server
- Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
- ggml-org (code). llama.cpp
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- NVIDIA (code). Triton Inference Server
- NVIDIA (docs). Triton Inference Server user guide: Model Repository, Model Configuration, Dynamic Batcher
- NVIDIA (2025). NVIDIA Dynamo-Triton (formerly Triton Inference Server), product page
- Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
- SGLang project (code). SGLang server arguments: python/sglang/srt/arg_groups/fields (schedule_policy, disable_radix_cache, disable_overlap_schedule defaults)
- InternLM team (code). LMDeploy: a toolkit for compressing, deploying and serving LLMs
- Hugging Face (code). Text Generation Inference (maintenance mode, archived March 2026)
- ggml-org (code). llama.cpp
- ggml-org (code). llama.cpp: ggml/src/ggml-common.h (quantization block layouts)
- ggml-org (docs). llama.cpp: llama-server README (slots, continuous batching, -ngl, -c)
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- vLLM project (docs). vLLM documentation
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
- Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
- Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve