TensorRT and TensorRT-LLM in depth
TensorRT is NVIDIA’s compiler for neural networks: it reads a model graph once, folds, fuses and times kernels for one GPU, and writes an engine that runs fast but only on that GPU and that TensorRT version. TensorRT-LLM (now written “TensorRT LLM”) began as that idea applied to LLMs and has become a PyTorch-based serving library with paged KV cache, in-flight batching, FP8 and FP4 quantization and speculative decoding. This session explains both, with the numbers, and ends with a clear rule for when it is worth choosing it over vLLM.
What you will learn
- What the TensorRT builder does to a graph: parsing, constant folding, layer fusion, precision, kernel timing (tactics), memory planning, and the serialized engine
- How to count the memory traffic a fusion saves, and why fusion matters most for large activations and for many tiny kernels
- Optimization profiles (min, opt, max) for dynamic shapes, why the opt shape matters, and why one engine is tied to a GPU and a TensorRT version
- How precision works today: strongly typed networks, explicit Q/DQ quantization, FP8 and NVFP4 scales, and what changed in TensorRT 11
- TensorRT-LLM’s runtime: in-flight batching, the paged KV cache and its defaults, the capacity scheduler, CUDA graphs, quantization recipes and speculative decoding
- The history that explains the docs you will read: engines and plugins before version 1.0, the PyTorch backend after it, and the removal of the TensorRT backend in 2026
- When TensorRT-LLM beats vLLM, when it is not worth the effort, and where plain TensorRT is still the right tool (vision encoders, edge, video models)
Why this matters
PyTorch runs a model one operator at a time, deciding at every call which kernel to use. An inference compiler makes those decisions once, before serving: it looks at the whole graph, merges operators, picks the fastest kernel for each one on your exact GPU, and saves the result. TensorRT is NVIDIA’s compiler of this kind, and TensorRT-LLM is NVIDIA’s library for serving LLMs fast on its GPUs.
In session 123 you saw where the time goes in one decode step: memory-bound matrix multiplications, many tiny kernels, and launch gaps. In session 127 you met fusion and CUDA graphs as separate tools. TensorRT applies all of them together, plus kernel timing and low precision. TensorRT-LLM adds what an LLM server needs on top: a paged KV cache, in-flight batching, quantized checkpoints and speculative decoding.
There is a second reason to study this session carefully: these tools changed a lot in two years, and much of what you find online is out of date. TensorRT-LLM used to build TensorRT engines with a command called trtllm-build. In version 1.0 (September 2025) the PyTorch-based path became the default, and in mid-2026 the TensorRT engine path was removed. [Certain] If you know which idea belongs to which period, you can read old blog posts, old configs and job adverts correctly.
The book (Noyan et al., Chapter 7) places TensorRT next to ONNX and vLLM in the deployment chain of a VLM. A VLM often uses both worlds: the vision encoder is a fixed-shape network that TensorRT handles very well, and the language model is served by an LLM engine with a KV cache.
Two products, one idea
First, names and roles, because they are easy to mix up.
| TensorRT | TensorRT-LLM (TensorRT LLM) | |
|---|---|---|
| What it is | An SDK with a builder (the compiler) and a runtime, for any neural network | A Python library and server for LLM inference on NVIDIA GPUs |
| Input | An ONNX file or a network built with its API | A Hugging Face checkpoint, or a quantized checkpoint from NVIDIA Model Optimizer (ModelOpt) |
| Output | A serialized engine (also called a plan file) | A running model behind the LLM Python API or the trtllm-serve server |
| Shapes | Fixed, or ranges given by optimization profiles | Fully dynamic: requests of any length join and leave every step |
| Typical use | CNNs, ViT encoders, detection, ASR, embedding models, edge devices | Chat, RAG, agents and VLM language models at high throughput |
The shared idea is ahead-of-time specialization: do expensive decisions once, for one GPU and one set of shapes, so that serving does as little work as possible. TensorRT does it by compiling an engine. TensorRT-LLM today does it with hand-written and generated kernels for each model family, fused operations, and CUDA graphs captured for each decode batch size. [Certain]
A short history you need for reading the docs
- Before 1.0. You converted a checkpoint with a per-model
convert_checkpoint.py, then rantrtllm-buildto compile a TensorRT engine. LLM-specific work was done by plugins, custom TensorRT layers such as the GPT attention plugin and the GEMM plugin. [Certain for the tools; Likely for the plugin names] - 1.0, September 2025. The release notes say: “the PyTorch-based architecture is now stable and the default experience, and the LLM API is now stable.” [Certain]
- 1.3.0rc20, June 2026. Announced as the last release with the TensorRT backend. From 1.3.0rc21 on, the engine path was removed step by step:
LLM(backend="tensorrt"),trtllm-build,trtllm-refit, the conversion scripts and thetensorrtpip dependency are gone. Hugging Face checkpoints load directly, with no build step. [Certain]
So, in late 2026, “TensorRT-LLM” no longer runs TensorRT engines. The name stayed; the ideas (fused kernels, specialization per GPU, low precision) moved into a PyTorch runtime. TensorRT itself is very much alive (version 11 is current) and is the right tool for many non-LLM networks. [Certain]
The TensorRT builder, step by step
The builder turns a graph into an engine in a fixed order of passes. Figure 1 shows a small transformer MLP block going through them.
- input: ONNX model, optimization profiles (shape ranges), builder config (memory limit, compatibility level, timing cache)
- Parse ONNX into a network definition (layers and tensors) // unsupported ops need a plugin
- Fold constants: compute every subgraph whose inputs are known at build time (shape arithmetic, weight transposes)
- Remove dead layers: drop anything that does not feed an output
- Fuse layers: merge chains such as MatMul + bias + activation, or element-wise chains, into one kernel
- Fix precision of every tensor from the network types and the Q/DQ (quantize, dequantize) nodes
- for each fused layer: run every candidate kernel (tactic) at the opt shape on the real GPU, keep the fastest // formula (131.2)
- Plan memory: give activations that are never alive at the same time the same buffer
- Serialize: write the chosen kernels, weights and memory plan into the engine file
- output: an engine for this GPU type, this TensorRT version and these shape ranges
How to read Figure 1
Step through the passes. After parsing, the graph has 11 nodes, including a shape subgraph (Shape, Gather, Reshape) that the exporter added and a debug output nobody reads. Constant folding and dead-layer removal leave 8 nodes. Fusion leaves 3 kernels: the norm, the first MatMul with bias and GELU in its epilogue, and the second MatMul with bias and the residual add. The precision step labels tensors with their types; the tactic step times candidate kernels; the memory step reuses buffers. The last two steps show what is saved in the engine and what the runtime does with it.
Fusion: the bytes it saves
Fusion is the largest and most predictable gain. A kernel reads its inputs from HBM (the GPU’s main memory) and writes its output back. If the next kernel only uses that output once, the round trip is wasted: a fused kernel keeps the value in registers and goes on. Count the bytes of a chain of \(n\) element-wise operators on a tensor of \(N\) elements:
- Read the symbols
- \(r_k, w_k\)
- how many tensors of size \(N\) operator \(k\) reads and writes
- \(r_{\text{ext}}, w_{\text{out}}\)
- tensors the fused kernel must still read from outside and write for later use
- \(b\)
- bytes per element (2 for BF16, 1 for FP8)
- \(n_{\text{kern}}, t_0\)
- number of kernels and the fixed cost of each, 2 µs in our model
- \(\eta_B\, \mathrm{BW}\)
- achieved bandwidth; we use 80% of an H100 SXM’s 3.35 TB/s, as in session 123
A real chain in every Llama layer with FP8 matmuls: the attention output projection (a GEMM) is added to the residual stream, the sum is normalized with RMSNorm, and the result is quantized to FP8 for the next GEMM. Four operators: A = GEMM output, B = residual add, C = RMSNorm, D = FP8 quantize. Each boundary between them can be fused or not, with one rule: RMSNorm needs a whole row of 4,096 numbers, and a GEMM output tile holds only part of a row, so A, B and C cannot all be in one kernel. [Likely]
One tensor is \(N b = 4096 \cdot 4096 \cdot 2 = 33.6\) MB. Unfused: A writes the GEMM output (1 tensor); B reads it and the residual and writes the sum (3); C reads the sum and writes the normed tensor (2); D reads it and writes FP8 (1 + 0.5). In units of \(Nb\): \(1 + 3 + 2 + 1.5 = 7.5\), so \(Q = 251.7\) MB. With 3 element-wise kernels: \(t = 3 \cdot 2 + 251.7\ \text{MB} / 2.68\ \text{TB/s} = 99.9\ \mu s\).
Fuse B into the GEMM epilogue (the GEMM reads the residual and writes the sum) and fuse C with D (the norm writes FP8 directly): \(Q = 117.4\) MB, one element-wise kernel, \(t = 45.8\ \mu s\). That is 2.2× less time for the same math. In decode, with one token, the same chain moves only 61 KB and costs 6.0 µs unfused versus 2.0 µs fused: there, fusion saves kernels, not bytes.
How to read Figure 2
Each arrow down is a write to HBM, each arrow up a read. With no fusion there are 4 kernels and 7.5 tensors of traffic. Tick boxes to merge neighbours: a fused boundary removes the write and the read of the tensor between them (unless it is needed later, like the residual sum). A|B and B|C together is refused: the norm would have to live inside a GEMM tile. Move \(T\) down to 1 (decode): the bytes become tiny and only the number of kernels matters.
Kernel timing and optimization profiles
For one layer there are often dozens of kernels that compute the same result: different tile sizes, different numbers of pipeline stages, split-K or not. Each is a tactic. The builder does not guess which is fastest: it runs them on the GPU and measures. [Certain] This is why a build takes minutes, why it needs the target GPU, and why two builds can pick different tactics: timing noise makes close candidates swap. A timing cache stores the measurements so a rebuild is faster and repeatable. [Certain]
- Read the symbols
- \(\mathcal{T}(\text{layer})\)
- the set of tactics that can run this layer
- \(\hat t(\tau; s)\)
- the measured time of tactic \(\tau\) at input shape \(s\)
- \(s_{\min}, s_{\text{opt}}, s_{\max}\)
- the three shapes of an optimization profile
A dynamic dimension is marked with \(-1\) in the network. For each such input you give an optimization profile: the smallest shape allowed (min), the shape to optimize for (opt), and the largest shape (max). The TensorRT docs say a profile describes “a range of dimensions for each network input and the dimensions the auto-tuner will use for optimization”. You can add several profiles; they are numbered from 0, each execution context uses its own profile, and you switch with setOptimizationProfileAsync() before setting input shapes. Some tactics only work when \(\text{min} = \text{opt} = \text{max}\), so a wide range can rule out the fastest kernels. [Certain]
Why does opt matter? A GEMM kernel works on tiles of the output. If \(M\) (tokens in the batch) is not a multiple of the tile height \(m_t\), the last tile is padded:
Small tiles waste little padding but reach a lower fraction \(\epsilon(m_t)\) of peak compute \(F\); large tiles are efficient but waste more at small \(M\). In our toy model we use \(\epsilon = 0.25, 0.40, 0.60, 0.75, 0.80\) for tiles of 16, 32, 64, 128 and 256 rows. These values are our assumptions, not measurements. [Likely] The lesson does not depend on them: tune at a decode shape and you get a small-tile kernel; if the same engine then runs a 4,096-token prefill, it is slow.
\(\text{opt} = 8\) tokens: all tiles are memory-bound and tie at 14.6 µs, so the smallest tile (16) is kept. At \(M = 4096\) that tactic takes 557.9 µs, while the best tile (256) takes 175.7 µs: 3.2× slower. \(\text{opt} = 1{,}024\): the builder keeps the 256-row tile; at \(M = 8\) it still takes 14.6 µs, because at small \(M\) the GEMM is limited by reading 33.6 MB of weights, not by math. In this model a large opt costs little at small \(M\), but a small opt is expensive at large \(M\). Two profiles, one for decode and one for prefill, fix both. Old TensorRT-LLM engines had an option to build multiple profiles for exactly this reason. [Likely]
How to read Figure 3
The dashed curve is the engine, the solid orange curve is the best possible kernel for each shape. Where they overlap, the profile is well chosen. With \(\text{opt} = 8\) they split above 64 tokens, and at 4,096 tokens the engine is 3.2× slower than it could be. Move opt to 1,024 and the curves meet. Put the runtime tokens outside \([\text{min}, \text{max}]\) and the readout shows the error a real engine raises. Tick the second profile: each shape now uses the profile that covers it, and both decode and prefill run at the best speed.
An engine is tied to a GPU and a version
The tactics were timed on one GPU type, and the engine stores compiled kernels for that architecture. So by default, from the TensorRT docs: [Certain]
| Rule (default) | What fails | Escape hatch | Cost |
|---|---|---|---|
| Same TensorRT version, checked to major, minor, patch and build | An engine from 10.x on a 11.x runtime | Build with the version-compatible flag; a copy of the lean runtime is put into the plan. Within 11.x, and 10 → 11, are supported. | “version-compatible engines can be slower” |
| Same compute capability (major and minor) | An H100 engine (sm 9.0) on an L40S (sm 8.9) | Hardware compatibility level AMPERE_PLUS (any GPU of compute capability 8.0 or newer) or SAME_COMPUTE_CAPABILITY | “can have lower throughput and/or higher latency” |
| Same platform (operating system and CPU architecture) | A Linux x86 engine on Windows or on ARM | Cross-platform build options | may differ from a native build |
The practical rule: treat an engine like a compiled binary. Build it in CI for each GPU type you deploy on, store it next to the TensorRT version and the profiles used, and rebuild when either changes. And treat it like a binary for security too: the docs warn to deserialize only engines you built yourself or got through a trusted channel. [Certain]
Precision: FP16, INT8, FP8 and FP4
Lower precision helps in two ways you know from session 104: fewer bytes per weight (decode is memory-bound) and faster tensor cores (prefill is compute-bound). TensorRT supports BF16 and FP16, INT8, INT4 (weight-only), FP8 E4M3 and FP4 E2M1. [Certain] The way you ask for them changed:
- Older TensorRT (up to 10.x): builder flags such as FP16 and INT8 let the builder choose a lower precision per layer if it was faster (a “weakly typed” network). For INT8 without Q/DQ nodes you gave a calibrator that ran sample data to find ranges (“implicit quantization”).
- TensorRT 11: networks are strongly typed by default: every tensor has the type the model says, and the builder does not change it. Per-precision builder flags,
setPrecision,IInt8Calibratorand implicit quantization were removed. To quantize, you put explicit Q/DQ nodes (QuantizeLinear and DequantizeLinear) into the ONNX graph, usually with NVIDIA ModelOpt. [Certain]
A Q/DQ pair stores a scale. With symmetric per-tensor scaling:
- Read the symbols
- \(s\)
- the scale, one FP32 number per tensor (or per channel, or per block)
- \(\operatorname{rnd}\)
- round to the nearest value of the target format: integers for INT8, the FP8 or FP4 grid for floats
- \(q_{\max}\)
- the largest value the format can hold
When the builder sees DQ → MatMul → Q, it fuses them and runs the MatMul in the low precision; the scales go into the GEMM epilogue. FP4 has only 15 distinct values, so a single scale per tensor is far too coarse. NVIDIA’s NVFP4 gives each block of 16 values its own FP8 scale, plus one FP32 scale per tensor (details in session 105). That costs \(4 + 8/16 = 4.5\) bits per weight. [Certain]
Llama 3.1 8B has 8,030,261,248 parameters; 6,979,321,856 of them are in the linear layers of the 32 blocks, and the embedding and LM head (not tied) have 1,050,673,152 together. Keep the embedding, LM head and norms in BF16, quantize only the linear layers:
- BF16: 16.06 GB. FP8: \(6.979 \times 1 + 2.101\) GB of BF16 parts \(= 9.08\) GB. NVFP4: \(6.979 \times 0.5625 + 2.101 = 6.03\) GB.
- A decode step at batch 1 must read all weights. At the full 3.35 TB/s of an H100 SXM, the upper limit is \(3.35 / 16.06 = 208.6\) tokens per second in BF16 and 368.9 in FP8.
A scale example: if \(\max|x| = 3.2\), the FP8 scale is \(3.2/448 = 0.00714\), and \(x = 0.05\) becomes \(0.05/0.00714 = 7.0\), which E4M3 stores exactly.
TensorRT-LLM: the runtime of an LLM server
An LLM is a bad fit for a classic engine: the batch changes every step, each request has its own context length, and the KV cache grows token by token. So TensorRT-LLM is built around a runtime loop, not around a graph. One iteration of that loop:
- input: active requests, waiting queue, KV block pool, limits
max_batch_sizeandmax_num_tokens - Capacity scheduler: admit waiting requests while KV blocks allow // default policy GUARANTEED_NO_EVICT
- Micro-batch scheduler: pick context (prefill) chunks and generation (decode) requests so the step has at most
max_num_tokenstokens - Allocate KV blocks of 32 tokens for new tokens; reuse cached blocks of shared prefixes // block reuse is on by default
- Run one forward pass over all scheduled tokens // decode-only batches replay a captured CUDA graph
- Sample one token per generation request (or verify draft tokens if speculative decoding is on)
- Stream tokens out; finished requests free their blocks at once
- While the GPU runs this step, the CPU already prepares the next one // the overlap scheduler, on by default
The defaults in this algorithm come from the code of v1.4.0rc3 (November 2026): tokens_per_block = 32, enable_block_reuse = True, free_gpu_memory_fraction = 0.9, max_batch_size = 2048, max_num_tokens = 8192, enable_chunked_prefill = False, disable_overlap_scheduler = False, capacity policy GUARANTEED_NO_EVICT (others: MAX_UTILIZATION, STATIC_BATCH), and CUDA graphs enabled for batches that contain only decode requests. [Certain] Defaults change between versions; read them from your installed version.
In-flight batching
TensorRT-LLM’s name for continuous batching (also called iteration-level batching, from Orca): requests join and leave the batch at every step, instead of the batch waiting for its longest request. [Certain] Let \(L_i\) be the number of tokens request \(i\) generates and \(B\) the number of slots. Static batching runs each group \(g\) of \(B\) requests for \(\max_{i \in g} L_i\) steps. The fraction of slot-steps doing useful work is
where \(S_{\text{IFB}}\) is the number of steps in-flight batching needs. Only the final drain, when the queue is empty, leaves slots idle. Our model counts steps and assumes a step costs the same with any number of active requests, and that a new request’s prefill fits into its first step; both are simplifications, so read the speedups as an upper bound for decode-bound traffic. [Likely]
Output lengths 10, 50, 20, 40 and then 30, 30, 10, 60. Static: the first group runs 50 steps, the second 60: 110 steps. Useful work is 250 slot-steps out of \(4 \cdot 110 = 440\): \(U = 56.8\%\). In-flight: a slot that frees up at step 10 takes request 5 at once; the simulation needs 100 steps, \(U = 250/400 = 62.5\%\). With only 8 requests the final drain dominates. With 256 requests of 16 to 512 tokens and 8 slots, Figure 4 gives 14,534 static steps against 8,479: 1.71× fewer steps, and \(U\) goes from 56.8% to 97.3%.
How to read Figure 4
Each row of a chart is one slot; each colored bar is one request; gaps are idle slot-steps. In the static chart, every group ends together, at its longest request, and the shorter requests leave holes. In the in-flight chart, the next request starts in the step after a slot frees, so holes appear only at the end. Choose narrow lengths (200 to 300): static batching wastes little and the gain is about 1.1×. Wide lengths give about 1.7×. More slots make the final drain a larger part of the run.
The paged KV cache and how many requests fit
The KV cache is split into fixed blocks of 32 tokens, and a request holds a list of blocks, as in vLLM’s PagedAttention (session 134). A request wastes at most one partly filled block, instead of a whole reserved max_seq_len. At start-up, after the model is loaded and a test forward pass has measured activation memory, the runtime gives 90% of the remaining free memory to the block pool. [Certain] So:
- Read the symbols
- \(f\)
free_gpu_memory_fraction, 0.9 by default- \(M_{\text{GPU}}, M_W, M_{\text{act}}\)
- GPU memory, weight bytes, and activations plus workspace (we assume 4 GB) [Likely]
- \(s_{\text{tok}}\)
- KV bytes per token: \(2 \cdot n_{\text{layers}} \cdot n_{\text{kv}} \cdot d_{\text{head}} \cdot \text{bytes}\); for Llama 3.1 8B in BF16, \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes (128 KiB)
- \(b, \ell, \ell_{\max}\)
- block size (32), the current length of a request, and the maximum length a contiguous cache must reserve
BF16 weights and BF16 KV: \(0.9 \cdot (80 - 16.06 - 4) = 53.95\) GB of KV cache, which is 411,571 tokens. At an average length of 2,000 tokens a request holds \(\lceil 2000/32 \rceil = 63\) blocks (2,016 tokens), so 204 requests fit. A contiguous cache that reserves 8,192 tokens per request fits only 50. With FP8 weights and an FP8 KV cache: 60.23 GB, 918,989 tokens, 455 requests paged. These are memory limits; the compute of each step also grows with the batch.
How to read Figure 5
The bar is the GPU memory. The readout gives the KV pool in GB and tokens, and how many requests of the chosen average length fit with 32-token pages versus a contiguous reserve of \(\ell_{\max}\) per request. Switch to FP8 weights: the weights shrink by 6.98 GB and the pool grows by 90% of that, 6.28 GB. Switch to an FP8 KV cache: the pool holds twice as many tokens. On an L40S with BF16 everything, only 95 requests of 2,000 tokens fit.
Quantization and speculative decoding
Quantized models come as checkpoints made by ModelOpt, either your own (its llm_ptq example) or pre-quantized ones that NVIDIA publishes on Hugging Face, such as nvidia/Llama-3.1-8B-Instruct-FP8. The recipes in the current docs include FP8 per-tensor, FP8 block scaling, FP8 rowwise, NVFP4, W4A16 and W4A8 with AWQ or GPTQ, and FP8 or NVFP4 KV caches, each with its own GPU support. [Certain] The methods themselves are explained in session 107 and session 110.
For speculative decoding (session 128, session 129), the LLM API takes a speculative_config. The current docs list draft/target models, EAGLE-3, MTP (for models like DeepSeek that have multi-token prediction modules), PARD, DFlash, a model-free suffix automaton method, and n-gram drafting (deprecated). [Certain] Medusa, ReDrafter and EAGLE 1 and 2 belonged to the old engine path and are not supported in the PyTorch backend. [Likely] If a draft method proposes \(k\) tokens and each is accepted with probability \(\alpha\), one target step gives on average
For \(\alpha = 0.7\) and \(k = 3\) that is 2.53 tokens per step, so decode at small batch gets up to about 2.5× fewer target passes, minus the cost of drafting. At large batch the GPU is already busy, and the gain shrinks or turns negative. [Likely]
PyTorch code
You cannot run TensorRT without an NVIDIA GPU, so the first program reproduces the ideas of this session in plain PyTorch on CPU: (1) a fused and an unfused residual add + RMSNorm + FP8 quantize give the same bits but move very different bytes; (2) the toy tactic model of Figure 3; (3) in-flight batching with a paged block pool and a no-eviction admission rule. It is deterministic.
import math, torch torch.manual_seed(0) # 1. Fusion: residual add + RMSNorm + FP8 quantize, unfused vs fused T, d = 64, 4096 attn = torch.randn(T, d, dtype=torch.bfloat16); x = torch.randn(T, d, dtype=torch.bfloat16) w = torch.ones(d, dtype=torch.bfloat16); eps = 1e-6 def rms(h): hf = h.float(); return (hf * torch.rsqrt(hf.pow(2).mean(-1, keepdim=True) + eps)).to(h.dtype) * w def quant_fp8(n): # per-tensor scale, E4M3 max is 448 s = n.float().abs().amax() / 448.0 return (n.float() / s).to(torch.float8_e4m3fn), s def unfused(attn, x): # 3 kernels, 3 round trips to HBM h = x + attn # read 2, write 1 n = rms(h) # read 1, write 1 q, s = quant_fp8(n) # read 1, write 1 (1 byte) return h, q, s def fused(attn, x): # one kernel: same math, values stay in registers h = (x.float() + attn.float()).to(torch.bfloat16) return h, *quant_fp8(rms(h)) h1, q1, s1 = unfused(attn, x); h2, q2, s2 = fused(attn, x) print("same result:", torch.equal(h1, h2), torch.equal(q1.view(torch.uint8), q2.view(torch.uint8)), torch.equal(s1, s2)) Td = T * d print("HBM bytes unfused:", (4+2) * Td + (2+2) * Td + (2+1) * Td, " fused:", 4 * Td + 2 * Td + 1 * Td) # 2. Toy tactic selection: tune at opt, run at M N = K = 4096; F, BW, t0 = 989e12, 0.8 * 3.35e12, 2e-6 EFF = {16: .25, 32: .40, 64: .60, 128: .75, 256: .80} # our assumption, not measured def t(tm, M): Mp = math.ceil(M / tm) * tm return t0 + max(2 * Mp * N * K / (F * EFF[tm]), (2*N*K + 2*M*K + 2*M*N) / BW) pick = lambda M: min(EFF, key=lambda tm: (t(tm, M), tm)) for opt in (8, 1024): tm = pick(opt) print(f"opt={opt}: tile {tm}; at M=4096 {t(tm, 4096)*1e6:.1f} us vs best {t(pick(4096), 4096)*1e6:.1f} us") # 3. In-flight batching with a paged KV pool vs static batching g = torch.Generator().manual_seed(131) R, B, BLK = 64, 8, 32 prompt = torch.randint(32, 257, (R,), generator=g).tolist() out = torch.randint(16, 513, (R,), generator=g).tolist() def static_iters(): return sum(max(out[i:i+B]) for i in range(0, R, B)) def ifb_iters(pool_blocks): queue, active, it, peak = list(range(R)), [], 0, 0 free = pool_blocks need = lambda r: math.ceil((prompt[r] + out[r]) / BLK) # reserve for the whole request while queue or active: while queue and len(active) < B and need(queue[0]) <= free: # admit only if it can finish r = queue.pop(0); free -= need(r); active.append([r, out[r]]) for a in active: a[1] -= 1 # one token each this step it += 1; peak = max(peak, pool_blocks - free) for a in [a for a in active if a[1] == 0]: free += need(a[0]); active.remove(a) return it, peak print("static iterations:", static_iters(), " tokens:", sum(out)) for pool in (400, 200, 100): it, peak = ifb_iters(pool) print(f"IFB, pool {pool} blocks: {it} iterations, peak {peak} blocks in use")
Output with PyTorch 2.8 on CPU (the same every run):
same result: True True True HBM bytes unfused: 3407872 fused: 1835008 opt=8: tile 16; at M=4096 557.9 us vs best 175.7 us opt=1024: tile 256; at M=4096 175.7 us vs best 175.7 us static iterations: 3652 tokens: 19079 IFB, pool 400 blocks: 2638 iterations, peak 158 blocks in use IFB, pool 200 blocks: 2638 iterations, peak 158 blocks in use IFB, pool 100 blocks: 3610 iterations, peak 100 blocks in use
Read the last three lines: with 400 or 200 blocks, the 8 slots are the limit and in-flight batching needs 28% fewer steps than static batching. With 100 blocks, the no-eviction rule cannot admit enough requests and almost all of the gain is lost. That is the reason FP8 KV caches and smaller blocks matter for throughput.
The real tools, for an NVIDIA GPU (not run here; check the flags against your installed versions):
# TensorRT: ONNX -> engine with one optimization profile, then benchmark it trtexec --onnx=vit_encoder.onnx --saveEngine=vit_encoder.plan \ --minShapes=pixels:1x3x224x224 --optShapes=pixels:16x3x224x224 --maxShapes=pixels:64x3x224x224 \ --timingCacheFile=timing.cache trtexec --loadEngine=vit_encoder.plan --shapes=pixels:16x3x224x224 # TensorRT-LLM (PyTorch backend): load a Hugging Face or ModelOpt checkpoint, no build step from tensorrt_llm import LLM, SamplingParams from tensorrt_llm.llmapi import KvCacheConfig llm = LLM(model="nvidia/Llama-3.1-8B-Instruct-FP8", kv_cache_config=KvCacheConfig(free_gpu_memory_fraction=0.85, enable_block_reuse=True), max_batch_size=256, max_num_tokens=8192) for o in llm.generate(["The capital of France is"], SamplingParams(max_tokens=32, temperature=0.0)): print(o.outputs[0].text) # or as an HTTP server: trtllm-serve nvidia/Llama-3.1-8B-Instruct-FP8
When TensorRT-LLM beats vLLM, and when it is not worth it
Both are open source, both are Python on top of PyTorch with custom kernels, and both have the main features: paged KV cache, continuous batching, chunked prefill, prefix caching, FP8, speculative decoding, tensor and pipeline parallelism, and disaggregated prefill and decode. So the choice is not “fast versus slow” in general. It depends on your hardware, your model and your team. We give no throughput numbers here: published comparisons depend on versions, models, GPUs and settings, and change with every release. Benchmark your own workload (session 142).
| Situation | Lean towards | Why |
|---|---|---|
| Large fleet of new NVIDIA GPUs (Hopper, Blackwell), a few stable models, cost per token is the main metric | TensorRT-LLM | NVIDIA’s kernels and formats (NVFP4 on Blackwell, FP8 block scaling) usually arrive here first; a few percent of throughput is worth engineering time at this scale [Likely] |
| Serving with NVIDIA’s stack (Dynamo, NIM, Triton Inference Server) | TensorRT-LLM | first-class integration [Likely] |
| AMD, TPU, CPU or mixed hardware | vLLM | TensorRT-LLM runs only on NVIDIA GPUs [Certain] |
| Many models, new models every week, research or prototypes | vLLM | broader model coverage and community, faster support of new architectures [Likely] |
| Small traffic, one GPU, latency already meets the SLO | whatever you run now | the gain cannot pay for migration and testing |
| A VLM’s vision encoder, a detector, an embedding model, Jetson devices | plain TensorRT | fixed shapes, classic fusion and INT8/FP8 engines are a perfect fit |
- input: model, GPU type, traffic trace (prompt and output lengths, arrival rate), latency SLOs (time to first token, time per output token)
- Check the support matrix: is the model, the quantization recipe and the speculative method supported on your GPU? If not, stop
- Replay the same trace on both engines, same precision, same parallelism, same GPU count // never compare FP8 to BF16
- Tune both: batch limits, max tokens per step, chunked prefill, CUDA graph sizes, KV fraction
- Measure goodput: requests per second that meet both SLOs
- Gain \(\leftarrow\) GPU-hours saved per month at your traffic; Cost \(\leftarrow\) engineering time, testing, on-call, upgrade cadence
- output: switch only if Gain is clearly larger than Cost, and plan to re-test at each upgrade of either engine
Where this is used
| Application | Tool | What matters most |
|---|---|---|
| High-volume chat and API serving on H100, H200 or B200 | TensorRT-LLM | FP8 or NVFP4 checkpoints, in-flight batching, FP8 KV cache, speculative decoding (EAGLE-3, MTP) |
| VLM serving: image or video question answering | TensorRT engine for the vision encoder, an LLM engine for the decoder | fixed-shape encoder profiles (batch of frames), prefix caching of the system prompt, many image tokens in prefill |
| Video retrieval and Video-RAG indexing | TensorRT (embedding and frame encoders) | large static batches, FP8 or INT8 engines, one optimization profile per frame batch size |
| Action recognition and key-moment detection on live streams | TensorRT on server GPUs or Jetson | low latency at small batch, fusion, engines built on the device type |
| Video fingerprinting and near-duplicate detection | TensorRT | throughput per GPU, INT8 accuracy checks against the FP32 model |
| Video segmentation (per-frame masks) | TensorRT | dynamic resolution needs well-chosen profiles; otherwise rebuild per resolution |
| Agents and long-context RAG | TensorRT-LLM or vLLM | KV block reuse for shared prefixes, chunked prefill so long prompts do not stall decode |
Common mistakes
- Copying an engine to another GPU type or TensorRT version. It fails to load (or, with compatibility modes, runs slower). Build per GPU type in CI and store the version with the engine.
- One optimization profile with a decode-size opt for all shapes. Large inputs then run small-tile kernels, 3.2× slower in our model. Use the expected shape as opt, or several profiles.
- A huge \([\text{min}, \text{max}]\) range “to be safe”. Memory for activations can grow with max, and some fast tactics need \(\text{min} = \text{opt} = \text{max}\). Keep ranges as tight as the traffic allows.
- Following a 2024 tutorial.
trtllm-build,convert_checkpoint.pyandbackend="tensorrt"are gone; INT8 calibrators and the FP16 builder flag are gone from TensorRT 11. Check the version of every guide you use. - Comparing engines at different precision. FP8 TensorRT-LLM against BF16 vLLM measures the format, not the engine.
- Quantizing without an accuracy check. FP8 is usually safe for LLM weights and activations; NVFP4 and INT4 need an evaluation on your task. Engines can also differ slightly from PyTorch because of other kernels and fusion order.
- Setting
free_gpu_memory_fractionwithout leaving room. Another process, a LoRA adapter cache or larger CUDA graphs can then run out of memory. Measure peak memory under real traffic. - Loading engines from strangers. An engine is executable code for the runtime; the docs warn to deserialize only trusted engines.
Interview questions
What does TensorRT do to a model when it builds an engine?basic
Why does fusion help, and when does it help most?core
What is an optimization profile, and what happens if you choose opt badly?core
Your colleague built an engine on an H100 and it fails to load on an L40S. Why, and what are the options?basic
How did precision control change in TensorRT 11?advanced
What is in-flight batching, and how much does it gain?core
How many 2,000-token requests of Llama 3.1 8B fit in the KV cache of an 80 GB H100 with TensorRT-LLM defaults?core
Does TensorRT-LLM still use TensorRT?advanced
What does the GUARANTEED_NO_EVICT capacity policy trade off?advanced
When would you not adopt TensorRT-LLM even if it is faster in a benchmark?core
How does speculative decoding interact with batch size in a server like TensorRT-LLM?advanced
Exercises
- Use (131.1) with our constants for the chain of Figure 2 at \(T = 2048\): give the bytes and time without fusion and with A|B and C|D fused.
- Compute the KV bytes per token of Llama 3.1 70B (80 layers, 8 KV heads of size 128) in BF16 and in FP8.
- Two slots, four requests with output lengths 5, 100, 100, 5 in this order. Count the steps of static and of in-flight batching, and the utilization of each.
- An engine has one profile with min 1, opt 8, max 8,192 tokens. Prefill batches of 4,096 tokens are slow. Use Figure 3 to explain why and propose two fixes.
- Will an engine built on an H100 SXM load on an H100 PCIe with the same TensorRT version? On an A100? Explain with compute capabilities (9.0, 9.0, 8.0).
- For Llama 3.1 8B on an H100 with FP8 weights and an FP8 KV cache, how many 2,000-token requests fit with 32-token pages, and with a contiguous reserve of 8,192 tokens?
Answerscheck
Key takeaways
- TensorRT compiles a graph once: fold, fuse, fix precision, time tactics on the real GPU, plan memory, serialize. The engine is fast and specific to one GPU type, one TensorRT version and the given shape ranges.
- Fusion saves HBM round trips (131.1): 2.2× less time for add + RMSNorm + FP8 quantize in prefill, and fewer kernels in decode.
- Optimization profiles give \([\text{min}, \text{max}]\) and the opt shape used for tuning (131.2, 131.3). Tune for the shapes you serve, or use several profiles.
- TensorRT 11 is strongly typed; quantize with explicit Q/DQ nodes (131.4). FP8 needs Ada or Hopper, NVFP4 needs Blackwell.
- TensorRT-LLM today is a PyTorch runtime: in-flight batching (131.5), 32-token KV blocks with reuse and 90% of free memory (131.6), CUDA graphs and an overlap scheduler for decode, ModelOpt checkpoints, and speculative decoding (131.7). Its TensorRT engine path was removed in 2026.
- Choose it over vLLM for large NVIDIA fleets and stable models after a like-for-like goodput test; it is not worth it for non-NVIDIA hardware, fast-changing models or small traffic.
Key formulas
References for this session
- NVIDIA (docs). TensorRT documentation
- NVIDIA (code). TensorRT-LLM
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- NVIDIA (docs). TensorRT documentation
- NVIDIA (code). TensorRT-LLM
- NVIDIA (docs). TensorRT documentation: Engine Compatibility (version, hardware and platform compatibility)
- NVIDIA (docs). TensorRT documentation: Dynamic Shapes, Core Concepts (optimization profiles)
- NVIDIA (docs). TensorRT 11.0.0 Release Notes (strongly typed networks by default, implicit quantization removed)
- NVIDIA (docs). TensorRT LLM: Migration Guide, TensorRT Backend Removed
- NVIDIA (docs). TensorRT LLM: Quantization (recipes and hardware support matrix)
- NVIDIA (docs). TensorRT LLM: Speculative Decoding
- NVIDIA (code). TensorRT LLM v1.4.0rc3: tensorrt_llm/llmapi/llm_args.py (KvCacheConfig, SchedulerConfig, CudaGraphConfig defaults)
- vLLM project (docs). vLLM documentation
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
- Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- Micikevicius et al. (2022). FP8 Formats for Deep Learning
- Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
- Li et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models