LLM & VLM courseBlog
131Track 5: LLM inference and serving · session 54 of 65

TensorRT and TensorRT-LLM in depth

TensorRT is NVIDIA’s compiler for neural networks: it reads a model graph once, folds, fuses and times kernels for one GPU, and writes an engine that runs fast but only on that GPU and that TensorRT version. TensorRT-LLM (now written “TensorRT LLM”) began as that idea applied to LLMs and has become a PyTorch-based serving library with paged KV cache, in-flight batching, FP8 and FP4 quantization and speculative decoding. This session explains both, with the numbers, and ends with a clear rule for when it is worth choosing it over vLLM.

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 to 1.5 hours)Level: IntermediateBefore this: Sessions 123 (runtime profiling), 127 (graph-level optimization), 105 (number formats) and 128 (speculative decoding math)

What you will learn

  • What the TensorRT builder does to a graph: parsing, constant folding, layer fusion, precision, kernel timing (tactics), memory planning, and the serialized engine
  • How to count the memory traffic a fusion saves, and why fusion matters most for large activations and for many tiny kernels
  • Optimization profiles (min, opt, max) for dynamic shapes, why the opt shape matters, and why one engine is tied to a GPU and a TensorRT version
  • How precision works today: strongly typed networks, explicit Q/DQ quantization, FP8 and NVFP4 scales, and what changed in TensorRT 11
  • TensorRT-LLM’s runtime: in-flight batching, the paged KV cache and its defaults, the capacity scheduler, CUDA graphs, quantization recipes and speculative decoding
  • The history that explains the docs you will read: engines and plugins before version 1.0, the PyTorch backend after it, and the removal of the TensorRT backend in 2026
  • When TensorRT-LLM beats vLLM, when it is not worth the effort, and where plain TensorRT is still the right tool (vision encoders, edge, video models)

Why this matters

PyTorch runs a model one operator at a time, deciding at every call which kernel to use. An inference compiler makes those decisions once, before serving: it looks at the whole graph, merges operators, picks the fastest kernel for each one on your exact GPU, and saves the result. TensorRT is NVIDIA’s compiler of this kind, and TensorRT-LLM is NVIDIA’s library for serving LLMs fast on its GPUs.

In session 123 you saw where the time goes in one decode step: memory-bound matrix multiplications, many tiny kernels, and launch gaps. In session 127 you met fusion and CUDA graphs as separate tools. TensorRT applies all of them together, plus kernel timing and low precision. TensorRT-LLM adds what an LLM server needs on top: a paged KV cache, in-flight batching, quantized checkpoints and speculative decoding.

There is a second reason to study this session carefully: these tools changed a lot in two years, and much of what you find online is out of date. TensorRT-LLM used to build TensorRT engines with a command called trtllm-build. In version 1.0 (September 2025) the PyTorch-based path became the default, and in mid-2026 the TensorRT engine path was removed. [Certain] If you know which idea belongs to which period, you can read old blog posts, old configs and job adverts correctly.

The book (Noyan et al., Chapter 7) places TensorRT next to ONNX and vLLM in the deployment chain of a VLM. A VLM often uses both worlds: the vision encoder is a fixed-shape network that TensorRT handles very well, and the language model is served by an LLM engine with a KV cache.

Two products, one idea

First, names and roles, because they are easy to mix up.

TensorRTTensorRT-LLM (TensorRT LLM)
What it isAn SDK with a builder (the compiler) and a runtime, for any neural networkA Python library and server for LLM inference on NVIDIA GPUs
InputAn ONNX file or a network built with its APIA Hugging Face checkpoint, or a quantized checkpoint from NVIDIA Model Optimizer (ModelOpt)
OutputA serialized engine (also called a plan file)A running model behind the LLM Python API or the trtllm-serve server
ShapesFixed, or ranges given by optimization profilesFully dynamic: requests of any length join and leave every step
Typical useCNNs, ViT encoders, detection, ASR, embedding models, edge devicesChat, RAG, agents and VLM language models at high throughput

The shared idea is ahead-of-time specialization: do expensive decisions once, for one GPU and one set of shapes, so that serving does as little work as possible. TensorRT does it by compiling an engine. TensorRT-LLM today does it with hand-written and generated kernels for each model family, fused operations, and CUDA graphs captured for each decode batch size. [Certain]

A short history you need for reading the docs

  • Before 1.0. You converted a checkpoint with a per-model convert_checkpoint.py, then ran trtllm-build to compile a TensorRT engine. LLM-specific work was done by plugins, custom TensorRT layers such as the GPT attention plugin and the GEMM plugin. [Certain for the tools; Likely for the plugin names]
  • 1.0, September 2025. The release notes say: “the PyTorch-based architecture is now stable and the default experience, and the LLM API is now stable.” [Certain]
  • 1.3.0rc20, June 2026. Announced as the last release with the TensorRT backend. From 1.3.0rc21 on, the engine path was removed step by step: LLM(backend="tensorrt"), trtllm-build, trtllm-refit, the conversion scripts and the tensorrt pip dependency are gone. Hugging Face checkpoints load directly, with no build step. [Certain]

So, in late 2026, “TensorRT-LLM” no longer runs TensorRT engines. The name stayed; the ideas (fused kernels, specialization per GPU, low precision) moved into a PyTorch runtime. TensorRT itself is very much alive (version 11 is current) and is the right tool for many non-LLM networks. [Certain]

The TensorRT builder, step by step

The builder turns a graph into an engine in a fixed order of passes. Figure 1 shows a small transformer MLP block going through them.

Algorithm 1: what the TensorRT builder does
  1. input: ONNX model, optimization profiles (shape ranges), builder config (memory limit, compatibility level, timing cache)
  2. Parse ONNX into a network definition (layers and tensors) // unsupported ops need a plugin
  3. Fold constants: compute every subgraph whose inputs are known at build time (shape arithmetic, weight transposes)
  4. Remove dead layers: drop anything that does not feed an output
  5. Fuse layers: merge chains such as MatMul + bias + activation, or element-wise chains, into one kernel
  6. Fix precision of every tensor from the network types and the Q/DQ (quantize, dequantize) nodes
  7. for each fused layer: run every candidate kernel (tactic) at the opt shape on the real GPU, keep the fastest // formula (131.2)
  8. Plan memory: give activations that are never alive at the same time the same buffer
  9. Serialize: write the chosen kernels, weights and memory plan into the engine file
  10. output: an engine for this GPU type, this TensorRT version and these shape ranges
Figure 1. The TensorRT build and run phases for one transformer MLP block exported to ONNX. The top row is the pass, the bottom row is the graph after that pass. Node counts are for this small example.
How to read Figure 1

Step through the passes. After parsing, the graph has 11 nodes, including a shape subgraph (Shape, Gather, Reshape) that the exporter added and a debug output nobody reads. Constant folding and dead-layer removal leave 8 nodes. Fusion leaves 3 kernels: the norm, the first MatMul with bias and GELU in its epilogue, and the second MatMul with bias and the residual add. The precision step labels tensors with their types; the tactic step times candidate kernels; the memory step reuses buffers. The last two steps show what is saved in the engine and what the runtime does with it.

Fusion: the bytes it saves

Fusion is the largest and most predictable gain. A kernel reads its inputs from HBM (the GPU’s main memory) and writes its output back. If the next kernel only uses that output once, the round trip is wasted: a fused kernel keeps the value in registers and goes on. Count the bytes of a chain of \(n\) element-wise operators on a tensor of \(N\) elements:

\[ Q_{\text{unfused}} = \sum_{k=1}^{n} (r_k + w_k)\, N b, \qquad Q_{\text{fused}} = (r_{\text{ext}} + w_{\text{out}})\, N b, \qquad t \approx n_{\text{kern}}\, t_0 + \frac{Q}{\eta_B\, \mathrm{BW}} \tag{131.1} \]
Read the symbols
\(r_k, w_k\)
how many tensors of size \(N\) operator \(k\) reads and writes
\(r_{\text{ext}}, w_{\text{out}}\)
tensors the fused kernel must still read from outside and write for later use
\(b\)
bytes per element (2 for BF16, 1 for FP8)
\(n_{\text{kern}}, t_0\)
number of kernels and the fixed cost of each, 2 µs in our model
\(\eta_B\, \mathrm{BW}\)
achieved bandwidth; we use 80% of an H100 SXM’s 3.35 TB/s, as in session 123

A real chain in every Llama layer with FP8 matmuls: the attention output projection (a GEMM) is added to the residual stream, the sum is normalized with RMSNorm, and the result is quantized to FP8 for the next GEMM. Four operators: A = GEMM output, B = residual add, C = RMSNorm, D = FP8 quantize. Each boundary between them can be fused or not, with one rule: RMSNorm needs a whole row of 4,096 numbers, and a GEMM output tile holds only part of a row, so A, B and C cannot all be in one kernel. [Likely]

Worked example: prefill of 4,096 tokens, hidden size 4,096, BF16

One tensor is \(N b = 4096 \cdot 4096 \cdot 2 = 33.6\) MB. Unfused: A writes the GEMM output (1 tensor); B reads it and the residual and writes the sum (3); C reads the sum and writes the normed tensor (2); D reads it and writes FP8 (1 + 0.5). In units of \(Nb\): \(1 + 3 + 2 + 1.5 = 7.5\), so \(Q = 251.7\) MB. With 3 element-wise kernels: \(t = 3 \cdot 2 + 251.7\ \text{MB} / 2.68\ \text{TB/s} = 99.9\ \mu s\).

Fuse B into the GEMM epilogue (the GEMM reads the residual and writes the sum) and fuse C with D (the norm writes FP8 directly): \(Q = 117.4\) MB, one element-wise kernel, \(t = 45.8\ \mu s\). That is 2.2× less time for the same math. In decode, with one token, the same chain moves only 61 KB and costs 6.0 µs unfused versus 2.0 µs fused: there, fusion saves kernels, not bytes.

GEMMelement-wise kernelHBM traffic (read or write)
Figure 2. Memory traffic of the chain GEMM output → residual add → RMSNorm → FP8 quantize, from formula (131.1), hidden size 4,096, BF16 activations. Boxes inside one solid outline are one kernel. The GEMM’s own weight reads and math are not counted.
How to read Figure 2

Each arrow down is a write to HBM, each arrow up a read. With no fusion there are 4 kernels and 7.5 tensors of traffic. Tick boxes to merge neighbours: a fused boundary removes the write and the read of the tensor between them (unless it is needed later, like the residual sum). A|B and B|C together is refused: the norm would have to live inside a GEMM tile. Move \(T\) down to 1 (decode): the bytes become tiny and only the number of kernels matters.

Kernel timing and optimization profiles

For one layer there are often dozens of kernels that compute the same result: different tile sizes, different numbers of pipeline stages, split-K or not. Each is a tactic. The builder does not guess which is fastest: it runs them on the GPU and measures. [Certain] This is why a build takes minutes, why it needs the target GPU, and why two builds can pick different tactics: timing noise makes close candidates swap. A timing cache stores the measurements so a rebuild is faster and repeatable. [Certain]

\[ \tau^{*} = \argmin_{\tau \in \mathcal{T}(\text{layer})} \hat t(\tau;\ s_{\text{opt}}), \qquad \text{runtime cost at shape } s: \ t(\tau^{*};\ s), \quad s_{\min} \le s \le s_{\max} \tag{131.2} \]
Read the symbols
\(\mathcal{T}(\text{layer})\)
the set of tactics that can run this layer
\(\hat t(\tau; s)\)
the measured time of tactic \(\tau\) at input shape \(s\)
\(s_{\min}, s_{\text{opt}}, s_{\max}\)
the three shapes of an optimization profile

A dynamic dimension is marked with \(-1\) in the network. For each such input you give an optimization profile: the smallest shape allowed (min), the shape to optimize for (opt), and the largest shape (max). The TensorRT docs say a profile describes “a range of dimensions for each network input and the dimensions the auto-tuner will use for optimization”. You can add several profiles; they are numbered from 0, each execution context uses its own profile, and you switch with setOptimizationProfileAsync() before setting input shapes. Some tactics only work when \(\text{min} = \text{opt} = \text{max}\), so a wide range can rule out the fastest kernels. [Certain]

Why does opt matter? A GEMM kernel works on tiles of the output. If \(M\) (tokens in the batch) is not a multiple of the tile height \(m_t\), the last tile is padded:

\[ M_{\text{pad}} = \left\lceil \frac{M}{m_t} \right\rceil m_t, \qquad t(m_t, M) \approx t_0 + \max\!\left( \frac{2 M_{\text{pad}} N K}{\epsilon(m_t)\, F},\ \frac{2NK + 2MK + 2MN}{\eta_B\, \mathrm{BW}} \right) \tag{131.3} \]

Small tiles waste little padding but reach a lower fraction \(\epsilon(m_t)\) of peak compute \(F\); large tiles are efficient but waste more at small \(M\). In our toy model we use \(\epsilon = 0.25, 0.40, 0.60, 0.75, 0.80\) for tiles of 16, 32, 64, 128 and 256 rows. These values are our assumptions, not measurements. [Likely] The lesson does not depend on them: tune at a decode shape and you get a small-tile kernel; if the same engine then runs a 4,096-token prefill, it is slow.

Worked example: one profile for decode and prefill (\(N = K = 4096\), H100, BF16)

\(\text{opt} = 8\) tokens: all tiles are memory-bound and tie at 14.6 µs, so the smallest tile (16) is kept. At \(M = 4096\) that tactic takes 557.9 µs, while the best tile (256) takes 175.7 µs: 3.2× slower. \(\text{opt} = 1{,}024\): the builder keeps the 256-row tile; at \(M = 8\) it still takes 14.6 µs, because at small \(M\) the GEMM is limited by reading 33.6 MB of weights, not by math. In this model a large opt costs little at small \(M\), but a small opt is expensive at large \(M\). Two profiles, one for decode and one for prefill, fix both. Old TensorRT-LLM engines had an option to build multiple profiles for exactly this reason. [Likely]

tactic chosen at opt (engine)best tactic for each shapeallowed range of a profile
Figure 3. Toy model (131.3) of one 4096 × 4096 GEMM in an engine. The builder keeps the tactic that is fastest at opt; the curve shows what that choice costs at every other shape. Log scales on both axes.
How to read Figure 3

The dashed curve is the engine, the solid orange curve is the best possible kernel for each shape. Where they overlap, the profile is well chosen. With \(\text{opt} = 8\) they split above 64 tokens, and at 4,096 tokens the engine is 3.2× slower than it could be. Move opt to 1,024 and the curves meet. Put the runtime tokens outside \([\text{min}, \text{max}]\) and the readout shows the error a real engine raises. Tick the second profile: each shape now uses the profile that covers it, and both decode and prefill run at the best speed.

An engine is tied to a GPU and a version

The tactics were timed on one GPU type, and the engine stores compiled kernels for that architecture. So by default, from the TensorRT docs: [Certain]

Rule (default)What failsEscape hatchCost
Same TensorRT version, checked to major, minor, patch and buildAn engine from 10.x on a 11.x runtimeBuild with the version-compatible flag; a copy of the lean runtime is put into the plan. Within 11.x, and 10 → 11, are supported.“version-compatible engines can be slower”
Same compute capability (major and minor)An H100 engine (sm 9.0) on an L40S (sm 8.9)Hardware compatibility level AMPERE_PLUS (any GPU of compute capability 8.0 or newer) or SAME_COMPUTE_CAPABILITY“can have lower throughput and/or higher latency”
Same platform (operating system and CPU architecture)A Linux x86 engine on Windows or on ARMCross-platform build optionsmay differ from a native build

The practical rule: treat an engine like a compiled binary. Build it in CI for each GPU type you deploy on, store it next to the TensorRT version and the profiles used, and rebuild when either changes. And treat it like a binary for security too: the docs warn to deserialize only engines you built yourself or got through a trusted channel. [Certain]

Precision: FP16, INT8, FP8 and FP4

Lower precision helps in two ways you know from session 104: fewer bytes per weight (decode is memory-bound) and faster tensor cores (prefill is compute-bound). TensorRT supports BF16 and FP16, INT8, INT4 (weight-only), FP8 E4M3 and FP4 E2M1. [Certain] The way you ask for them changed:

  • Older TensorRT (up to 10.x): builder flags such as FP16 and INT8 let the builder choose a lower precision per layer if it was faster (a “weakly typed” network). For INT8 without Q/DQ nodes you gave a calibrator that ran sample data to find ranges (“implicit quantization”).
  • TensorRT 11: networks are strongly typed by default: every tensor has the type the model says, and the builder does not change it. Per-precision builder flags, setPrecision, IInt8Calibrator and implicit quantization were removed. To quantize, you put explicit Q/DQ nodes (QuantizeLinear and DequantizeLinear) into the ONNX graph, usually with NVIDIA ModelOpt. [Certain]

A Q/DQ pair stores a scale. With symmetric per-tensor scaling:

\[ s = \frac{\max |x|}{q_{\max}}, \qquad \hat x = s \cdot \operatorname{clamp}\!\big(\operatorname{rnd}(x / s),\ -q_{\max},\ q_{\max}\big), \qquad q_{\max} = \begin{cases} 127 & \text{INT8} \\ 448 & \text{FP8 E4M3} \\ 6 & \text{FP4 E2M1} \end{cases} \tag{131.4} \]
Read the symbols
\(s\)
the scale, one FP32 number per tensor (or per channel, or per block)
\(\operatorname{rnd}\)
round to the nearest value of the target format: integers for INT8, the FP8 or FP4 grid for floats
\(q_{\max}\)
the largest value the format can hold

When the builder sees DQ → MatMul → Q, it fuses them and runs the MatMul in the low precision; the scales go into the GEMM epilogue. FP4 has only 15 distinct values, so a single scale per tensor is far too coarse. NVIDIA’s NVFP4 gives each block of 16 values its own FP8 scale, plus one FP32 scale per tensor (details in session 105). That costs \(4 + 8/16 = 4.5\) bits per weight. [Certain]

Worked example: Llama 3.1 8B weight bytes and the decode speed limit

Llama 3.1 8B has 8,030,261,248 parameters; 6,979,321,856 of them are in the linear layers of the 32 blocks, and the embedding and LM head (not tied) have 1,050,673,152 together. Keep the embedding, LM head and norms in BF16, quantize only the linear layers:

  • BF16: 16.06 GB. FP8: \(6.979 \times 1 + 2.101\) GB of BF16 parts \(= 9.08\) GB. NVFP4: \(6.979 \times 0.5625 + 2.101 = 6.03\) GB.
  • A decode step at batch 1 must read all weights. At the full 3.35 TB/s of an H100 SXM, the upper limit is \(3.35 / 16.06 = 208.6\) tokens per second in BF16 and 368.9 in FP8.

A scale example: if \(\max|x| = 3.2\), the FP8 scale is \(3.2/448 = 0.00714\), and \(x = 0.05\) becomes \(0.05/0.00714 = 7.0\), which E4M3 stores exactly.

TensorRT-LLM: the runtime of an LLM server

An LLM is a bad fit for a classic engine: the batch changes every step, each request has its own context length, and the KV cache grows token by token. So TensorRT-LLM is built around a runtime loop, not around a graph. One iteration of that loop:

Algorithm 2: one iteration of the TensorRT-LLM executor (simplified)
  1. input: active requests, waiting queue, KV block pool, limits max_batch_size and max_num_tokens
  2. Capacity scheduler: admit waiting requests while KV blocks allow // default policy GUARANTEED_NO_EVICT
  3. Micro-batch scheduler: pick context (prefill) chunks and generation (decode) requests so the step has at most max_num_tokens tokens
  4. Allocate KV blocks of 32 tokens for new tokens; reuse cached blocks of shared prefixes // block reuse is on by default
  5. Run one forward pass over all scheduled tokens // decode-only batches replay a captured CUDA graph
  6. Sample one token per generation request (or verify draft tokens if speculative decoding is on)
  7. Stream tokens out; finished requests free their blocks at once
  8. While the GPU runs this step, the CPU already prepares the next one // the overlap scheduler, on by default

The defaults in this algorithm come from the code of v1.4.0rc3 (November 2026): tokens_per_block = 32, enable_block_reuse = True, free_gpu_memory_fraction = 0.9, max_batch_size = 2048, max_num_tokens = 8192, enable_chunked_prefill = False, disable_overlap_scheduler = False, capacity policy GUARANTEED_NO_EVICT (others: MAX_UTILIZATION, STATIC_BATCH), and CUDA graphs enabled for batches that contain only decode requests. [Certain] Defaults change between versions; read them from your installed version.

In-flight batching

TensorRT-LLM’s name for continuous batching (also called iteration-level batching, from Orca): requests join and leave the batch at every step, instead of the batch waiting for its longest request. [Certain] Let \(L_i\) be the number of tokens request \(i\) generates and \(B\) the number of slots. Static batching runs each group \(g\) of \(B\) requests for \(\max_{i \in g} L_i\) steps. The fraction of slot-steps doing useful work is

\[ U_{\text{static}} = \frac{\sum_i L_i}{B \sum_{g} \max_{i \in g} L_i}, \qquad U_{\text{IFB}} = \frac{\sum_i L_i}{B \cdot S_{\text{IFB}}} \ \longrightarrow\ 1 \ \text{ while the queue is not empty} \tag{131.5} \]

where \(S_{\text{IFB}}\) is the number of steps in-flight batching needs. Only the final drain, when the queue is empty, leaves slots idle. Our model counts steps and assumes a step costs the same with any number of active requests, and that a new request’s prefill fits into its first step; both are simplifications, so read the speedups as an upper bound for decode-bound traffic. [Likely]

Worked example: 8 requests, 4 slots

Output lengths 10, 50, 20, 40 and then 30, 30, 10, 60. Static: the first group runs 50 steps, the second 60: 110 steps. Useful work is 250 slot-steps out of \(4 \cdot 110 = 440\): \(U = 56.8\%\). In-flight: a slot that frees up at step 10 takes request 5 at once; the simulation needs 100 steps, \(U = 250/400 = 62.5\%\). With only 8 requests the final drain dominates. With 256 requests of 16 to 512 tokens and 8 slots, Figure 4 gives 14,534 static steps against 8,479: 1.71× fewer steps, and \(U\) goes from 56.8% to 97.3%.

a request generating tokensslot idle
Figure 4. 256 requests with output lengths drawn from a seeded random generator (seed 131), served with static batching (top) and in-flight batching (bottom). Both rows use the same time axis, in decode steps. Formula (131.5).
How to read Figure 4

Each row of a chart is one slot; each colored bar is one request; gaps are idle slot-steps. In the static chart, every group ends together, at its longest request, and the shorter requests leave holes. In the in-flight chart, the next request starts in the step after a slot frees, so holes appear only at the end. Choose narrow lengths (200 to 300): static batching wastes little and the gain is about 1.1×. Wide lengths give about 1.7×. More slots make the final drain a larger part of the run.

The paged KV cache and how many requests fit

The KV cache is split into fixed blocks of 32 tokens, and a request holds a list of blocks, as in vLLM’s PagedAttention (session 134). A request wastes at most one partly filled block, instead of a whole reserved max_seq_len. At start-up, after the model is loaded and a test forward pass has measured activation memory, the runtime gives 90% of the remaining free memory to the block pool. [Certain] So:

\[ N_{\text{tok}} = \frac{f \,(M_{\text{GPU}} - M_{W} - M_{\text{act}})}{s_{\text{tok}}}, \qquad n_{\text{paged}} = \left\lfloor \frac{N_{\text{tok}}}{b \lceil \ell / b \rceil} \right\rfloor, \qquad n_{\text{contig}} = \left\lfloor \frac{N_{\text{tok}}}{\ell_{\max}} \right\rfloor \tag{131.6} \]
Read the symbols
\(f\)
free_gpu_memory_fraction, 0.9 by default
\(M_{\text{GPU}}, M_W, M_{\text{act}}\)
GPU memory, weight bytes, and activations plus workspace (we assume 4 GB) [Likely]
\(s_{\text{tok}}\)
KV bytes per token: \(2 \cdot n_{\text{layers}} \cdot n_{\text{kv}} \cdot d_{\text{head}} \cdot \text{bytes}\); for Llama 3.1 8B in BF16, \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes (128 KiB)
\(b, \ell, \ell_{\max}\)
block size (32), the current length of a request, and the maximum length a contiguous cache must reserve
Worked example: Llama 3.1 8B on an 80 GB H100

BF16 weights and BF16 KV: \(0.9 \cdot (80 - 16.06 - 4) = 53.95\) GB of KV cache, which is 411,571 tokens. At an average length of 2,000 tokens a request holds \(\lceil 2000/32 \rceil = 63\) blocks (2,016 tokens), so 204 requests fit. A contiguous cache that reserves 8,192 tokens per request fits only 50. With FP8 weights and an FP8 KV cache: 60.23 GB, 918,989 tokens, 455 requests paged. These are memory limits; the compute of each step also grows with the batch.

weightsactivations and workspace (4 GB)KV block poolleft free (10%)
Figure 5. Memory plan of Llama 3.1 8B in TensorRT-LLM from formula (131.6): GPU memory, weights, a 4 GB activation reserve (our assumption), and the KV block pool with the default fraction 0.9. GPU sizes are nominal.
How to read Figure 5

The bar is the GPU memory. The readout gives the KV pool in GB and tokens, and how many requests of the chosen average length fit with 32-token pages versus a contiguous reserve of \(\ell_{\max}\) per request. Switch to FP8 weights: the weights shrink by 6.98 GB and the pool grows by 90% of that, 6.28 GB. Switch to an FP8 KV cache: the pool holds twice as many tokens. On an L40S with BF16 everything, only 95 requests of 2,000 tokens fit.

Quantization and speculative decoding

Quantized models come as checkpoints made by ModelOpt, either your own (its llm_ptq example) or pre-quantized ones that NVIDIA publishes on Hugging Face, such as nvidia/Llama-3.1-8B-Instruct-FP8. The recipes in the current docs include FP8 per-tensor, FP8 block scaling, FP8 rowwise, NVFP4, W4A16 and W4A8 with AWQ or GPTQ, and FP8 or NVFP4 KV caches, each with its own GPU support. [Certain] The methods themselves are explained in session 107 and session 110.

For speculative decoding (session 128, session 129), the LLM API takes a speculative_config. The current docs list draft/target models, EAGLE-3, MTP (for models like DeepSeek that have multi-token prediction modules), PARD, DFlash, a model-free suffix automaton method, and n-gram drafting (deprecated). [Certain] Medusa, ReDrafter and EAGLE 1 and 2 belonged to the old engine path and are not supported in the PyTorch backend. [Likely] If a draft method proposes \(k\) tokens and each is accepted with probability \(\alpha\), one target step gives on average

\[ \E[\text{tokens per target step}] = \frac{1 - \alpha^{k+1}}{1 - \alpha} \tag{131.7} \]

For \(\alpha = 0.7\) and \(k = 3\) that is 2.53 tokens per step, so decode at small batch gets up to about 2.5× fewer target passes, minus the cost of drafting. At large batch the GPU is already busy, and the gain shrinks or turns negative. [Likely]

PyTorch code

You cannot run TensorRT without an NVIDIA GPU, so the first program reproduces the ideas of this session in plain PyTorch on CPU: (1) a fused and an unfused residual add + RMSNorm + FP8 quantize give the same bits but move very different bytes; (2) the toy tactic model of Figure 3; (3) in-flight batching with a paged block pool and a no-eviction admission rule. It is deterministic.

import math, torch
torch.manual_seed(0)

# 1. Fusion: residual add + RMSNorm + FP8 quantize, unfused vs fused
T, d = 64, 4096
attn = torch.randn(T, d, dtype=torch.bfloat16); x = torch.randn(T, d, dtype=torch.bfloat16)
w = torch.ones(d, dtype=torch.bfloat16); eps = 1e-6
def rms(h): hf = h.float(); return (hf * torch.rsqrt(hf.pow(2).mean(-1, keepdim=True) + eps)).to(h.dtype) * w
def quant_fp8(n):                                  # per-tensor scale, E4M3 max is 448
    s = n.float().abs().amax() / 448.0
    return (n.float() / s).to(torch.float8_e4m3fn), s
def unfused(attn, x):                              # 3 kernels, 3 round trips to HBM
    h = x + attn                                   # read 2, write 1
    n = rms(h)                                     # read 1, write 1
    q, s = quant_fp8(n)                            # read 1, write 1 (1 byte)
    return h, q, s
def fused(attn, x):                                # one kernel: same math, values stay in registers
    h = (x.float() + attn.float()).to(torch.bfloat16)
    return h, *quant_fp8(rms(h))
h1, q1, s1 = unfused(attn, x); h2, q2, s2 = fused(attn, x)
print("same result:", torch.equal(h1, h2), torch.equal(q1.view(torch.uint8), q2.view(torch.uint8)), torch.equal(s1, s2))
Td = T * d
print("HBM bytes unfused:", (4+2) * Td + (2+2) * Td + (2+1) * Td, " fused:", 4 * Td + 2 * Td + 1 * Td)

# 2. Toy tactic selection: tune at opt, run at M
N = K = 4096; F, BW, t0 = 989e12, 0.8 * 3.35e12, 2e-6
EFF = {16: .25, 32: .40, 64: .60, 128: .75, 256: .80}       # our assumption, not measured
def t(tm, M):
    Mp = math.ceil(M / tm) * tm
    return t0 + max(2 * Mp * N * K / (F * EFF[tm]), (2*N*K + 2*M*K + 2*M*N) / BW)
pick = lambda M: min(EFF, key=lambda tm: (t(tm, M), tm))
for opt in (8, 1024):
    tm = pick(opt)
    print(f"opt={opt}: tile {tm}; at M=4096 {t(tm, 4096)*1e6:.1f} us vs best {t(pick(4096), 4096)*1e6:.1f} us")

# 3. In-flight batching with a paged KV pool vs static batching
g = torch.Generator().manual_seed(131)
R, B, BLK = 64, 8, 32
prompt = torch.randint(32, 257, (R,), generator=g).tolist()
out = torch.randint(16, 513, (R,), generator=g).tolist()
def static_iters():
    return sum(max(out[i:i+B]) for i in range(0, R, B))
def ifb_iters(pool_blocks):
    queue, active, it, peak = list(range(R)), [], 0, 0
    free = pool_blocks
    need = lambda r: math.ceil((prompt[r] + out[r]) / BLK)      # reserve for the whole request
    while queue or active:
        while queue and len(active) < B and need(queue[0]) <= free:   # admit only if it can finish
            r = queue.pop(0); free -= need(r); active.append([r, out[r]])
        for a in active: a[1] -= 1                                   # one token each this step
        it += 1; peak = max(peak, pool_blocks - free)
        for a in [a for a in active if a[1] == 0]: free += need(a[0]); active.remove(a)
    return it, peak
print("static iterations:", static_iters(), " tokens:", sum(out))
for pool in (400, 200, 100):
    it, peak = ifb_iters(pool)
    print(f"IFB, pool {pool} blocks: {it} iterations, peak {peak} blocks in use")

Output with PyTorch 2.8 on CPU (the same every run):

same result: True True True
HBM bytes unfused: 3407872  fused: 1835008
opt=8: tile 16; at M=4096 557.9 us vs best 175.7 us
opt=1024: tile 256; at M=4096 175.7 us vs best 175.7 us
static iterations: 3652  tokens: 19079
IFB, pool 400 blocks: 2638 iterations, peak 158 blocks in use
IFB, pool 200 blocks: 2638 iterations, peak 158 blocks in use
IFB, pool 100 blocks: 3610 iterations, peak 100 blocks in use

Read the last three lines: with 400 or 200 blocks, the 8 slots are the limit and in-flight batching needs 28% fewer steps than static batching. With 100 blocks, the no-eviction rule cannot admit enough requests and almost all of the gain is lost. That is the reason FP8 KV caches and smaller blocks matter for throughput.

The real tools, for an NVIDIA GPU (not run here; check the flags against your installed versions):

# TensorRT: ONNX -> engine with one optimization profile, then benchmark it
trtexec --onnx=vit_encoder.onnx --saveEngine=vit_encoder.plan \
        --minShapes=pixels:1x3x224x224 --optShapes=pixels:16x3x224x224 --maxShapes=pixels:64x3x224x224 \
        --timingCacheFile=timing.cache
trtexec --loadEngine=vit_encoder.plan --shapes=pixels:16x3x224x224

# TensorRT-LLM (PyTorch backend): load a Hugging Face or ModelOpt checkpoint, no build step
from tensorrt_llm import LLM, SamplingParams
from tensorrt_llm.llmapi import KvCacheConfig
llm = LLM(model="nvidia/Llama-3.1-8B-Instruct-FP8",
          kv_cache_config=KvCacheConfig(free_gpu_memory_fraction=0.85, enable_block_reuse=True),
          max_batch_size=256, max_num_tokens=8192)
for o in llm.generate(["The capital of France is"], SamplingParams(max_tokens=32, temperature=0.0)):
    print(o.outputs[0].text)
# or as an HTTP server:  trtllm-serve nvidia/Llama-3.1-8B-Instruct-FP8

When TensorRT-LLM beats vLLM, and when it is not worth it

Both are open source, both are Python on top of PyTorch with custom kernels, and both have the main features: paged KV cache, continuous batching, chunked prefill, prefix caching, FP8, speculative decoding, tensor and pipeline parallelism, and disaggregated prefill and decode. So the choice is not “fast versus slow” in general. It depends on your hardware, your model and your team. We give no throughput numbers here: published comparisons depend on versions, models, GPUs and settings, and change with every release. Benchmark your own workload (session 142).

SituationLean towardsWhy
Large fleet of new NVIDIA GPUs (Hopper, Blackwell), a few stable models, cost per token is the main metricTensorRT-LLMNVIDIA’s kernels and formats (NVFP4 on Blackwell, FP8 block scaling) usually arrive here first; a few percent of throughput is worth engineering time at this scale [Likely]
Serving with NVIDIA’s stack (Dynamo, NIM, Triton Inference Server)TensorRT-LLMfirst-class integration [Likely]
AMD, TPU, CPU or mixed hardwarevLLMTensorRT-LLM runs only on NVIDIA GPUs [Certain]
Many models, new models every week, research or prototypesvLLMbroader model coverage and community, faster support of new architectures [Likely]
Small traffic, one GPU, latency already meets the SLOwhatever you run nowthe gain cannot pay for migration and testing
A VLM’s vision encoder, a detector, an embedding model, Jetson devicesplain TensorRTfixed shapes, classic fusion and INT8/FP8 engines are a perfect fit
Algorithm 3: deciding whether to move a service to TensorRT-LLM
  1. input: model, GPU type, traffic trace (prompt and output lengths, arrival rate), latency SLOs (time to first token, time per output token)
  2. Check the support matrix: is the model, the quantization recipe and the speculative method supported on your GPU? If not, stop
  3. Replay the same trace on both engines, same precision, same parallelism, same GPU count // never compare FP8 to BF16
  4. Tune both: batch limits, max tokens per step, chunked prefill, CUDA graph sizes, KV fraction
  5. Measure goodput: requests per second that meet both SLOs
  6. Gain \(\leftarrow\) GPU-hours saved per month at your traffic; Cost \(\leftarrow\) engineering time, testing, on-call, upgrade cadence
  7. output: switch only if Gain is clearly larger than Cost, and plan to re-test at each upgrade of either engine

Where this is used

ApplicationToolWhat matters most
High-volume chat and API serving on H100, H200 or B200TensorRT-LLMFP8 or NVFP4 checkpoints, in-flight batching, FP8 KV cache, speculative decoding (EAGLE-3, MTP)
VLM serving: image or video question answeringTensorRT engine for the vision encoder, an LLM engine for the decoderfixed-shape encoder profiles (batch of frames), prefix caching of the system prompt, many image tokens in prefill
Video retrieval and Video-RAG indexingTensorRT (embedding and frame encoders)large static batches, FP8 or INT8 engines, one optimization profile per frame batch size
Action recognition and key-moment detection on live streamsTensorRT on server GPUs or Jetsonlow latency at small batch, fusion, engines built on the device type
Video fingerprinting and near-duplicate detectionTensorRTthroughput per GPU, INT8 accuracy checks against the FP32 model
Video segmentation (per-frame masks)TensorRTdynamic resolution needs well-chosen profiles; otherwise rebuild per resolution
Agents and long-context RAGTensorRT-LLM or vLLMKV block reuse for shared prefixes, chunked prefill so long prompts do not stall decode

Common mistakes

  • Copying an engine to another GPU type or TensorRT version. It fails to load (or, with compatibility modes, runs slower). Build per GPU type in CI and store the version with the engine.
  • One optimization profile with a decode-size opt for all shapes. Large inputs then run small-tile kernels, 3.2× slower in our model. Use the expected shape as opt, or several profiles.
  • A huge \([\text{min}, \text{max}]\) range “to be safe”. Memory for activations can grow with max, and some fast tactics need \(\text{min} = \text{opt} = \text{max}\). Keep ranges as tight as the traffic allows.
  • Following a 2024 tutorial. trtllm-build, convert_checkpoint.py and backend="tensorrt" are gone; INT8 calibrators and the FP16 builder flag are gone from TensorRT 11. Check the version of every guide you use.
  • Comparing engines at different precision. FP8 TensorRT-LLM against BF16 vLLM measures the format, not the engine.
  • Quantizing without an accuracy check. FP8 is usually safe for LLM weights and activations; NVFP4 and INT4 need an evaluation on your task. Engines can also differ slightly from PyTorch because of other kernels and fusion order.
  • Setting free_gpu_memory_fraction without leaving room. Another process, a LoRA adapter cache or larger CUDA graphs can then run out of memory. Measure peak memory under real traffic.
  • Loading engines from strangers. An engine is executable code for the runtime; the docs warn to deserialize only trusted engines.

Interview questions

What does TensorRT do to a model when it builds an engine?basic
It parses the graph (usually ONNX) into a network, folds constants and removes dead layers, fuses chains of layers into single kernels (for example MatMul with bias and activation in the epilogue, or element-wise chains), fixes the precision of each tensor from the types and Q/DQ nodes, then times candidate kernels (tactics) for each layer on the actual GPU at the optimization-profile opt shape and keeps the fastest. It plans memory so buffers are reused, and serializes the chosen kernels, weights and plan into an engine file that the runtime loads and executes.
Why does fusion help, and when does it help most?core
Without fusion, each operator writes its output to HBM and the next one reads it back. A fused kernel keeps intermediate values in registers or shared memory, so it saves those round trips and also saves kernel launches. For large activations (prefill), the saving is in bytes: for residual add + RMSNorm + FP8 quantize at 4,096 tokens, traffic goes from 251.7 MB to 117.4 MB and time from about 100 to 46 µs in our model. At decode with one token the bytes are tiny, and the saving is in kernels: 6 µs to 2 µs, which also reduces launch gaps.
What is an optimization profile, and what happens if you choose opt badly?core
For every input with a dynamic dimension (marked \(-1\)) you give min, opt and max shapes. The engine accepts any shape in \([\text{min}, \text{max}]\); the auto-tuner times tactics at opt. If opt is far from your real shapes, the chosen kernels are tuned for the wrong regime: in our GEMM model, a profile tuned at 8 tokens runs a 4,096-token batch 3.2× slower than the best kernel. Shapes outside \([\text{min}, \text{max}]\) are rejected. You can add several profiles (for example one for small and one for large batches) and select the one that covers the input; each execution context uses its own profile.
Your colleague built an engine on an H100 and it fails to load on an L40S. Why, and what are the options?basic
By default an engine works only on GPUs with the same compute capability (H100 is 9.0, L40S is 8.9), with the same TensorRT version, and on the same OS and CPU architecture, because it contains kernels compiled and timed for that GPU. Options: build a separate engine on an L40S (best performance), or build with a hardware compatibility level such as AMPERE_PLUS, which runs on any GPU of compute capability 8.0 or newer but can be slower. For TensorRT version changes, a version-compatible build embeds a lean runtime, also at some speed cost.
How did precision control change in TensorRT 11?advanced
Networks are strongly typed by default: each tensor has the type the model gives, and the builder no longer picks lower precisions on its own. The weak-typing APIs (per-precision builder flags, setPrecision, setOutputType), the INT8 calibrator and implicit quantization were removed. To run INT8, FP8 or FP4, you put explicit QuantizeLinear and DequantizeLinear nodes with scales into the graph, typically with NVIDIA ModelOpt, and the builder fuses DQ → MatMul → Q into a low-precision kernel. The advantage is that precision is decided once, explicitly, and is reproducible.
What is in-flight batching, and how much does it gain?core
It is TensorRT-LLM’s name for continuous (iteration-level) batching: at every decode step, finished requests leave and waiting requests join, instead of a batch running until its longest request ends. The gain depends on the spread of output lengths. Static batching wastes the slot-steps between each request’s end and its group’s longest request. In our simulation of 256 requests with 8 slots, lengths 16 to 512 need 1.71× fewer steps (utilization 56.8% to 97.3%), but narrow lengths 200 to 300 gain only 1.14×.
How many 2,000-token requests of Llama 3.1 8B fit in the KV cache of an 80 GB H100 with TensorRT-LLM defaults?core
KV per token is \(2 \times 32\ \text{layers} \times 8\ \text{KV heads} \times 128 \times 2\ \text{bytes} = 128\ \text{KiB}\) in BF16. With 16.06 GB of BF16 weights, a 4 GB activation reserve and the default free_gpu_memory_fraction of 0.9, the pool is \(0.9 \times (80 - 16.06 - 4) = 53.95\) GB, about 411,600 tokens. With 32-token blocks a 2,000-token request uses 63 blocks, so about 204 requests fit; reserving 8,192 tokens each would fit only 50. FP8 weights and an FP8 KV cache raise it to about 455.
Does TensorRT-LLM still use TensorRT?advanced
Not for running LLMs, as of late 2026. Before 1.0 it built TensorRT engines with trtllm-build and used plugins for attention and GEMMs. Version 1.0 (September 2025) made the PyTorch-based runtime the default, and from 1.3.0rc21 (July 2026) the TensorRT backend was removed: no engine build, Hugging Face or ModelOpt checkpoints load directly. The performance now comes from custom CUDA kernels, fusions, CUDA graphs for decode, the overlap scheduler and quantized formats. TensorRT itself remains the tool for non-LLM networks.
What does the GUARANTEED_NO_EVICT capacity policy trade off?advanced
It admits a new request only if the scheduler can guarantee that the request will finish without being paused or evicted, so it reserves KV capacity conservatively. This gives stable latency and no recomputation, but fewer concurrent requests when the pool is small. MAX_UTILIZATION packs more requests and may have to pause some when the cache fills. Our code shows the effect of conservative admission: with a pool of 100 blocks, in-flight batching needed 3,610 steps instead of 2,638, almost the static 3,652. [Likely for the exact semantics; check the docs of your version]
When would you not adopt TensorRT-LLM even if it is faster in a benchmark?core
When the hardware is not NVIDIA or is mixed; when models change often and support for new architectures lags; when traffic is small, so saved GPU-hours cannot pay for migration, testing and on-call; when the latency SLO is already met; when the team knows another engine well; or when the benchmark was not like-for-like (different precision, batch limits, or traffic shape). The decision should rest on goodput for your own trace under your SLO, converted to cost.
How does speculative decoding interact with batch size in a server like TensorRT-LLM?advanced
Speculation turns several sequential decode steps into one verification pass over \(k\) draft tokens. At small batch, decode is memory-bound, so verifying \(k+1\) tokens costs about the same as one, and you gain up to \(\frac{1 - \alpha^{k+1}}{1 - \alpha}\) tokens per step, 2.53 for \(\alpha = 0.7\), \(k = 3\). At large batch the step is closer to compute-bound and the extra verified tokens cost real compute, while rejected drafts waste it, so the gain shrinks and can turn negative. Servers therefore tune max_draft_len, and some disable speculation at high load.

Exercises

  1. Use (131.1) with our constants for the chain of Figure 2 at \(T = 2048\): give the bytes and time without fusion and with A|B and C|D fused.
  2. Compute the KV bytes per token of Llama 3.1 70B (80 layers, 8 KV heads of size 128) in BF16 and in FP8.
  3. Two slots, four requests with output lengths 5, 100, 100, 5 in this order. Count the steps of static and of in-flight batching, and the utilization of each.
  4. An engine has one profile with min 1, opt 8, max 8,192 tokens. Prefill batches of 4,096 tokens are slow. Use Figure 3 to explain why and propose two fixes.
  5. Will an engine built on an H100 SXM load on an H100 PCIe with the same TensorRT version? On an A100? Explain with compute capabilities (9.0, 9.0, 8.0).
  6. For Llama 3.1 8B on an H100 with FP8 weights and an FP8 KV cache, how many 2,000-token requests fit with 32-token pages, and with a contiguous reserve of 8,192 tokens?
Answerscheck
1. \(Nb = 2048 \cdot 4096 \cdot 2 = 16.8\) MB. Unfused: \(7.5 \times 16.8 = 125.8\) MB, \(t = 3 \cdot 2 + 125.8/2.68 = 53.0\ \mu s\). A|B and C|D fused: \(3.5 \times 16.8 = 58.7\) MB, one element-wise kernel, \(t = 2 + 21.9 = 23.9\ \mu s\). 2. \(2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327{,}680\) bytes (320 KiB) in BF16; half, 160 KiB, in FP8. 3. Static: \(\max(5,100) + \max(100,5) = 200\) steps, \(U = 210/400 = 52.5\%\). In-flight: request 3 starts at step 6 and ends at step 105; request 4 starts at step 101 and ends at 105: 105 steps, \(U = 210/210 = 100\%\). 4. The tactics were timed at 8 tokens, where every tile is memory-bound and the smallest tile wins; at 4,096 tokens the small tile reaches a low fraction of peak (557.9 versus 175.7 µs in the model). Fix: set opt near the common prefill size, or add a second profile for large inputs and select it for prefill. 5. H100 PCIe: same compute capability 9.0, so it loads, but the tactics were timed on another GPU (different clocks and bandwidth) and may not be the fastest; the docs note a possible small performance drop and no guarantee. A100: compute capability 8.0 differs, so it fails to deserialize unless built with a hardware compatibility level such as AMPERE_PLUS. 6. Pool \(0.9 \cdot (80 - 9.08 - 4) = 60.23\) GB at 64 KiB per token \(= 918{,}989\) tokens; 455 requests with pages (2,016 tokens each), 112 with a contiguous 8,192-token reserve.

Key takeaways

  • TensorRT compiles a graph once: fold, fuse, fix precision, time tactics on the real GPU, plan memory, serialize. The engine is fast and specific to one GPU type, one TensorRT version and the given shape ranges.
  • Fusion saves HBM round trips (131.1): 2.2× less time for add + RMSNorm + FP8 quantize in prefill, and fewer kernels in decode.
  • Optimization profiles give \([\text{min}, \text{max}]\) and the opt shape used for tuning (131.2, 131.3). Tune for the shapes you serve, or use several profiles.
  • TensorRT 11 is strongly typed; quantize with explicit Q/DQ nodes (131.4). FP8 needs Ada or Hopper, NVFP4 needs Blackwell.
  • TensorRT-LLM today is a PyTorch runtime: in-flight batching (131.5), 32-token KV blocks with reuse and 90% of free memory (131.6), CUDA graphs and an overlap scheduler for decode, ModelOpt checkpoints, and speculative decoding (131.7). Its TensorRT engine path was removed in 2026.
  • Choose it over vLLM for large NVIDIA fleets and stable models after a like-for-like goodput test; it is not worth it for non-NVIDIA hardware, fast-changing models or small traffic.

Key formulas

\[ t \approx n_{\text{kern}} t_0 + \frac{Q}{\eta_B \mathrm{BW}}, \qquad \tau^{*} = \argmin_\tau \hat t(\tau; s_{\text{opt}}), \qquad s = \frac{\max|x|}{q_{\max}} \]
\[ U_{\text{static}} = \frac{\sum_i L_i}{B \sum_g \max_{i\in g} L_i}, \qquad N_{\text{tok}} = \frac{f (M_{\text{GPU}} - M_W - M_{\text{act}})}{s_{\text{tok}}}, \qquad \E[\text{tokens}] = \frac{1-\alpha^{k+1}}{1-\alpha} \]

References for this session

  1. NVIDIA (docs). TensorRT documentation
  2. NVIDIA (code). TensorRT-LLM
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  4. NVIDIA (docs). TensorRT documentation
  5. NVIDIA (code). TensorRT-LLM
  6. NVIDIA (docs). TensorRT documentation: Engine Compatibility (version, hardware and platform compatibility)
  7. NVIDIA (docs). TensorRT documentation: Dynamic Shapes, Core Concepts (optimization profiles)
  8. NVIDIA (docs). TensorRT 11.0.0 Release Notes (strongly typed networks by default, implicit quantization removed)
  9. NVIDIA (docs). TensorRT LLM: Migration Guide, TensorRT Backend Removed
  10. NVIDIA (docs). TensorRT LLM: Quantization (recipes and hardware support matrix)
  11. NVIDIA (docs). TensorRT LLM: Speculative Decoding
  12. NVIDIA (code). TensorRT LLM v1.4.0rc3: tensorrt_llm/llmapi/llm_args.py (KvCacheConfig, SchedulerConfig, CudaGraphConfig defaults)
  13. vLLM project (docs). vLLM documentation
  14. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  15. Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
  16. Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  17. Micikevicius et al. (2022). FP8 Formats for Deep Learning
  18. Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
  19. Li et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
  20. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models