GPU kernels 2: write kernels in the Triton language
Triton is a Python-like language for writing GPU kernels. You describe what one program does with a whole block of data, and the compiler handles threads, memory moves and synchronization. In this session you write a vector add, a fused softmax and an RMSNorm kernel, autotune their block sizes, compare them with PyTorch eager and torch.compile, and learn not to confuse the Triton language with NVIDIA's Triton Inference Server.
What you will learn
- The Triton programming model: a grid of programs, each working on a block of data with tl.arange, masks and pointer arithmetic
- Why Triton is easier than CUDA: what the compiler does for you, and what you still decide
- How to count the bytes an eager softmax moves (8MN + 4M elements) and why one fused kernel moves only 2MN
- How to write a fused softmax and an RMSNorm kernel, and what to do when a row does not fit in one block (online softmax)
- What BLOCK_SIZE, num_warps and num_stages control, and how triton.autotune picks them
- How to benchmark a kernel fairly against PyTorch eager and torch.compile, and how to judge it with effective bandwidth
- The name trap: the Triton language (a kernel compiler) is not the Triton Inference Server (a serving system)
Why this matters
In session 85 you saw that most operations in an LLM, apart from the big matrix multiplications, are limited by memory traffic, not by math. The fix is fusion: do several steps in one kernel, so the data travels from GPU memory (HBM) to the chip only once. But writing fused kernels in CUDA C++ is slow work: you manage threads, shared memory, synchronization and memory alignment by hand. Triton is a language that removes most of that work.
Triton is a Python-like language and compiler for GPU kernels. You write a Python function, decorate it with @triton.jit, and Triton compiles it to GPU machine code. It was introduced by Tillet, Kung and Cox in 2019 and is now developed by OpenAI and a community of contributors. [Certain] It runs on NVIDIA GPUs and also on AMD GPUs. [Certain] It matters to you for three reasons:
- torch.compile writes Triton. When you call
torch.compileon a GPU model, its backend (TorchInductor) generates Triton kernels for the fused element-wise and reduction parts (session 50). [Certain] To read what the compiler made, you must read Triton. - Real systems ship hand-written Triton kernels. Liger Kernel is a set of Triton kernels for LLM training (RMSNorm, RoPE, SwiGLU, cross-entropy). Its paper reports about 20% more training throughput and about 60% less GPU memory than the Hugging Face versions, on average. [Certain, numbers from the Liger paper] vLLM and other serving engines use Triton kernels too, for example for mixture-of-experts layers. [Likely]
- Interviews ask for it. “Write a fused softmax in Triton”, “Why is the fused version faster?”, “What is BLOCK_SIZE?”, “Is Triton the same as Triton Inference Server?” are common questions for ML systems roles.
The plan: first the programming model (blocks, not threads). Then a first kernel (vector add). Then two real LLM kernels: fused softmax and RMSNorm. Then autotuning, fair benchmarks against PyTorch eager and torch.compile, and the name trap at the end.
The programming model: blocks of data, not single threads
In CUDA you write the code for one thread. Each thread handles one element (or a few), and you think about warps, shared memory and barriers yourself. In Triton you write the code for one program, and a program handles a whole block of elements at once. A program is roughly one CUDA thread block. [Certain] Inside it, Triton decides how to spread the block over threads and warps.
Three ideas are enough to start:
- The grid. You launch many copies of the same program. Each copy asks
tl.program_id(0)for its number, calledpid. For \(n\) elements and a block of size \(B\), you need
programs. With \(n = 1{,}000\) and \(B = 256\), that is \(G = 4\) programs, and they cover \(4 \times 256 = 1{,}024\) slots, 24 more than needed.
- Offsets and masks. Each program builds a vector of indices with
tl.arange(0, B)and shifts it to its own part of the data. Slots past the end are switched off with a mask:
- Read the symbols
- \(\mathrm{pid}\)
- the number of this program, from 0 to \(G - 1\)
- \(B\)
- the block size; in Triton it must be a power of two and known at compile time (
tl.constexpr) - \(\mathbf{o}\)
- a vector of \(B\) element offsets: which elements this program touches
- \(\mathbf{m}\)
- a vector of \(B\) true/false values: true where the offset is inside the data
- Pointers and block loads. A tensor argument arrives as a pointer to its first element.
x_ptr + offsis a block of \(B\) pointers.tl.load(x_ptr + offs, mask=mask, other=0.0)reads all of them at once and putsotherwhere the mask is false.tl.storewrites a block back. Math such as+,tl.exp,tl.maxandtl.sumworks on whole blocks, like NumPy.
Figure 1 shows the grid, the offsets and the mask. Click any slot to see which program owns it and whether it is masked.
How to read Figure 1
Change \(n\) and watch the number of rows: it is always \(\lceil n/B \rceil\) (formula 86.1). The grey slots at the end are the price of a fixed, power-of-two block: they exist in the program but the mask turns their loads and stores off. Bigger blocks mean fewer programs but more waste in the last one. For a real vector of a million elements, the waste is tiny; for a softmax row of 5,000 columns padded to 8,192, it is not (see Figure 3).
Your first kernel: vector add
Here is the whole idea in pseudo-code. Each program adds one block.
- input: pointers
x_ptr,y_ptr,out_ptr, length \(n\), compile-time block size \(B\) - \(\mathrm{pid} \leftarrow\)
tl.program_id(0)// which block am I? - \(\mathbf{o} \leftarrow \mathrm{pid} \cdot B +\)
tl.arange(0, B); \(\;\mathbf{m} \leftarrow \mathbf{o} \lt n\) // formula (86.2) - \(\mathbf{x} \leftarrow\) load(
x_ptr\(+ \mathbf{o}\), mask \(\mathbf{m}\)); \(\;\mathbf{y} \leftarrow\) load(y_ptr\(+ \mathbf{o}\), mask \(\mathbf{m}\)) // one block read each - store(
out_ptr\(+ \mathbf{o}\), \(\mathbf{x} + \mathbf{y}\), mask \(\mathbf{m}\)) - host: launch the kernel on a grid of \(\lceil n/B \rceil\) programs (formula 86.1)
In Triton this is almost line by line the same:
import torch, triton import triton.language as tl @triton.jit def add_kernel(x_ptr, y_ptr, out_ptr, n, BLOCK: tl.constexpr): pid = tl.program_id(0) # which program am I? offs = pid * BLOCK + tl.arange(0, BLOCK) # formula (86.2) mask = offs < n x = tl.load(x_ptr + offs, mask=mask) y = tl.load(y_ptr + offs, mask=mask) tl.store(out_ptr + offs, x + y, mask=mask) def add(x, y): out = torch.empty_like(x) n = x.numel() grid = (triton.cdiv(n, 1024),) # formula (86.1) add_kernel[grid](x, y, out, n, BLOCK=1024) # tensors become pointers return out
Three details to notice. First, add_kernel[grid](...) is the launch: the square brackets hold the grid, the round ones the arguments. Second, BLOCK is a tl.constexpr: Triton compiles a separate version of the kernel for each value, so the shapes of all blocks are known when the code is compiled. Third, there is no thread index anywhere. You never wrote threadIdx.x.
Add two FP32 vectors of \(n = 2^{20} = 1{,}048{,}576\) elements. The kernel reads two vectors and writes one: \(3 \times 4 \times 2^{20} = 12.6\) MB. An H100 SXM reads about 3.35 TB/s from HBM. [Certain, NVIDIA datasheet] So the kernel cannot take less than \(12.6\ \text{MB} / 3.35\ \text{TB/s} \approx 3.8\ \mu s\). It does only one addition per 12 bytes, so the math units are nearly idle: this kernel is purely memory-bound. A good Triton vector add should come close to this limit on large inputs; on tiny inputs, the launch overhead (a few microseconds) dominates. [Likely]
What the compiler does for you, and what you still decide
The Triton paper calls its unit of work a tile: a block of data that a program loads, computes on, and stores. [Certain] Because the compiler sees whole tiles, not single threads, it can make many low-level choices that a CUDA programmer makes by hand.
| Job | In CUDA C++ | In Triton |
|---|---|---|
| Split work over threads and warps | You, with threadIdx, blockIdx | Compiler; you only choose num_warps |
| Coalesced memory access (neighbouring threads read neighbouring addresses) | You arrange the indexing | Compiler, from the block layout |
Shared memory and barriers (__syncthreads) | You allocate and synchronize | Compiler, for reductions and for tl.dot |
| Tensor cores for matrix multiply | You use special instructions or libraries | Compiler, when you call tl.dot on blocks |
| Software pipelining (load the next tile while computing this one) | You write double buffering | Compiler; you choose num_stages |
| Which data each program handles; block sizes; what to fuse | You | Still you |
So Triton does not make design free. You still choose the grid, the block sizes and the order of loads, and you still have to count bytes. What disappears is the error-prone bookkeeping. The price: a little less control than CUDA for the very last few percent of speed. The fastest attention kernels on the newest GPUs (FlashAttention-3) are still written in CUDA. [Certain]
Fused softmax: count the bytes first
Softmax over each row of an \(M \times N\) matrix (for example, attention scores, or logits over a vocabulary) is, with the max subtracted for numerical stability (session 2):
Write it in PyTorch the plain way, one operation per line, and each line becomes one kernel that reads its inputs from HBM and writes its output back:
x_max = x.max(dim=1)[0] # read MN, write M z = x - x_max[:, None] # read MN + M, write MN num = torch.exp(z) # read MN, write MN den = num.sum(dim=1) # read MN, write M out = num / den[:, None] # read MN + M, write MN
Add the columns. The eager version reads \(5MN + 2M\) elements and writes \(3MN + 2M\). [Certain, the same count is in the Triton tutorial] A fused kernel loads each row once, keeps it on the chip (in registers), does all five steps there, and writes the result once:
For large \(N\), the ratio is about \(8MN / 2MN = 4\). When a kernel is memory-bound, its time is at least its bytes divided by the bandwidth:
So a fused softmax can be up to about 4 times faster, and that is what the Triton tutorial measures against the naive version. [Certain, from the tutorial; the exact speed depends on the GPU] It also wins a little against torch.softmax, which is already a single fused kernel but is written for any shape and layout. [Certain, as stated in the tutorial]
\(M = 8{,}192\), \(N = 4{,}096\), 4 bytes per element. Eager: \((8MN + 4M) \times 4 = 1{,}073.9\) MB. Fused: \(2MN \times 4 = 268.4\) MB. Ratio: 4.0. At 3.35 TB/s (formula 86.5), the lower bounds are \(320.6\ \mu s\) for eager and \(80.1\ \mu s\) for fused. The eager version also launches 5 kernels instead of 1.
Figure 2 walks through the five eager kernels and then the fused one, with the bytes each moves.
How to read Figure 2
Step through the eager kernels: each one reads a full \(M \times N\) matrix, and three of them also write one. The intermediate tensors z and num exist only to be read again by the next kernel. Step 6 is the fused kernel: one read, one write, and the intermediates never leave the chip. Change \(N\): the ratio stays close to 4 because the \(4M\) term is small next to \(8MN\). BF16 halves every number but not the ratio.
The fused softmax kernel
One program handles one row. This works when a full row fits in one block (up to some tens of thousands of columns; see Figure 3).
- input: pointers to input and output, row stride, \(N\) columns, block size \(B \ge N\) (a power of two)
- \(r \leftarrow\)
tl.program_id(0)// the row this program owns - \(\mathbf{o} \leftarrow (0, \dots, B - 1)\); \(\;\mathbf{m} \leftarrow \mathbf{o} \lt N\)
- \(\mathbf{x} \leftarrow\) load(row \(r\) at \(\mathbf{o}\), mask \(\mathbf{m}\), other \(= -\infty\)) // masked slots give \(e^{-\infty} = 0\)
- \(m \leftarrow \max(\mathbf{x})\); \(\;\mathbf{e} \leftarrow \exp(\mathbf{x} - m)\); \(\;s \leftarrow \sum \mathbf{e}\) // all in registers
- store(row \(r\) at \(\mathbf{o}\), \(\mathbf{e} / s\), mask \(\mathbf{m}\)) // the only write
- host: \(B \leftarrow\) next power of two \(\ge N\); launch \(M\) programs
Look at line 4. The fill value for masked slots must not change the answer. For a max and a sum of exponentials, \(-\infty\) is right: it never wins the max and \(e^{-\infty} = 0\) adds nothing. For a plain sum, the right fill is 0. Choosing the wrong other is a classic bug (see Common mistakes).
@triton.jit def softmax_kernel(out_ptr, in_ptr, in_stride, out_stride, n_cols, BLOCK: tl.constexpr): row = tl.program_id(0) offs = tl.arange(0, BLOCK) mask = offs < n_cols x = tl.load(in_ptr + row * in_stride + offs, mask=mask, other=-float("inf")) x = x - tl.max(x, axis=0) # formula (86.3), stable num = tl.exp(x) den = tl.sum(num, axis=0) tl.store(out_ptr + row * out_stride + offs, num / den, mask=mask) def softmax(x): M, N = x.shape out = torch.empty_like(x) BLOCK = triton.next_power_of_2(N) # the whole row in one block softmax_kernel[(M,)](out, x, x.stride(0), out.stride(0), N, BLOCK=BLOCK, num_warps=8) return out
The official tutorial adds one more idea: instead of launching \(M\) programs, it launches only as many as fit on the GPU at once and lets each one loop over several rows (a “persistent” kernel). It picks that count from the kernel’s register and shared-memory use and the number of SMs. [Certain, from the tutorial]
When a row does not fit: online softmax
For a vocabulary of 128,000 logits, one block would need 131,072 slots, which is too many registers (Figure 3). Then the program must loop over the row in chunks. The problem: the max \(m\) is not known until the end. The online softmax trick keeps a running max \(m\) and a running sum \(\ell\), and rescales the sum whenever the max grows:
- Read the symbols
- \(m, \ell\)
- the max and the sum of \(e^{x - m}\) over all chunks seen so far; start at \(m = -\infty\), \(\ell = 0\)
- \(m', \ell'\)
- the same values after adding one more chunk
- \(e^{m - m'}\)
- a correction factor (at most 1) that re-expresses the old sum relative to the new max
After the last chunk, \(\ell\) equals the true denominator. A second pass writes \(e^{x_j - m} / \ell\). This costs one extra read of the row but no extra intermediate tensor. The same trick, applied to attention scores, is the heart of FlashAttention (session 124).
Chunk 1 = (1, 3): \(m = 3\), \(\ell = e^{-2} + e^{0} = 1.1353\). Chunk 2 = (2, 5): \(m' = 5\), \(\ell' = 1.1353 \cdot e^{3 - 5} + e^{2-5} + e^{0} = 1.2034\). Computing directly, \(\sum_k e^{x_k - 5} = 1.2034\): the same. The softmax is (0.0152, 0.1125, 0.0414, 0.8310).
An RMSNorm kernel
Every block of Llama-style models normalizes its input with RMSNorm (session 22). For a row \(\mathbf{x} \in \R^{N}\) with a learned gain \(\mathbf{w}\):
- Read the symbols
- \(N\)
- the hidden size (4,096 for Llama 3 8B)
- \(\varepsilon\)
- a small constant such as \(10^{-5}\), so we never divide by zero
- \(w_j\)
- the learned gain for feature \(j\), shared by all rows
The plain PyTorch line x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * w launches about six kernels (square, mean, add, rsqrt, two multiplies). Count them like the softmax, with all tensors in one element type: they read \(4MN + 3M + N\) elements and write \(3MN + 3M\). A fused kernel reads the row and \(\mathbf{w}\) once and writes once: \(2MN + N\). That is about 3.5 times less traffic.
\(M = 8{,}192\), \(N = 4{,}096\), 2 bytes. Eager: \((7MN + 6M + N) \times 2 = 469.9\) MB, at least \(140.3\ \mu s\) on an H100. Fused: \((2MN + N) \times 2 = 134.2\) MB, at least \(40.1\ \mu s\). Ratio 3.5. Real code is often worse for eager, because it also casts BF16 to FP32 and back, which adds two more full passes.
- input: pointers to \(\mathbf{x}\), \(\mathbf{w}\), output; row stride; \(N\); \(\varepsilon\); tunable block size \(B\) (can be smaller than \(N\))
- \(r \leftarrow\)
tl.program_id(0); \(\;\mathbf{a} \leftarrow\) zeros(\(B\)) in FP32 // accumulator - for start \(= 0, B, 2B, \dots \lt N\): load chunk \(\mathbf{x}_c\) (mask, other \(= 0\)), cast to FP32, \(\mathbf{a} \leftarrow \mathbf{a} + \mathbf{x}_c^2\)
- \(\rho \leftarrow 1 / \sqrt{\sum \mathbf{a} / N + \varepsilon}\) // formula (86.7); save \(\rho\) for the backward pass if training
- for start \(= 0, B, 2B, \dots \lt N\): load \(\mathbf{x}_c\) and \(\mathbf{w}_c\), store \(\mathbf{x}_c \cdot \rho \cdot \mathbf{w}_c\) cast to the output type
- host: launch \(M\) programs; let the autotuner choose \(B\) and
num_warps
The second loop reads the row again. For \(N = 4{,}096\) in BF16 a row is only 8 KB, so the second read usually hits the on-chip L2 cache rather than HBM. [Likely] If \(B \ge N\), there is one chunk, and you can keep the row in registers and skip the second load completely.
Block size, warps and autotuning
Three numbers decide how a Triton kernel runs, and you choose all three:
- BLOCK_SIZE (a
tl.constexpr): how many elements one program works on at a time. - num_warps: how many warps (groups of 32 threads) run one program. More warps means fewer elements per thread.
- num_stages: how many tiles ahead the compiler loads in loops (software pipelining). More stages hide memory latency but use more shared memory. It mostly matters for matrix multiplication loops.
A useful first estimate: in a kernel that keeps one block of FP32 values in registers, each thread holds
On an H100, an SM has 65,536 registers of 32 bits and a thread can use at most 255. [Certain, CUDA programming guide] If a thread needs more, the compiler spills values to slow local memory, and the kernel slows down a lot. If threads need many registers, fewer warps fit on an SM at once, and there are fewer warps to hide memory latency (session 42). The real register count is higher than \(e\): addresses, indices and temporaries need registers too. So formula (86.8) is a lower bound, good for spotting obviously bad choices. Figure 3 applies it.
How to read Figure 3
Start at \(N = 4{,}096\) and 4 warps: 32 values per thread, fine. Move to \(N = 32{,}768\): with 4 warps each thread would need 256 values, above the 255 limit, so it spills; with 16 warps it is 64. This is why long rows need more warps, or a chunked loop like Algorithm 3. Now try \(N = 5{,}000\): the block becomes 8,192 and 39% of the slots are masked. Because these effects interact and depend on the GPU, the practical answer is to measure, which is what the autotuner does.
triton.autotune
The autotuner is a decorator. You give it a list of triton.Config candidates (block sizes, num_warps, num_stages) and a key: a list of argument names. The first time the kernel is called with a new value of the key arguments, Triton runs every candidate, times it, and caches the fastest for that key. [Certain, Triton API docs]
@triton.autotune( configs=[triton.Config({"BLOCK": b}, num_warps=w) for b in (256, 512, 1024, 2048) for w in (2, 4, 8)], key=["n_cols"], # re-tune when the row length changes ) @triton.jit def rmsnorm_kernel(out_ptr, x_ptr, w_ptr, stride, n_cols, eps, BLOCK: tl.constexpr): row = tl.program_id(0) x_row = x_ptr + row * stride acc = tl.zeros([BLOCK], dtype=tl.float32) for start in range(0, n_cols, BLOCK): # pass 1: sum of squares offs = start + tl.arange(0, BLOCK) x = tl.load(x_row + offs, mask=offs < n_cols, other=0.0).to(tl.float32) acc += x * x rstd = tl.rsqrt(tl.sum(acc, axis=0) / n_cols + eps) # formula (86.7) for start in range(0, n_cols, BLOCK): # pass 2: normalize and scale offs = start + tl.arange(0, BLOCK) m = offs < n_cols x = tl.load(x_row + offs, mask=m, other=0.0).to(tl.float32) w = tl.load(w_ptr + offs, mask=m, other=0.0).to(tl.float32) tl.store(out_ptr + row * stride + offs, (x * rstd * w).to(out_ptr.dtype.element_ty), mask=m) def rmsnorm(x, w, eps=1e-5): M, N = x.shape out = torch.empty_like(x) rmsnorm_kernel[(M,)](out, x, w, x.stride(0), N, eps) # BLOCK comes from the chosen config return out
Three practical points. (1) The first call is slow: it compiles and times 12 configs. Never include it in a benchmark. (2) A good key lists the arguments that change the best choice (here the row length), but not ones that change every call, or the kernel re-tunes all the time. (3) If a kernel writes into one of its inputs, every trial run changes that input again; the autotuner has reset_to_zero and restore_value options for this. [Certain, Triton API docs]
Compare with PyTorch eager and torch.compile
You now have three ways to run the same RMSNorm:
| Way | What runs on the GPU | Strength | Weakness |
|---|---|---|---|
| PyTorch eager | One pre-built kernel per operation (about 6 for RMSNorm) | No compile step; easy to debug | Every intermediate goes through HBM; many launches |
torch.compile | Inductor fuses the chain and generates a Triton kernel [Certain] | One line of code; also fuses the operations around it | Compile time; graph breaks; the kernel it writes may not be the best possible |
| Hand-written Triton | Your kernel | Full control: layout, fusion with neighbours, a custom backward | You own correctness, the backward pass and the tuning |
For a simple element-wise and reduction chain like RMSNorm or softmax, torch.compile often produces a kernel close to a good hand-written one, because it generates Triton itself. [Likely] Hand-written kernels win when the fusion crosses boundaries the compiler does not cross, for example a cross-entropy fused with the final matrix multiply so the full logits tensor is never stored (one of Liger’s kernels), or attention (FlashAttention). [Likely]
How to benchmark fairly
GPU calls are asynchronous: Python returns before the kernel ends. Timing with time.time() around a call measures almost nothing. Use CUDA events or triton.testing.do_bench, which warms up, repeats the call and synchronizes. Then judge the result with the effective bandwidth: the minimum bytes the operation must move, divided by the measured time:
Use the minimum bytes (for softmax, \(2MN\) elements) for all three versions, not the bytes each one actually moved. Then the number tells you how close each version is to the best possible on this GPU. A memory-bound kernel that reaches 80–90% of the peak bandwidth is very good; that is roughly the best that well-tuned kernels reach in practice. [Likely] The softmax plot in the Triton tutorial is drawn exactly this way, in GB/s. Figure 4 is the calculator: enter a time you measured, read the efficiency, and see where the kernel sits on the roofline.
How to read Figure 4
The dot sits far to the left of the ridge (the corner of the roof): a few FLOPs per element over 4 to 12 bytes per element gives an arithmetic intensity of about 1 FLOP per byte or less, while the H100 ridge is at about 295 FLOP/byte. So these kernels can only be limited by bandwidth, and the right score is GB/s, not FLOP/s. Slide the time down: the readout refuses times faster than the bandwidth allows, because no kernel can beat formula (86.5). If a time you measured seems to do so, you probably forgot to synchronize.
The name trap: Triton is two different things
Two well-known projects share the name, and interviewers like to check that you know the difference.
| Triton (the language) | Triton Inference Server | |
|---|---|---|
| What it is | A language and compiler for writing GPU kernels | A server program that hosts trained models and answers requests |
| Who makes it | Started by Philippe Tillet (Harvard, 2019 paper); developed at OpenAI with a community [Certain] | NVIDIA [Certain] |
| Level | Inside one operation (softmax, matmul, attention) | Whole models, HTTP/gRPC endpoints, many users |
| You use it to | Write @triton.jit kernels; read torch.compile output | Deploy models from TensorRT, PyTorch, ONNX and other back-ends, with dynamic batching [Certain] |
| Install | pip install triton (comes with PyTorch on Linux GPUs) [Likely] | Usually a Docker container from NVIDIA [Likely] |
They can even meet: a model served by Triton Inference Server can run kernels written in the Triton language. But they solve different problems at different levels. When someone says “we use Triton”, ask which one. Serving systems are covered in session 133.
A complete, checkable script
The script below checks both kernels against PyTorch and then times eager, torch.compile and Triton. It needs a Linux machine with an NVIDIA or AMD GPU for timing. Without a GPU, Triton’s interpreter mode (TRITON_INTERPRET=1) runs the kernels on the CPU with NumPy, slowly, which is enough to check correctness. [Likely] The random inputs are seeded, so the checks are deterministic.
import os, torch if not torch.cuda.is_available(): os.environ["TRITON_INTERPRET"] = "1" # CPU interpreter: correctness only import triton, triton.language as tl dev = "cuda" if torch.cuda.is_available() else "cpu" torch.manual_seed(0) # softmax_kernel / softmax and rmsnorm_kernel / rmsnorm: paste the code from above here def rmsnorm_eager(x, w, eps=1e-5): xf = x.float() return (xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + eps) * w.float()).to(x.dtype) M, N = (8192, 4096) if dev == "cuda" else (64, 1000) # N not a power of 2: tests the mask x = torch.randn(M, N, device=dev) w = torch.randn(N, device=dev) # 1) correctness assert torch.allclose(softmax(x), torch.softmax(x, dim=1), atol=1e-6) assert torch.allclose(rmsnorm(x, w), rmsnorm_eager(x, w), atol=1e-5) print("both kernels match PyTorch") # 2) timing (GPU only): do_bench warms up, repeats and synchronizes; result in ms if dev == "cuda": xb, wb = x.bfloat16(), w.bfloat16() compiled = torch.compile(rmsnorm_eager) compiled(xb, wb) # compile outside the timing min_bytes = (2 * M * N + N) * 2 # formula (86.9): read x and w, write out, BF16 for name, fn in [("eager", rmsnorm_eager), ("compile", compiled), ("triton", rmsnorm)]: ms = triton.testing.do_bench(lambda: fn(xb, wb)) print(f"{name:8s} {ms * 1e3:8.1f} us {min_bytes / (ms * 1e-3) / 1e9:7.0f} GB/s")
We do not print expected times here: they depend on the GPU, the driver and the PyTorch and Triton versions. Run it and put your own numbers into Figure 4. What you should see on a recent GPU: the eager version is clearly the slowest (formula 86.4 says by up to about 3.5 times in bytes, more with the casts), and torch.compile and the Triton kernel are close to each other. [Likely]
Where this is used
| Where | Triton kernels in use | Why Triton |
|---|---|---|
| torch.compile (TorchInductor) | Generated kernels for fused element-wise and reduction chains; Triton matmul templates with max-autotune [Certain] | One code generator for many fusion patterns |
| LLM training libraries (Liger Kernel, Unsloth) | RMSNorm, RoPE, SwiGLU, fused linear + cross-entropy [Certain for Liger; Likely for Unsloth] | Fewer bytes and less memory for activations and logits |
| LLM serving engines (vLLM, SGLang) | Fused mixture-of-experts layers, some attention and quantization kernels [Likely] | Fast to write for new models and new GPUs |
| Attention research | The Triton fused-attention tutorial, FlashAttention-style kernels for new attention variants [Certain for the tutorial] | Try a new attention idea in days instead of weeks |
| Quantized inference | Matrix multiply with on-the-fly dequantization of 4-bit or 8-bit weights [Likely] | Decode is memory-bound; reading fewer weight bytes is the whole gain (Track 5) |
| Video and VLM pipelines | Fused frame preprocessing (uint8 to float, resize, mean/std normalize) before a vision encoder; fused norms and softmax inside ViT encoders for video retrieval, action recognition and key-moment detection [Likely] | Video gives many frames per request, so per-frame memory traffic adds up quickly |
Common mistakes
- Wrong
othervalue for masked loads. Filling with 0 before a max can give a wrong max when all real values are negative; filling with \(-\infty\) before a plain sum gives NaN. Choose the value that does not change the reduction: \(-\infty\) for max and softmax, 0 for sums. - Forgetting the mask on the store. The load was masked, but an unmasked
tl.storewrites past the end of the tensor and corrupts other memory, sometimes silently. - Assuming the tensor is contiguous. A transposed or sliced tensor has a different row stride. Pass
x.stride(0)(as above) or call.contiguous()first. - A block that is not a power of two, or not a
tl.constexpr.tl.arangeneeds a compile-time power-of-two length. Usetriton.next_power_of_2(N)and mask the rest. - Accumulating in BF16 or FP16. Sums of thousands of squares lose precision or overflow in 16-bit. Cast loads to FP32 (
.to(tl.float32)), reduce in FP32, cast only the stored result. - Timing without synchronization, or including compile and autotune time. Use
triton.testing.do_benchor CUDA events, and call the kernel once before timing. - An autotune
keythat changes every call (for example the batch size in serving). The kernel re-benchmarks all configs again and again. Key on the shape that really changes the best config, or bucket it. - Writing only the forward pass for training. A Triton kernel is not differentiable by itself. Wrap it in a
torch.autograd.Functionwith a backward kernel (for RMSNorm, save \(\rho\) per row), and test the gradient withtorch.autograd.gradcheckin FP64 on small inputs.
Interview questions
What is Triton, and how is its programming model different from CUDA?basic
tl.load, tl.sum, tl.dot. The compiler maps the block to threads and warps, uses shared memory and tensor cores, and pipelines loads. You still choose the grid, block sizes, what to fuse and the memory layout.Is the Triton language the same as Triton Inference Server?basic
Why must BLOCK_SIZE be a tl.constexpr and a power of two, and how do you handle sizes that are not?core
tl.arange requires a power-of-two length. For other sizes, round up with triton.next_power_of_2 or loop over chunks, and mask the slots past the end in both loads and stores. The cost is wasted lanes, for example 39% when a 5,000-column row is padded to 8,192.Why is a fused softmax up to about 4 times faster than the naive PyTorch version?core
Your softmax row has 128,000 columns. What changes in the kernel?advanced
Walk me through an RMSNorm kernel. What precision do you use, and what do you save for the backward pass?core
tl.rsqrt, then store \(x \rho w\) cast back to BF16. Accumulating in FP32 avoids precision loss over thousands of terms. For training, store \(\rho\) per row (\(M\) floats) so the backward pass does not need to recompute the reduction; the backward kernel also needs \(x\), \(w\) and must reduce the gradient of \(w\) over all rows.What do num_warps and num_stages control?advanced
num_warps is how many warps of 32 threads execute one program, so it sets how many elements each thread holds (\(B / (32 \cdot \texttt{num\_warps})\)) and how many registers it needs. Too few warps for a big block causes register spills; too many leaves threads with little work and more synchronization cost in reductions. num_stages sets how many loop iterations ahead the compiler prefetches tiles into shared memory (software pipelining); it hides memory latency in loops like matmul but uses more shared memory, which can lower occupancy.How does triton.autotune work, and what can go wrong?core
triton.Config objects (meta-parameters such as BLOCK, plus num_warps and num_stages) and a key list of argument names. On the first call with a new key value, Triton benchmarks every config and caches the fastest. Pitfalls: the first call is slow (compile plus timing); a key that changes every call causes constant re-tuning; kernels that modify inputs in place get run many times during tuning, so use reset_to_zero or restore_value; and results tuned on one GPU may not be best on another.When would you write a Triton kernel instead of relying on torch.compile?advanced
How do you benchmark a kernel and decide whether it is good?core
triton.testing.do_bench, which synchronizes. Compute effective bandwidth = minimum bytes the operation must move divided by time, and compare it with the GPU’s peak (3.35 TB/s on an H100 SXM). For memory-bound kernels, 80–90% of the peak is very good; far less means wasted traffic, poor coalescing, too few programs, or launch overhead on small inputs. For compute-bound kernels such as matmul, compare FLOP/s with the tensor-core peak instead. Test several shapes, including sizes that are not powers of two.Inductor generated a Triton kernel for your model. How do you look at it, and why would you?advanced
TORCH_LOGS="output_code" (or use TORCH_COMPILE_DEBUG=1) to print or save the generated code. [Likely] You see which operations were fused into one kernel, the block sizes it chose and the loads and stores. This shows whether a chain you expected to be fused was split (for example by a graph break or a layout change), and it is a good starting point for a hand-written kernel.Exercises
- A vector of \(n = 10{,}000\) elements, \(\texttt{BLOCK} = 1024\). How many programs are launched, and how many slots are masked?
- Count the HBM traffic of the naive softmax for \(M = 1{,}024\), \(N = 32{,}768\) in BF16, and of the fused kernel. What is the ratio?
- A one-row-per-program kernel holds an FP32 row of \(N = 16{,}384\) in registers. What is the smallest
num_warpsthat keeps the row data at or below 255 values per thread? Why would you choose more in practice? - Run online softmax by hand on \((4, 1, 6)\) in chunks of sizes 2 and 1. Give \(m\) and \(\ell\) after each chunk, and the final probabilities.
- Your fused RMSNorm for \(M = 8{,}192\), \(N = 4{,}096\), BF16, takes 50 µs on an H100 SXM. What is its effective bandwidth and efficiency?
- Rewrite the vector-add kernel so it computes
out = a * x + ywith a scalara. Which arguments change and does the byte count change?
Answerscheck
a (a plain float, not a pointer) and store a * x + y. The bytes stay \(3n\) elements: the scalar is passed with the launch, so the extra multiply is free on a memory-bound kernel.Key takeaways
- A Triton kernel is the code of one program that works on a block of data. Launch \(\lceil n/B \rceil\) programs (86.1); each builds offsets and a mask (86.2) and uses block loads and stores.
- The compiler handles threads, coalescing, shared memory, tensor cores and pipelining. You still choose the grid, BLOCK_SIZE, num_warps, num_stages and what to fuse.
- Count bytes first. Eager softmax moves \(8MN + 4M\) elements, the fused kernel \(2MN\) (86.4); since \(t \ge\) bytes / bandwidth (86.5), fusion can give up to about 4×. RMSNorm: about 3.5×.
- Rows that do not fit in one block use online softmax (86.6), the same trick behind FlashAttention. Accumulate in FP32; mask with values that do not change the reduction.
- Registers per thread are at least \(B / (32 \cdot \texttt{num\_warps})\) (86.8); let
triton.autotunemeasure configs per key, and benchmark against eager and torch.compile with effective bandwidth (86.9). - The Triton language is a kernel compiler; Triton Inference Server is NVIDIA’s model server. Different things.
Key formulas
References for this session
- OpenAI and community (docs). Triton language documentation
- Tillet et al. (2019). Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations
- OpenAI and community (docs). Triton language documentation
- Tillet et al. (2019). Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations
- NVIDIA (code). Triton Inference Server
- NVIDIA (docs). CUDA C++ Programming Guide
- Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
- Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
- Milakov & Gimelshein (2018). Online normalizer calculation for softmax
- Zhang & Sennrich (2019). Root Mean Square Layer Normalization
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Hsu, Dai, Kothapalli, Song, Tang, Zhu, Shimizu, Sahni, Ning, Chen (2024). Liger Kernel: Efficient Triton Kernels for LLM Training
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Triton developers (docs). Triton tutorial: Fused Softmax