LLM & VLM courseBlog
86Track 5: LLM inference and serving · session 9 of 65

GPU kernels 2: write kernels in the Triton language

Triton is a Python-like language for writing GPU kernels. You describe what one program does with a whole block of data, and the compiler handles threads, memory moves and synchronization. In this session you write a vector add, a fused softmax and an RMSNorm kernel, autotune their block sizes, compare them with PyTorch eager and torch.compile, and learn not to confuse the Triton language with NVIDIA's Triton Inference Server.

Time: 3 to 4 hours (reading 2 hours, exercises and code 1 to 2 hours)Level: IntermediateBefore this: Sessions 42 and 85 (GPU architecture, CUDA basics); Session 22 for RMSNorm

What you will learn

  • The Triton programming model: a grid of programs, each working on a block of data with tl.arange, masks and pointer arithmetic
  • Why Triton is easier than CUDA: what the compiler does for you, and what you still decide
  • How to count the bytes an eager softmax moves (8MN + 4M elements) and why one fused kernel moves only 2MN
  • How to write a fused softmax and an RMSNorm kernel, and what to do when a row does not fit in one block (online softmax)
  • What BLOCK_SIZE, num_warps and num_stages control, and how triton.autotune picks them
  • How to benchmark a kernel fairly against PyTorch eager and torch.compile, and how to judge it with effective bandwidth
  • The name trap: the Triton language (a kernel compiler) is not the Triton Inference Server (a serving system)

Why this matters

In session 85 you saw that most operations in an LLM, apart from the big matrix multiplications, are limited by memory traffic, not by math. The fix is fusion: do several steps in one kernel, so the data travels from GPU memory (HBM) to the chip only once. But writing fused kernels in CUDA C++ is slow work: you manage threads, shared memory, synchronization and memory alignment by hand. Triton is a language that removes most of that work.

Triton is a Python-like language and compiler for GPU kernels. You write a Python function, decorate it with @triton.jit, and Triton compiles it to GPU machine code. It was introduced by Tillet, Kung and Cox in 2019 and is now developed by OpenAI and a community of contributors. [Certain] It runs on NVIDIA GPUs and also on AMD GPUs. [Certain] It matters to you for three reasons:

  • torch.compile writes Triton. When you call torch.compile on a GPU model, its backend (TorchInductor) generates Triton kernels for the fused element-wise and reduction parts (session 50). [Certain] To read what the compiler made, you must read Triton.
  • Real systems ship hand-written Triton kernels. Liger Kernel is a set of Triton kernels for LLM training (RMSNorm, RoPE, SwiGLU, cross-entropy). Its paper reports about 20% more training throughput and about 60% less GPU memory than the Hugging Face versions, on average. [Certain, numbers from the Liger paper] vLLM and other serving engines use Triton kernels too, for example for mixture-of-experts layers. [Likely]
  • Interviews ask for it. “Write a fused softmax in Triton”, “Why is the fused version faster?”, “What is BLOCK_SIZE?”, “Is Triton the same as Triton Inference Server?” are common questions for ML systems roles.

The plan: first the programming model (blocks, not threads). Then a first kernel (vector add). Then two real LLM kernels: fused softmax and RMSNorm. Then autotuning, fair benchmarks against PyTorch eager and torch.compile, and the name trap at the end.

The programming model: blocks of data, not single threads

In CUDA you write the code for one thread. Each thread handles one element (or a few), and you think about warps, shared memory and barriers yourself. In Triton you write the code for one program, and a program handles a whole block of elements at once. A program is roughly one CUDA thread block. [Certain] Inside it, Triton decides how to spread the block over threads and warps.

Three ideas are enough to start:

  1. The grid. You launch many copies of the same program. Each copy asks tl.program_id(0) for its number, called pid. For \(n\) elements and a block of size \(B\), you need
\[ G = \left\lceil \frac{n}{B} \right\rceil \tag{86.1} \]

programs. With \(n = 1{,}000\) and \(B = 256\), that is \(G = 4\) programs, and they cover \(4 \times 256 = 1{,}024\) slots, 24 more than needed.

  1. Offsets and masks. Each program builds a vector of indices with tl.arange(0, B) and shifts it to its own part of the data. Slots past the end are switched off with a mask:
\[ \mathbf{o} = \mathrm{pid} \cdot B + (0, 1, \dots, B - 1), \qquad \mathbf{m} = (\mathbf{o} \lt n) \tag{86.2} \]
Read the symbols
\(\mathrm{pid}\)
the number of this program, from 0 to \(G - 1\)
\(B\)
the block size; in Triton it must be a power of two and known at compile time (tl.constexpr)
\(\mathbf{o}\)
a vector of \(B\) element offsets: which elements this program touches
\(\mathbf{m}\)
a vector of \(B\) true/false values: true where the offset is inside the data
  1. Pointers and block loads. A tensor argument arrives as a pointer to its first element. x_ptr + offs is a block of \(B\) pointers. tl.load(x_ptr + offs, mask=mask, other=0.0) reads all of them at once and puts other where the mask is false. tl.store writes a block back. Math such as +, tl.exp, tl.max and tl.sum works on whole blocks, like NumPy.

Figure 1 shows the grid, the offsets and the mask. Click any slot to see which program owns it and whether it is masked.

BLOCK_SIZE \(B\):
active slot (mask true)masked slot (mask false, no load or store)selected slot
Figure 1. The Triton grid for a 1-D kernel. Each row is one program. Program \(\mathrm{pid}\) owns offsets \(\mathrm{pid} \cdot B\) to \(\mathrm{pid} \cdot B + B - 1\) (formula 86.2). Only the last program can have masked slots.
How to read Figure 1

Change \(n\) and watch the number of rows: it is always \(\lceil n/B \rceil\) (formula 86.1). The grey slots at the end are the price of a fixed, power-of-two block: they exist in the program but the mask turns their loads and stores off. Bigger blocks mean fewer programs but more waste in the last one. For a real vector of a million elements, the waste is tiny; for a softmax row of 5,000 columns padded to 8,192, it is not (see Figure 3).

Your first kernel: vector add

Here is the whole idea in pseudo-code. Each program adds one block.

Algorithm 1: vector add, the code of one Triton program
  1. input: pointers x_ptr, y_ptr, out_ptr, length \(n\), compile-time block size \(B\)
  2. \(\mathrm{pid} \leftarrow\) tl.program_id(0) // which block am I?
  3. \(\mathbf{o} \leftarrow \mathrm{pid} \cdot B +\) tl.arange(0, B); \(\;\mathbf{m} \leftarrow \mathbf{o} \lt n\) // formula (86.2)
  4. \(\mathbf{x} \leftarrow\) load(x_ptr \(+ \mathbf{o}\), mask \(\mathbf{m}\)); \(\;\mathbf{y} \leftarrow\) load(y_ptr \(+ \mathbf{o}\), mask \(\mathbf{m}\)) // one block read each
  5. store(out_ptr \(+ \mathbf{o}\), \(\mathbf{x} + \mathbf{y}\), mask \(\mathbf{m}\))
  6. host: launch the kernel on a grid of \(\lceil n/B \rceil\) programs (formula 86.1)

In Triton this is almost line by line the same:

import torch, triton
import triton.language as tl

@triton.jit
def add_kernel(x_ptr, y_ptr, out_ptr, n, BLOCK: tl.constexpr):
    pid = tl.program_id(0)                       # which program am I?
    offs = pid * BLOCK + tl.arange(0, BLOCK)     # formula (86.2)
    mask = offs < n
    x = tl.load(x_ptr + offs, mask=mask)
    y = tl.load(y_ptr + offs, mask=mask)
    tl.store(out_ptr + offs, x + y, mask=mask)

def add(x, y):
    out = torch.empty_like(x)
    n = x.numel()
    grid = (triton.cdiv(n, 1024),)               # formula (86.1)
    add_kernel[grid](x, y, out, n, BLOCK=1024)   # tensors become pointers
    return out

Three details to notice. First, add_kernel[grid](...) is the launch: the square brackets hold the grid, the round ones the arguments. Second, BLOCK is a tl.constexpr: Triton compiles a separate version of the kernel for each value, so the shapes of all blocks are known when the code is compiled. Third, there is no thread index anywhere. You never wrote threadIdx.x.

Worked example: how fast can vector add be?

Add two FP32 vectors of \(n = 2^{20} = 1{,}048{,}576\) elements. The kernel reads two vectors and writes one: \(3 \times 4 \times 2^{20} = 12.6\) MB. An H100 SXM reads about 3.35 TB/s from HBM. [Certain, NVIDIA datasheet] So the kernel cannot take less than \(12.6\ \text{MB} / 3.35\ \text{TB/s} \approx 3.8\ \mu s\). It does only one addition per 12 bytes, so the math units are nearly idle: this kernel is purely memory-bound. A good Triton vector add should come close to this limit on large inputs; on tiny inputs, the launch overhead (a few microseconds) dominates. [Likely]

What the compiler does for you, and what you still decide

The Triton paper calls its unit of work a tile: a block of data that a program loads, computes on, and stores. [Certain] Because the compiler sees whole tiles, not single threads, it can make many low-level choices that a CUDA programmer makes by hand.

JobIn CUDA C++In Triton
Split work over threads and warpsYou, with threadIdx, blockIdxCompiler; you only choose num_warps
Coalesced memory access (neighbouring threads read neighbouring addresses)You arrange the indexingCompiler, from the block layout
Shared memory and barriers (__syncthreads)You allocate and synchronizeCompiler, for reductions and for tl.dot
Tensor cores for matrix multiplyYou use special instructions or librariesCompiler, when you call tl.dot on blocks
Software pipelining (load the next tile while computing this one)You write double bufferingCompiler; you choose num_stages
Which data each program handles; block sizes; what to fuseYouStill you

So Triton does not make design free. You still choose the grid, the block sizes and the order of loads, and you still have to count bytes. What disappears is the error-prone bookkeeping. The price: a little less control than CUDA for the very last few percent of speed. The fastest attention kernels on the newest GPUs (FlashAttention-3) are still written in CUDA. [Certain]

Fused softmax: count the bytes first

Softmax over each row of an \(M \times N\) matrix (for example, attention scores, or logits over a vocabulary) is, with the max subtracted for numerical stability (session 2):

\[ \softmax(\mathbf{x})_j = \frac{e^{x_j - m}}{\sum_{k=1}^{N} e^{x_k - m}}, \qquad m = \max_k x_k \tag{86.3} \]

Write it in PyTorch the plain way, one operation per line, and each line becomes one kernel that reads its inputs from HBM and writes its output back:

x_max = x.max(dim=1)[0]            # read MN,     write M
z = x - x_max[:, None]             # read MN + M, write MN
num = torch.exp(z)                 # read MN,     write MN
den = num.sum(dim=1)               # read MN,     write M
out = num / den[:, None]           # read MN + M, write MN

Add the columns. The eager version reads \(5MN + 2M\) elements and writes \(3MN + 2M\). [Certain, the same count is in the Triton tutorial] A fused kernel loads each row once, keeps it on the chip (in registers), does all five steps there, and writes the result once:

\[ \text{eager: } (5MN + 2M) + (3MN + 2M) = 8MN + 4M, \qquad \text{fused: } MN + MN = 2MN \tag{86.4} \]

For large \(N\), the ratio is about \(8MN / 2MN = 4\). When a kernel is memory-bound, its time is at least its bytes divided by the bandwidth:

\[ t \;\ge\; \frac{\text{bytes moved}}{\text{HBM bandwidth}} \tag{86.5} \]

So a fused softmax can be up to about 4 times faster, and that is what the Triton tutorial measures against the naive version. [Certain, from the tutorial; the exact speed depends on the GPU] It also wins a little against torch.softmax, which is already a single fused kernel but is written for any shape and layout. [Certain, as stated in the tutorial]

Worked example: softmax over 8,192 rows of 4,096 FP32 scores

\(M = 8{,}192\), \(N = 4{,}096\), 4 bytes per element. Eager: \((8MN + 4M) \times 4 = 1{,}073.9\) MB. Fused: \(2MN \times 4 = 268.4\) MB. Ratio: 4.0. At 3.35 TB/s (formula 86.5), the lower bounds are \(320.6\ \mu s\) for eager and \(80.1\ \mu s\) for fused. The eager version also launches 5 kernels instead of 1.

Figure 2 walks through the five eager kernels and then the fused one, with the bytes each moves.

Element type:
bytes read from HBMbytes written to HBM
Figure 2. Memory traffic of softmax, eager versus fused. The bars at the bottom add up all bytes so far (formula 86.4). Times are lower bounds at the 3.35 TB/s of an H100 SXM (formula 86.5); real kernels reach a fraction of that.
How to read Figure 2

Step through the eager kernels: each one reads a full \(M \times N\) matrix, and three of them also write one. The intermediate tensors z and num exist only to be read again by the next kernel. Step 6 is the fused kernel: one read, one write, and the intermediates never leave the chip. Change \(N\): the ratio stays close to 4 because the \(4M\) term is small next to \(8MN\). BF16 halves every number but not the ratio.

The fused softmax kernel

One program handles one row. This works when a full row fits in one block (up to some tens of thousands of columns; see Figure 3).

Algorithm 2: fused softmax, the code of one program (one row)
  1. input: pointers to input and output, row stride, \(N\) columns, block size \(B \ge N\) (a power of two)
  2. \(r \leftarrow\) tl.program_id(0) // the row this program owns
  3. \(\mathbf{o} \leftarrow (0, \dots, B - 1)\); \(\;\mathbf{m} \leftarrow \mathbf{o} \lt N\)
  4. \(\mathbf{x} \leftarrow\) load(row \(r\) at \(\mathbf{o}\), mask \(\mathbf{m}\), other \(= -\infty\)) // masked slots give \(e^{-\infty} = 0\)
  5. \(m \leftarrow \max(\mathbf{x})\); \(\;\mathbf{e} \leftarrow \exp(\mathbf{x} - m)\); \(\;s \leftarrow \sum \mathbf{e}\) // all in registers
  6. store(row \(r\) at \(\mathbf{o}\), \(\mathbf{e} / s\), mask \(\mathbf{m}\)) // the only write
  7. host: \(B \leftarrow\) next power of two \(\ge N\); launch \(M\) programs

Look at line 4. The fill value for masked slots must not change the answer. For a max and a sum of exponentials, \(-\infty\) is right: it never wins the max and \(e^{-\infty} = 0\) adds nothing. For a plain sum, the right fill is 0. Choosing the wrong other is a classic bug (see Common mistakes).

@triton.jit
def softmax_kernel(out_ptr, in_ptr, in_stride, out_stride, n_cols, BLOCK: tl.constexpr):
    row = tl.program_id(0)
    offs = tl.arange(0, BLOCK)
    mask = offs < n_cols
    x = tl.load(in_ptr + row * in_stride + offs, mask=mask, other=-float("inf"))
    x = x - tl.max(x, axis=0)                     # formula (86.3), stable
    num = tl.exp(x)
    den = tl.sum(num, axis=0)
    tl.store(out_ptr + row * out_stride + offs, num / den, mask=mask)

def softmax(x):
    M, N = x.shape
    out = torch.empty_like(x)
    BLOCK = triton.next_power_of_2(N)             # the whole row in one block
    softmax_kernel[(M,)](out, x, x.stride(0), out.stride(0), N, BLOCK=BLOCK, num_warps=8)
    return out

The official tutorial adds one more idea: instead of launching \(M\) programs, it launches only as many as fit on the GPU at once and lets each one loop over several rows (a “persistent” kernel). It picks that count from the kernel’s register and shared-memory use and the number of SMs. [Certain, from the tutorial]

When a row does not fit: online softmax

For a vocabulary of 128,000 logits, one block would need 131,072 slots, which is too many registers (Figure 3). Then the program must loop over the row in chunks. The problem: the max \(m\) is not known until the end. The online softmax trick keeps a running max \(m\) and a running sum \(\ell\), and rescales the sum whenever the max grows:

\[ m' = \max\big(m,\ \max_{j \in \text{chunk}} x_j\big), \qquad \ell' = \ell \, e^{m - m'} + \sum_{j \in \text{chunk}} e^{x_j - m'} \tag{86.6} \]
Read the symbols
\(m, \ell\)
the max and the sum of \(e^{x - m}\) over all chunks seen so far; start at \(m = -\infty\), \(\ell = 0\)
\(m', \ell'\)
the same values after adding one more chunk
\(e^{m - m'}\)
a correction factor (at most 1) that re-expresses the old sum relative to the new max

After the last chunk, \(\ell\) equals the true denominator. A second pass writes \(e^{x_j - m} / \ell\). This costs one extra read of the row but no extra intermediate tensor. The same trick, applied to attention scores, is the heart of FlashAttention (session 124).

Worked example: online softmax on (1, 3, 2, 5) in chunks of 2

Chunk 1 = (1, 3): \(m = 3\), \(\ell = e^{-2} + e^{0} = 1.1353\). Chunk 2 = (2, 5): \(m' = 5\), \(\ell' = 1.1353 \cdot e^{3 - 5} + e^{2-5} + e^{0} = 1.2034\). Computing directly, \(\sum_k e^{x_k - 5} = 1.2034\): the same. The softmax is (0.0152, 0.1125, 0.0414, 0.8310).

An RMSNorm kernel

Every block of Llama-style models normalizes its input with RMSNorm (session 22). For a row \(\mathbf{x} \in \R^{N}\) with a learned gain \(\mathbf{w}\):

\[ \mathrm{RMSNorm}(\mathbf{x})_j = \frac{x_j}{\sqrt{\frac{1}{N}\sum_{k=1}^{N} x_k^2 + \varepsilon}} \; w_j \tag{86.7} \]
Read the symbols
\(N\)
the hidden size (4,096 for Llama 3 8B)
\(\varepsilon\)
a small constant such as \(10^{-5}\), so we never divide by zero
\(w_j\)
the learned gain for feature \(j\), shared by all rows

The plain PyTorch line x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * w launches about six kernels (square, mean, add, rsqrt, two multiplies). Count them like the softmax, with all tensors in one element type: they read \(4MN + 3M + N\) elements and write \(3MN + 3M\). A fused kernel reads the row and \(\mathbf{w}\) once and writes once: \(2MN + N\). That is about 3.5 times less traffic.

Worked example: RMSNorm for 8,192 tokens of a 4,096-wide model in BF16

\(M = 8{,}192\), \(N = 4{,}096\), 2 bytes. Eager: \((7MN + 6M + N) \times 2 = 469.9\) MB, at least \(140.3\ \mu s\) on an H100. Fused: \((2MN + N) \times 2 = 134.2\) MB, at least \(40.1\ \mu s\). Ratio 3.5. Real code is often worse for eager, because it also casts BF16 to FP32 and back, which adds two more full passes.

Algorithm 3: RMSNorm with a loop over chunks, the code of one program (one row)
  1. input: pointers to \(\mathbf{x}\), \(\mathbf{w}\), output; row stride; \(N\); \(\varepsilon\); tunable block size \(B\) (can be smaller than \(N\))
  2. \(r \leftarrow\) tl.program_id(0); \(\;\mathbf{a} \leftarrow\) zeros(\(B\)) in FP32 // accumulator
  3. for start \(= 0, B, 2B, \dots \lt N\): load chunk \(\mathbf{x}_c\) (mask, other \(= 0\)), cast to FP32, \(\mathbf{a} \leftarrow \mathbf{a} + \mathbf{x}_c^2\)
  4. \(\rho \leftarrow 1 / \sqrt{\sum \mathbf{a} / N + \varepsilon}\) // formula (86.7); save \(\rho\) for the backward pass if training
  5. for start \(= 0, B, 2B, \dots \lt N\): load \(\mathbf{x}_c\) and \(\mathbf{w}_c\), store \(\mathbf{x}_c \cdot \rho \cdot \mathbf{w}_c\) cast to the output type
  6. host: launch \(M\) programs; let the autotuner choose \(B\) and num_warps

The second loop reads the row again. For \(N = 4{,}096\) in BF16 a row is only 8 KB, so the second read usually hits the on-chip L2 cache rather than HBM. [Likely] If \(B \ge N\), there is one chunk, and you can keep the row in registers and skip the second load completely.

Block size, warps and autotuning

Three numbers decide how a Triton kernel runs, and you choose all three:

  • BLOCK_SIZE (a tl.constexpr): how many elements one program works on at a time.
  • num_warps: how many warps (groups of 32 threads) run one program. More warps means fewer elements per thread.
  • num_stages: how many tiles ahead the compiler loads in loops (software pipelining). More stages hide memory latency but use more shared memory. It mostly matters for matrix multiplication loops.

A useful first estimate: in a kernel that keeps one block of FP32 values in registers, each thread holds

\[ e = \frac{B}{32 \cdot \texttt{num\_warps}} \ \text{ values, so at least } e \text{ registers of 32 bits per thread} \tag{86.8} \]

On an H100, an SM has 65,536 registers of 32 bits and a thread can use at most 255. [Certain, CUDA programming guide] If a thread needs more, the compiler spills values to slow local memory, and the kernel slows down a lot. If threads need many registers, fewer warps fit on an SM at once, and there are fewer warps to hide memory latency (session 42). The real register count is higher than \(e\): addresses, indices and temporaries need registers too. So formula (86.8) is a lower bound, good for spotting obviously bad choices. Figure 3 applies it.

num_warps:
Figure 3. A one-row-per-program kernel (like Algorithm 2) with \(B = \) next power of two \(\ge N\), on an H100 (65,536 registers per SM, at most 255 per thread, at most 64 warps per SM). The register bar counts only the row data (formula 86.8), so real kernels use more; the warps bar is an upper bound.
How to read Figure 3

Start at \(N = 4{,}096\) and 4 warps: 32 values per thread, fine. Move to \(N = 32{,}768\): with 4 warps each thread would need 256 values, above the 255 limit, so it spills; with 16 warps it is 64. This is why long rows need more warps, or a chunked loop like Algorithm 3. Now try \(N = 5{,}000\): the block becomes 8,192 and 39% of the slots are masked. Because these effects interact and depend on the GPU, the practical answer is to measure, which is what the autotuner does.

triton.autotune

The autotuner is a decorator. You give it a list of triton.Config candidates (block sizes, num_warps, num_stages) and a key: a list of argument names. The first time the kernel is called with a new value of the key arguments, Triton runs every candidate, times it, and caches the fastest for that key. [Certain, Triton API docs]

@triton.autotune(
    configs=[triton.Config({"BLOCK": b}, num_warps=w)
             for b in (256, 512, 1024, 2048) for w in (2, 4, 8)],
    key=["n_cols"],                              # re-tune when the row length changes
)
@triton.jit
def rmsnorm_kernel(out_ptr, x_ptr, w_ptr, stride, n_cols, eps, BLOCK: tl.constexpr):
    row = tl.program_id(0)
    x_row = x_ptr + row * stride
    acc = tl.zeros([BLOCK], dtype=tl.float32)
    for start in range(0, n_cols, BLOCK):         # pass 1: sum of squares
        offs = start + tl.arange(0, BLOCK)
        x = tl.load(x_row + offs, mask=offs < n_cols, other=0.0).to(tl.float32)
        acc += x * x
    rstd = tl.rsqrt(tl.sum(acc, axis=0) / n_cols + eps)      # formula (86.7)
    for start in range(0, n_cols, BLOCK):         # pass 2: normalize and scale
        offs = start + tl.arange(0, BLOCK)
        m = offs < n_cols
        x = tl.load(x_row + offs, mask=m, other=0.0).to(tl.float32)
        w = tl.load(w_ptr + offs, mask=m, other=0.0).to(tl.float32)
        tl.store(out_ptr + row * stride + offs, (x * rstd * w).to(out_ptr.dtype.element_ty), mask=m)

def rmsnorm(x, w, eps=1e-5):
    M, N = x.shape
    out = torch.empty_like(x)
    rmsnorm_kernel[(M,)](out, x, w, x.stride(0), N, eps)   # BLOCK comes from the chosen config
    return out

Three practical points. (1) The first call is slow: it compiles and times 12 configs. Never include it in a benchmark. (2) A good key lists the arguments that change the best choice (here the row length), but not ones that change every call, or the kernel re-tunes all the time. (3) If a kernel writes into one of its inputs, every trial run changes that input again; the autotuner has reset_to_zero and restore_value options for this. [Certain, Triton API docs]

Compare with PyTorch eager and torch.compile

You now have three ways to run the same RMSNorm:

WayWhat runs on the GPUStrengthWeakness
PyTorch eagerOne pre-built kernel per operation (about 6 for RMSNorm)No compile step; easy to debugEvery intermediate goes through HBM; many launches
torch.compileInductor fuses the chain and generates a Triton kernel [Certain]One line of code; also fuses the operations around itCompile time; graph breaks; the kernel it writes may not be the best possible
Hand-written TritonYour kernelFull control: layout, fusion with neighbours, a custom backwardYou own correctness, the backward pass and the tuning

For a simple element-wise and reduction chain like RMSNorm or softmax, torch.compile often produces a kernel close to a good hand-written one, because it generates Triton itself. [Likely] Hand-written kernels win when the fusion crosses boundaries the compiler does not cross, for example a cross-entropy fused with the final matrix multiply so the full logits tensor is never stored (one of Liger’s kernels), or attention (FlashAttention). [Likely]

How to benchmark fairly

GPU calls are asynchronous: Python returns before the kernel ends. Timing with time.time() around a call measures almost nothing. Use CUDA events or triton.testing.do_bench, which warms up, repeats the call and synchronizes. Then judge the result with the effective bandwidth: the minimum bytes the operation must move, divided by the measured time:

\[ \mathrm{BW}_{\mathrm{eff}} = \frac{\text{minimum bytes}}{t_{\text{measured}}}, \qquad \text{efficiency} = \frac{\mathrm{BW}_{\mathrm{eff}}}{\mathrm{BW}_{\mathrm{peak}}} \tag{86.9} \]

Use the minimum bytes (for softmax, \(2MN\) elements) for all three versions, not the bytes each one actually moved. Then the number tells you how close each version is to the best possible on this GPU. A memory-bound kernel that reaches 80–90% of the peak bandwidth is very good; that is roughly the best that well-tuned kernels reach in practice. [Likely] The softmax plot in the Triton tutorial is drawn exactly this way, in GB/s. Figure 4 is the calculator: enter a time you measured, read the efficiency, and see where the kernel sits on the roofline.

Kernel:
GPU:Type:
Figure 4. Effective bandwidth (formula 86.9) and the position on the roofline for a kernel time you enter. Peak HBM bandwidth: H100 SXM 3.35 TB/s, A100 80GB SXM 2.04 TB/s, RTX 4090 1.01 TB/s; dense BF16 tensor-core peaks 989, 312 and 165 TFLOP/s. FLOPs per element are rough counts (add 1, softmax 5, RMSNorm 4). The time is your input, not a measurement of ours.
How to read Figure 4

The dot sits far to the left of the ridge (the corner of the roof): a few FLOPs per element over 4 to 12 bytes per element gives an arithmetic intensity of about 1 FLOP per byte or less, while the H100 ridge is at about 295 FLOP/byte. So these kernels can only be limited by bandwidth, and the right score is GB/s, not FLOP/s. Slide the time down: the readout refuses times faster than the bandwidth allows, because no kernel can beat formula (86.5). If a time you measured seems to do so, you probably forgot to synchronize.

The name trap: Triton is two different things

Two well-known projects share the name, and interviewers like to check that you know the difference.

Triton (the language)Triton Inference Server
What it isA language and compiler for writing GPU kernelsA server program that hosts trained models and answers requests
Who makes itStarted by Philippe Tillet (Harvard, 2019 paper); developed at OpenAI with a community [Certain]NVIDIA [Certain]
LevelInside one operation (softmax, matmul, attention)Whole models, HTTP/gRPC endpoints, many users
You use it toWrite @triton.jit kernels; read torch.compile outputDeploy models from TensorRT, PyTorch, ONNX and other back-ends, with dynamic batching [Certain]
Installpip install triton (comes with PyTorch on Linux GPUs) [Likely]Usually a Docker container from NVIDIA [Likely]

They can even meet: a model served by Triton Inference Server can run kernels written in the Triton language. But they solve different problems at different levels. When someone says “we use Triton”, ask which one. Serving systems are covered in session 133.

A complete, checkable script

The script below checks both kernels against PyTorch and then times eager, torch.compile and Triton. It needs a Linux machine with an NVIDIA or AMD GPU for timing. Without a GPU, Triton’s interpreter mode (TRITON_INTERPRET=1) runs the kernels on the CPU with NumPy, slowly, which is enough to check correctness. [Likely] The random inputs are seeded, so the checks are deterministic.

import os, torch
if not torch.cuda.is_available():
    os.environ["TRITON_INTERPRET"] = "1"          # CPU interpreter: correctness only
import triton, triton.language as tl
dev = "cuda" if torch.cuda.is_available() else "cpu"
torch.manual_seed(0)

# softmax_kernel / softmax and rmsnorm_kernel / rmsnorm: paste the code from above here

def rmsnorm_eager(x, w, eps=1e-5):
    xf = x.float()
    return (xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + eps) * w.float()).to(x.dtype)

M, N = (8192, 4096) if dev == "cuda" else (64, 1000)   # N not a power of 2: tests the mask
x = torch.randn(M, N, device=dev)
w = torch.randn(N, device=dev)

# 1) correctness
assert torch.allclose(softmax(x), torch.softmax(x, dim=1), atol=1e-6)
assert torch.allclose(rmsnorm(x, w), rmsnorm_eager(x, w), atol=1e-5)
print("both kernels match PyTorch")

# 2) timing (GPU only): do_bench warms up, repeats and synchronizes; result in ms
if dev == "cuda":
    xb, wb = x.bfloat16(), w.bfloat16()
    compiled = torch.compile(rmsnorm_eager)
    compiled(xb, wb)                                  # compile outside the timing
    min_bytes = (2 * M * N + N) * 2                   # formula (86.9): read x and w, write out, BF16
    for name, fn in [("eager", rmsnorm_eager), ("compile", compiled), ("triton", rmsnorm)]:
        ms = triton.testing.do_bench(lambda: fn(xb, wb))
        print(f"{name:8s} {ms * 1e3:8.1f} us   {min_bytes / (ms * 1e-3) / 1e9:7.0f} GB/s")

We do not print expected times here: they depend on the GPU, the driver and the PyTorch and Triton versions. Run it and put your own numbers into Figure 4. What you should see on a recent GPU: the eager version is clearly the slowest (formula 86.4 says by up to about 3.5 times in bytes, more with the casts), and torch.compile and the Triton kernel are close to each other. [Likely]

Where this is used

WhereTriton kernels in useWhy Triton
torch.compile (TorchInductor)Generated kernels for fused element-wise and reduction chains; Triton matmul templates with max-autotune [Certain]One code generator for many fusion patterns
LLM training libraries (Liger Kernel, Unsloth)RMSNorm, RoPE, SwiGLU, fused linear + cross-entropy [Certain for Liger; Likely for Unsloth]Fewer bytes and less memory for activations and logits
LLM serving engines (vLLM, SGLang)Fused mixture-of-experts layers, some attention and quantization kernels [Likely]Fast to write for new models and new GPUs
Attention researchThe Triton fused-attention tutorial, FlashAttention-style kernels for new attention variants [Certain for the tutorial]Try a new attention idea in days instead of weeks
Quantized inferenceMatrix multiply with on-the-fly dequantization of 4-bit or 8-bit weights [Likely]Decode is memory-bound; reading fewer weight bytes is the whole gain (Track 5)
Video and VLM pipelinesFused frame preprocessing (uint8 to float, resize, mean/std normalize) before a vision encoder; fused norms and softmax inside ViT encoders for video retrieval, action recognition and key-moment detection [Likely]Video gives many frames per request, so per-frame memory traffic adds up quickly

Common mistakes

  • Wrong other value for masked loads. Filling with 0 before a max can give a wrong max when all real values are negative; filling with \(-\infty\) before a plain sum gives NaN. Choose the value that does not change the reduction: \(-\infty\) for max and softmax, 0 for sums.
  • Forgetting the mask on the store. The load was masked, but an unmasked tl.store writes past the end of the tensor and corrupts other memory, sometimes silently.
  • Assuming the tensor is contiguous. A transposed or sliced tensor has a different row stride. Pass x.stride(0) (as above) or call .contiguous() first.
  • A block that is not a power of two, or not a tl.constexpr. tl.arange needs a compile-time power-of-two length. Use triton.next_power_of_2(N) and mask the rest.
  • Accumulating in BF16 or FP16. Sums of thousands of squares lose precision or overflow in 16-bit. Cast loads to FP32 (.to(tl.float32)), reduce in FP32, cast only the stored result.
  • Timing without synchronization, or including compile and autotune time. Use triton.testing.do_bench or CUDA events, and call the kernel once before timing.
  • An autotune key that changes every call (for example the batch size in serving). The kernel re-benchmarks all configs again and again. Key on the shape that really changes the best config, or bucket it.
  • Writing only the forward pass for training. A Triton kernel is not differentiable by itself. Wrap it in a torch.autograd.Function with a backward kernel (for RMSNorm, save \(\rho\) per row), and test the gradient with torch.autograd.gradcheck in FP64 on small inputs.

Interview questions

What is Triton, and how is its programming model different from CUDA?basic
Triton is a Python-embedded language and compiler for GPU kernels. In CUDA you write code for one thread and manage threads, shared memory, barriers and coalescing yourself. In Triton you write code for one program (about one thread block) that operates on whole blocks of data with vector operations like tl.load, tl.sum, tl.dot. The compiler maps the block to threads and warps, uses shared memory and tensor cores, and pipelines loads. You still choose the grid, block sizes, what to fuse and the memory layout.
Is the Triton language the same as Triton Inference Server?basic
No. The Triton language (from Tillet’s 2019 paper, developed at OpenAI) is a kernel compiler: it works inside one operation. Triton Inference Server is NVIDIA’s model-serving software: it loads models from TensorRT, PyTorch, ONNX and other back-ends and serves requests over HTTP or gRPC with dynamic batching. They work at different levels; a served model could use kernels written in Triton.
Why must BLOCK_SIZE be a tl.constexpr and a power of two, and how do you handle sizes that are not?core
Triton compiles a separate kernel for each constexpr value, so all block shapes are known at compile time; this lets it choose the layout over threads, the register allocation and the vector width. tl.arange requires a power-of-two length. For other sizes, round up with triton.next_power_of_2 or loop over chunks, and mask the slots past the end in both loads and stores. The cost is wasted lanes, for example 39% when a 5,000-column row is padded to 8,192.
Why is a fused softmax up to about 4 times faster than the naive PyTorch version?core
Softmax is memory-bound: a few FLOPs per element. The naive version runs five kernels (max, subtract, exp, sum, divide) that together read \(5MN + 2M\) elements and write \(3MN + 2M\), so \(8MN + 4M\) in total. The fused kernel loads each row once into registers, does all steps on chip and writes once: \(2MN\). Time is bounded by bytes over bandwidth, so the ratio is about 4. It also launches 1 kernel instead of 5.
Your softmax row has 128,000 columns. What changes in the kernel?advanced
One block would be 131,072 slots, which needs far more than 255 registers per thread even with 32 warps (\(131{,}072 / 1{,}024 = 128\) values per thread, plus temporaries), so it spills or fails. Instead loop over the row in chunks and use online softmax: keep a running max \(m\) and sum \(\ell\), and rescale \(\ell\) by \(e^{m - m'}\) when the max grows. A second pass writes \(e^{x - m}/\ell\). For one very long row you can also split the row over several programs and combine their \((m, \ell)\) pairs in a second small kernel, the same idea as Flash-Decoding.
Walk me through an RMSNorm kernel. What precision do you use, and what do you save for the backward pass?core
One program per row. Load the row (masked, other 0), cast to FP32, sum the squares, compute \(\rho = 1/\sqrt{\text{mean} + \varepsilon}\) with tl.rsqrt, then store \(x \rho w\) cast back to BF16. Accumulating in FP32 avoids precision loss over thousands of terms. For training, store \(\rho\) per row (\(M\) floats) so the backward pass does not need to recompute the reduction; the backward kernel also needs \(x\), \(w\) and must reduce the gradient of \(w\) over all rows.
What do num_warps and num_stages control?advanced
num_warps is how many warps of 32 threads execute one program, so it sets how many elements each thread holds (\(B / (32 \cdot \texttt{num\_warps})\)) and how many registers it needs. Too few warps for a big block causes register spills; too many leaves threads with little work and more synchronization cost in reductions. num_stages sets how many loop iterations ahead the compiler prefetches tiles into shared memory (software pipelining); it hides memory latency in loops like matmul but uses more shared memory, which can lower occupancy.
How does triton.autotune work, and what can go wrong?core
You pass a list of triton.Config objects (meta-parameters such as BLOCK, plus num_warps and num_stages) and a key list of argument names. On the first call with a new key value, Triton benchmarks every config and caches the fastest. Pitfalls: the first call is slow (compile plus timing); a key that changes every call causes constant re-tuning; kernels that modify inputs in place get run many times during tuning, so use reset_to_zero or restore_value; and results tuned on one GPU may not be best on another.
When would you write a Triton kernel instead of relying on torch.compile?advanced
First profile. If torch.compile already fuses the hot chain into one kernel at high effective bandwidth, stop. Write a kernel when (a) the fusion crosses a boundary the compiler will not cross, such as fusing the final matmul with cross-entropy so the logits are never stored, or attention; (b) you need a custom memory layout or a custom backward that saves less memory; (c) graph breaks or dynamic shapes prevent compilation; or (d) you need a new operator, such as a new quantization format. Then compare your kernel against both eager and compiled versions with the same minimum-bytes metric.
How do you benchmark a kernel and decide whether it is good?core
Warm up first (compile, autotune), then time many runs with CUDA events or triton.testing.do_bench, which synchronizes. Compute effective bandwidth = minimum bytes the operation must move divided by time, and compare it with the GPU’s peak (3.35 TB/s on an H100 SXM). For memory-bound kernels, 80–90% of the peak is very good; far less means wasted traffic, poor coalescing, too few programs, or launch overhead on small inputs. For compute-bound kernels such as matmul, compare FLOP/s with the tensor-core peak instead. Test several shapes, including sizes that are not powers of two.
Inductor generated a Triton kernel for your model. How do you look at it, and why would you?advanced
Set TORCH_LOGS="output_code" (or use TORCH_COMPILE_DEBUG=1) to print or save the generated code. [Likely] You see which operations were fused into one kernel, the block sizes it chose and the loads and stores. This shows whether a chain you expected to be fused was split (for example by a graph break or a layout change), and it is a good starting point for a hand-written kernel.

Exercises

  1. A vector of \(n = 10{,}000\) elements, \(\texttt{BLOCK} = 1024\). How many programs are launched, and how many slots are masked?
  2. Count the HBM traffic of the naive softmax for \(M = 1{,}024\), \(N = 32{,}768\) in BF16, and of the fused kernel. What is the ratio?
  3. A one-row-per-program kernel holds an FP32 row of \(N = 16{,}384\) in registers. What is the smallest num_warps that keeps the row data at or below 255 values per thread? Why would you choose more in practice?
  4. Run online softmax by hand on \((4, 1, 6)\) in chunks of sizes 2 and 1. Give \(m\) and \(\ell\) after each chunk, and the final probabilities.
  5. Your fused RMSNorm for \(M = 8{,}192\), \(N = 4{,}096\), BF16, takes 50 µs on an H100 SXM. What is its effective bandwidth and efficiency?
  6. Rewrite the vector-add kernel so it computes out = a * x + y with a scalar a. Which arguments change and does the byte count change?
Answerscheck
1. \(\lceil 10{,}000 / 1{,}024 \rceil = 10\) programs covering 10,240 slots, so 240 masked slots, all in the last program. 2. Eager: \((8MN + 4M) \times 2 = 536.9\) MB; fused: \(2MN \times 2 = 134.2\) MB; ratio 4.0. 3. \(16{,}384 / (32 \cdot w) \le 255\) needs \(w \ge 2.01\), so the smallest power of two is \(w = 4\) (128 values per thread). In practice choose 8 or 16: the real register count is higher than the row alone, and fewer registers per thread let more warps run per SM. 4. Chunk (4, 1): \(m = 4\), \(\ell = 1 + e^{-3} = 1.0498\). Chunk (6): \(m' = 6\), \(\ell' = 1.0498\, e^{-2} + 1 = 1.1421\). Probabilities: \(e^{-2}/1.1421 = 0.1185\), \(e^{-5}/1.1421 = 0.0059\), \(1/1.1421 = 0.8756\). 5. Minimum bytes \((2MN + N) \times 2 = 134.2\) MB; \(134.2\ \text{MB} / 50\ \mu s = 2.68\) TB/s; efficiency \(2.68 / 3.35 = 80\%\). 6. Add an argument a (a plain float, not a pointer) and store a * x + y. The bytes stay \(3n\) elements: the scalar is passed with the launch, so the extra multiply is free on a memory-bound kernel.

Key takeaways

  • A Triton kernel is the code of one program that works on a block of data. Launch \(\lceil n/B \rceil\) programs (86.1); each builds offsets and a mask (86.2) and uses block loads and stores.
  • The compiler handles threads, coalescing, shared memory, tensor cores and pipelining. You still choose the grid, BLOCK_SIZE, num_warps, num_stages and what to fuse.
  • Count bytes first. Eager softmax moves \(8MN + 4M\) elements, the fused kernel \(2MN\) (86.4); since \(t \ge\) bytes / bandwidth (86.5), fusion can give up to about 4×. RMSNorm: about 3.5×.
  • Rows that do not fit in one block use online softmax (86.6), the same trick behind FlashAttention. Accumulate in FP32; mask with values that do not change the reduction.
  • Registers per thread are at least \(B / (32 \cdot \texttt{num\_warps})\) (86.8); let triton.autotune measure configs per key, and benchmark against eager and torch.compile with effective bandwidth (86.9).
  • The Triton language is a kernel compiler; Triton Inference Server is NVIDIA’s model server. Different things.

Key formulas

\[ G = \left\lceil \tfrac{n}{B} \right\rceil, \quad \mathbf{o} = \mathrm{pid}\cdot B + (0, \dots, B-1), \quad \mathbf{m} = (\mathbf{o} \lt n) \]
\[ \text{softmax bytes: } 8MN + 4M \;\to\; 2MN, \qquad t \ge \frac{\text{bytes}}{\mathrm{BW}}, \qquad \mathrm{BW}_{\mathrm{eff}} = \frac{\text{min bytes}}{t} \]
\[ \ell' = \ell\, e^{m - m'} + \sum_{j} e^{x_j - m'}, \qquad \mathrm{RMSNorm}(\mathbf{x})_j = \frac{x_j\, w_j}{\sqrt{\tfrac1N \sum_k x_k^2 + \varepsilon}} \]

References for this session

  1. OpenAI and community (docs). Triton language documentation
  2. Tillet et al. (2019). Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations
  3. OpenAI and community (docs). Triton language documentation
  4. Tillet et al. (2019). Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations
  5. NVIDIA (code). Triton Inference Server
  6. NVIDIA (docs). CUDA C++ Programming Guide
  7. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
  8. Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
  9. Milakov & Gimelshein (2018). Online normalizer calculation for softmax
  10. Zhang & Sennrich (2019). Root Mean Square Layer Normalization
  11. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  12. Hsu, Dai, Kothapalli, Song, Tang, Zhu, Shimizu, Sahni, Ning, Chen (2024). Liger Kernel: Efficient Triton Kernels for LLM Training
  13. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  14. Triton developers (docs). Triton tutorial: Fused Softmax