Visual tokenizers for generation
An LLM can only predict the next item from a fixed list of integers. To make it draw pictures, we need a visual tokenizer: a network that turns an image into a short grid of integers from a learned codebook, and turns such a grid back into an image. This session builds VQ-VAE and VQGAN step by step, shows how to train through a non-differentiable lookup, how codebooks collapse and how to fix it, and why understanding models and generation models want different kinds of visual tokens.
What you will learn
- How vector quantization turns each encoder output vector into one integer (a token), and how to count tokens and bits for any image or video tokenizer
- The VQ-VAE loss: reconstruction, codebook and commitment terms, the straight-through gradient, and the EMA codebook update
- Codebook collapse: why most codes can die, how to measure it (usage and perplexity), and the fixes used in practice (data init, dead-code restart, low-dimensional normalized codes)
- VQGAN: why a perceptual loss, a patch discriminator and an adaptive weight give sharp images at 16 times down-sampling, and the reconstruction numbers behind it
- Newer quantizers without a learned lookup: finite scalar quantization (FSQ) and lookup-free quantization (LFQ)
- Discrete versus continuous visual tokens: why understanding models use SigLIP-style features while token-based generators use VQ codes, with the Janus ablation as evidence
- How shared token vocabularies make any-to-any models (Chameleon, Emu3) possible, and what they cost in sequence length
Why this matters
A language model does one thing: given a list of integers, it predicts the next integer from a fixed vocabulary. If we want the same model to draw an image, we must first turn images into lists of integers, and later turn integers back into images. The network that does this is a visual tokenizer.
For text, a tokenizer is a fixed rule (BPE, see session 8). For images there is no natural alphabet. A 256 × 256 RGB image has 196,608 numbers, each with 256 possible values. One token per pixel value would give an absurdly long sequence. So we learn the alphabet: an encoder squeezes each small patch of the image into one vector, and that vector is replaced by the nearest entry of a learned table of vectors, the codebook. The index of that entry is the token. The best-known version, VQGAN, turns a 256 × 256 image into a 16 × 16 grid: 256 tokens, each an integer below 1,024 or 16,384. [Certain] A decoder turns the grid back into an image.
This is how DALL·E (first version), Parti-style and LlamaGen-style image generators, Chameleon, Emu3 and the generation path of Janus work. [Certain for DALL·E, LlamaGen, Chameleon, Emu3 and Janus; Likely for Parti] The tokens become new words in the LLM vocabulary, and image generation becomes next-token prediction. Chameleon, for example, has one vocabulary of 65,536 entries, and 8,192 of them are image codes. [Certain]
In session 223 the latent was a continuous vector, good for diffusion. This session makes the latent discrete. The next session, session 225, goes back to continuous latents; in session 232 you will see both kinds of tokens inside one model.
From a continuous latent to a codebook
Start with an ordinary convolutional autoencoder. The encoder \(E\) takes an image of size \(H \times W \times 3\) and returns a grid of vectors of size \(h \times w \times D\), with \(h = H/f\) and \(w = W/f\). The number \(f\) is the down-sampling factor: each grid cell describes an \(f \times f\) patch of pixels. Call one of these vectors \(\mathbf{z}_e\).
Now add a codebook: a table of \(K\) learned vectors \(\mathbf{e}_1, \dots, \mathbf{e}_K\), each of size \(D\). Vector quantization replaces every encoder vector by its nearest code:
- Read the symbols
- \(\mathbf{z}_e \in \R^D\)
- the encoder output at one grid position (one patch)
- \(\mathbf{e}_j \in \R^D\)
- code number \(j\) in the codebook; the codebook is a \(K \times D\) matrix
- \(k^\star\)
- the index of the nearest code: this integer is the visual token
- \(\mathbf{z}_q\)
- the quantized vector, which the decoder receives instead of \(\mathbf{z}_e\)
The decoder \(G\) never sees \(\mathbf{z}_e\), only the codes. So the whole image is described by \(h \times w\) integers. To store or generate an image, we only need those integers; the codebook and the decoder are fixed and shared.
Figure 1 shows quantization in two dimensions (\(D = 2\)). The codebook splits the plane into regions; every point inside a region becomes the same token. These regions are called Voronoi cells.
How to read Figure 1
With \(K = 4\) the codes are the corners of the unit square, as in the worked example below. Move \(\mathbf{z}_e\) across a border: the token jumps from one integer to another, while \(\mathbf{z}_e\) only moved a little. This jump is why quantization has no useful gradient. Increase \(K\): the cells get smaller, the quantization error (the length of the line) usually shrinks, and each token carries \(\log_2 K\) bits. A real tokenizer does the same in \(D = 8\) to \(256\) dimensions with \(K\) from 1,024 to 262,144.
Figure 1 shows one vector. Figure 2 runs the whole tokenizer on a real (tiny) picture: a 16 × 16 image of a tree, a house, a cloud and the sun. To keep every number visible, the encoder is a toy: it averages the \(f \times f\) pixels of each patch, so \(\mathbf{z}_e\) is a color with \(D = 3\) numbers (red, green, blue). The codebook is a fixed palette of up to 8 colors, and the decoder paints each patch with its code color. A trained tokenizer learns all three parts, but the loop is exactly the same: encode a patch, measure the distance to every code, keep the index of the nearest one.
How to read Figure 2
Press Step three times. Patch 1 is pure sky: \(\mathbf{z}_e = (0.432, 0.678, 0.944)\), its distance to the sky code is 0.001 and to every other code at least 0.349, so token 0 goes into the grid. Press Next patch until patch 4 (top right). It is half sun and half sky, \(\mathbf{z}_e = (0.836, 0.794, 0.396)\): the sun code is at distance 0.054 but the wall code at 0.049, so the token is 6 and the sun is painted as a wall. That is quantization error. At the end, with \(K = 8\) and \(f = 4\), the 16 tokens use 5 different codes and the reconstruction error (mean squared error per pixel value) is 0.0475. Choose \(f = 2\): 64 tokens, error 0.0362. Choose \(f = 8\): only 4 tokens, error 0.0942; the house and the tree disappear. Lower \(K\) to 2: only sky and grass remain.
Counting tokens and bits
Two numbers describe the size of a tokenized image: how many tokens, and how many bits each token carries.
\(N\) is the sequence length the LLM must handle: attention cost grows with \(N^2\) and generation time grows with \(N\), because each token costs one forward pass. \(N \log_2 K\) is an upper limit on the information the tokens can hold. Everything the decoder draws beyond that must be invented by the decoder.
VQ-VAE on ImageNet 128 × 128. \(f = 4\), so a 32 × 32 grid, \(K = 512\) (9 bits). Raw image: \(128 \cdot 128 \cdot 3 \cdot 8 = 393{,}216\) bits. Tokens: \(1{,}024 \cdot 9 = 9{,}216\) bits. Ratio about 42.7, the number the VQ-VAE paper reports (42.6). [Certain]
VQGAN on ImageNet 256 × 256. \(f = 16\), a 16 × 16 grid, \(K = 1{,}024\) (10 bits). Raw: \(1{,}572{,}864\) bits. Tokens: \(256 \cdot 10 = 2{,}560\) bits = 320 bytes. Ratio 614.
Chameleon at 512 × 512. 1,024 tokens from a codebook of 8,192 (13 bits): 13,312 bits, ratio about 473. 1,024 tokens per image means 32 × 32, so \(f = 16\). [Certain for 1,024 tokens and 8,192 codes]
DALL·E (2021). 256 × 256 to a 32 × 32 grid with 8,192 codes. The paper says this “reduces the context size of the transformer by a factor of 192”: \(256 \cdot 256 \cdot 3 / 1{,}024 = 192\). [Certain]
Figure 3 lets you do this arithmetic for any image or video tokenizer.
How to read Figure 3
Click VQGAN: 256 tokens, 2,560 bits, ratio 614. Now click Emu3 (\(f = 8\), \(K = 32{,}768\)): a 512 × 512 image is 4,096 tokens, the whole context of a small LLM. Halving \(f\) multiplies \(N\) by 4 and attention cost by 16. Adding bits per token is cheap for sequence length but makes the output softmax and the embedding table larger. Slide the frames: a 32-frame clip at Emu3 settings is 32,768 tokens, which is why video tokenizers compress time too.
Training through the argmin: the straight-through estimator
We want to train the encoder, decoder and codebook together, with a reconstruction loss such as \(\lVert \mathbf{x} - G(\mathbf{z}_q) \rVert^2\). Backpropagation reaches \(\mathbf{z}_q\) without trouble. But \(\mathbf{z}_q\) is chosen by an argmin, a step function of \(\mathbf{z}_e\): its derivative is zero almost everywhere and undefined on the borders (you saw the jumps in Figure 1). The encoder would get no learning signal.
VQ-VAE uses the straight-through estimator (STE): in the backward pass, pretend quantization is the identity, and copy the gradient from \(\mathbf{z}_q\) to \(\mathbf{z}_e\) unchanged. In code this is one line with a stop-gradient \(\text{sg}[\cdot]\) (identity in the forward pass, zero gradient in the backward pass):
Why does this work at all? \(\mathbf{z}_e\) and \(\mathbf{z}_q\) live in the same \(D\)-dimensional space and are close to each other. The gradient at \(\mathbf{z}_q\) says how the decoder input should move to lower the loss; moving \(\mathbf{z}_e\) in the same direction is a reasonable guess. If the move crosses a border, the next forward pass picks a different code. The guess is biased, but it is cheap and works well in practice.
The VQ-VAE loss
The VQ-VAE loss (van den Oord et al., 2017) has three terms, each training a different part:
- Read the symbols
- reconstruction
- trains the decoder, and the encoder through the STE (224.3). The paper writes it as \(-\log p(\mathbf{x} \mid \mathbf{z}_q)\); with a Gaussian decoder this is the squared error up to a constant
- codebook
- moves the chosen code \(\mathbf{e}_{k^\star}\) toward the encoder output; the encoder is frozen here by \(\text{sg}\)
- commitment
- pulls the encoder output toward its chosen code, so the encoder “commits” to a code and its outputs do not drift or grow without limit; the code is frozen here
- \(\beta\)
- weight of the commitment term; the paper uses \(\beta = 0.25\) and reports that results did not change for \(\beta\) from 0.1 to 2.0 [Certain]
In practice the loss is averaged over all \(h \times w\) grid positions. Why the two separate terms instead of one \(\lVert \mathbf{z}_e - \mathbf{e} \rVert^2\)? Because the two sides should move at different speeds. The codebook should follow the encoder quickly (weight 1). The encoder should be only gently held near the codes (weight \(\beta\)), since its main job is to serve the reconstruction.
The VAE view. VQ-VAE is a VAE whose posterior \(q(z = k \mid \mathbf{x})\) is one-hot on the nearest code, with a uniform prior over \(K\) codes. Then the KL term of the ELBO from session 223 is a constant:
So training ignores it. The prior is not learned during tokenizer training. A second model, trained afterwards on the token grids, becomes the real prior: a PixelCNN in the original paper, a transformer in VQGAN and in every LLM-based generator.
Codebook: \(\mathbf{e}_0 = (0,0)\), \(\mathbf{e}_1 = (1,0)\), \(\mathbf{e}_2 = (0,1)\), \(\mathbf{e}_3 = (1,1)\). Encoder output \(\mathbf{z}_e = (0.9, 0.2)\). This is the default state of Figure 1.
Quantize (224.1). Squared distances: \(0.85,\ 0.05,\ 1.45,\ 0.65\). Nearest is code 1, so the token is 1 and \(\mathbf{z}_q = (1, 0)\).
Loss terms (224.4). Codebook term \(= 0.05\). Commitment term \(= 0.25 \times 0.05 = 0.0125\).
Gradients. For the code: \(\partial / \partial \mathbf{e}_1\) of \(\lVert \mathbf{z}_e - \mathbf{e}_1 \rVert^2\) is \(2(\mathbf{e}_1 - \mathbf{z}_e) = (0.2, -0.4)\). One SGD step with learning rate 0.1 moves \(\mathbf{e}_1\) to \((0.98, 0.04)\), and the squared distance falls from 0.05 to 0.032. For the encoder: suppose the decoder sends back \(\partial \mathcal{L}/\partial \mathbf{z}_q = (0.3, -0.1)\). The STE copies it to \(\mathbf{z}_e\); the commitment term adds \(2\beta(\mathbf{z}_e - \mathbf{e}_1) = (-0.05, 0.1)\). Total gradient on \(\mathbf{z}_e\): \((0.25, 0.0)\).
Figure 4 repeats this step 30 times, so you can watch the three loss terms and the straight-through gradient work together. To keep it small, the encoder output \(\mathbf{z}_e\) is a free parameter (think of an encoder that is only a bias), and the decoder is the identity, so the reconstruction loss is \(\lVert \mathbf{x} - \tilde{\mathbf{z}}_q \rVert^2\) for a target point \(\mathbf{x}\). With the target \(\mathbf{x} = (0.85, 0.05)\), the first step is exactly the worked example above: \(\partial \mathcal{L}/\partial \mathbf{z}_q = 2(\mathbf{z}_q - \mathbf{x}) = (0.3, -0.1)\).
zq = zq + (zq - z).detach().How to read Figure 4
Press Step through the first iteration: you will see the numbers of the worked example (distances 0.85, 0.05, 1.45, 0.65; token 1; gradient (0.25, 0.0) on \(\mathbf{z}_e\); code 1 moves to (0.98, 0.04)). Press Play: code 1 and \(\mathbf{z}_e\) move together toward the target, and at iteration 30 the reconstruction loss is about 0.0001. Now pick target (0.3, 0.9), which lies in the cell of code 2. With the STE, \(\mathbf{z}_e\) is pushed across two borders: over the first five iterations the token goes 1, 1, 1, 3, 2 and the reconstruction loss goes 1.30, 1.20, 1.04, 0.50, 0.10. The big drops come from switching to a better code, not from moving a code. Switch to “no gradient through argmin”: \(\mathbf{z}_e\) and code 1 only pull on each other, the token stays 1 forever, and the reconstruction loss gets stuck at 0.93. Go back to the straight-through mode and raise \(\beta\) to 2: the commitment term now holds \(\mathbf{z}_e\) so close to code 1 that it never leaves cell 1; instead code 1 and \(\mathbf{z}_e\) travel together toward the target (reconstruction loss 0.0152 at iteration 30).
The EMA codebook update
The codebook term is just k-means written as a loss: the best \(\mathbf{e}_k\) is the mean of the encoder outputs assigned to it. The appendix of the VQ-VAE paper proposes to compute this mean directly, as an exponential moving average (EMA) over mini-batches, instead of using gradients:
- Read the symbols
- \(n_k\)
- how many encoder outputs in this batch chose code \(k\)
- \(N_k, \mathbf{m}_k\)
- running count and running sum of the vectors assigned to code \(k\)
- \(\gamma\)
- decay; the paper found \(\gamma = 0.99\) to work well [Certain]
With EMA, the codebook term is dropped from (224.4) and the loss keeps only reconstruction and commitment. Example: a code with \(N_k = 10\) and \(\mathbf{m}_k = (10, 0)\) sits at \((1, 0)\). A batch assigns two vectors with sum \((1.8, 0.4)\) to it. With \(\gamma = 0.99\): \(N_k = 9.92\), \(\mathbf{m}_k = (9.918, 0.004)\), and the code moves to \((0.99980, 0.00040)\): a small, smooth step toward the new data.
- input: batch of images \(\mathbf{x}\), encoder \(E\), decoder \(G\), codebook \(\{\mathbf{e}_k\}\) with EMA states \(N_k, \mathbf{m}_k\), weights \(\beta, \gamma\)
- \(\mathbf{z}_e \leftarrow E(\mathbf{x})\) // grid of \(h \times w\) vectors per image
- for each grid vector: \(k^\star \leftarrow \argmin_j \lVert \mathbf{z}_e - \mathbf{e}_j \rVert^2\), \(\mathbf{z}_q \leftarrow \mathbf{e}_{k^\star}\) // formula (224.1); the \(k^\star\) are the tokens
- update \(N_k, \mathbf{m}_k, \mathbf{e}_k\) with the assignments // formula (224.6), no gradient
- optionally: move codes with tiny \(N_k\) (“dead codes”) onto random current encoder outputs
- \(\tilde{\mathbf{z}}_q \leftarrow \mathbf{z}_e + \text{sg}[\mathbf{z}_q - \mathbf{z}_e]\) // straight-through, formula (224.3)
- \(\mathcal{L} \leftarrow \lVert \mathbf{x} - G(\tilde{\mathbf{z}}_q)\rVert^2 + \beta \lVert \mathbf{z}_e - \text{sg}[\mathbf{z}_q]\rVert^2\)
- backpropagate \(\mathcal{L}\) and take an optimizer step on \(E\) and \(G\)
Codebook collapse
The most common failure of VQ training is that most codes are never chosen. A code that wins no encoder output gets no codebook gradient and no EMA update, so it never moves, and it never wins in the future either. It is dead. If 900 of 1,024 codes die, the tokenizer really has about 124 tokens, and reconstruction suffers. This is called codebook collapse (or low codebook usage).
Two numbers measure it. Usage is the fraction of codes chosen at least once on a large set of images. Perplexity is the exponential of the entropy of the code frequencies \(p_k\):
Perplexity is the “effective number of codes”: if two codes are used half the time each, PPL = 2; if all \(K\) are used equally, PPL = \(K\). It is the same perplexity you met for language models in session 6, applied to code frequencies.
Why do codes die? Usually because of a bad start: the codes are initialized randomly, far from where the encoder outputs actually are. The few codes that happen to be closest take all the data and follow it; the rest are stranded. Figure 5 shows this with a frozen encoder: 400 encoder outputs in 5 clusters, and a codebook that starts in a corner.
How to read Figure 5
Choose “Gradient” and press “20 steps” four times (80 steps). With \(K = 8\), only 4 codes reach the data and split the five clusters between them; the other 4 stay in the corner forever. Usage is 4 of 8, perplexity 2.91 and the mean squared quantization error 0.593, and more steps do not change this. EMA alone behaves almost the same (5 of 8 codes, error 0.584 after 80 steps): it also only updates codes that win. With restarts, all 8 codes are used and the error falls to 0.085 after 80 steps. Each restarted code is dropped onto a real encoder output, starts winning points, and the codebook spreads over all five clusters. Press Play to watch the steps one by one: the codes that win points move into the clusters, and with restarts a dead code jumps from the corner straight onto a data point. Raise \(K\) above 5: extra codes split clusters, and the error keeps falling.
The fixes used in practice:
- Initialize from data: run k-means on the first batches of encoder outputs and use the centroids as the initial codes.
- Restart dead codes: as in Figure 5. Many open-source VQ implementations do this. [Likely]
- Low-dimensional, \(\ell_2\)-normalized codes (ViT-VQGAN, then LlamaGen): project the encoder output to a small \(D\) such as 8 or 32, normalize both it and the codes to unit length, and look up by cosine similarity. In a low-dimensional space on a sphere, codes are crowded together and almost every code is near some data. [Certain that both papers use it]
- An entropy penalty that rewards using many codes on average (MaskGIT, MAGVIT-v2, formula 224.11 below).
The effect of the code dimension is large. LlamaGen’s ablation (codebook of 16,384, \(f = 16\), ImageNet 256 × 256) is shown in Figure 6.
VQGAN: sharp codes with perceptual and adversarial losses
A VQ-VAE trained with squared error has the same problem as the VAE of session 223: when it is unsure about fine detail, the squared error is lowest for the average of all plausible details, which looks blurry. At \(f = 4\) this is tolerable. But a transformer over a 32 × 32 grid (1,024 tokens) was expensive in 2020, and the VQGAN authors wanted \(f = 16\): 256 tokens. Each token must then summarize a 16 × 16 patch, and a squared-error decoder produces mush.
VQGAN (Esser, Rombach and Ommer, 2020) keeps the quantizer of VQ-VAE and changes the loss:
- The pixel loss is replaced by a perceptual loss (LPIPS): compare images by the features of a pretrained CNN rather than by raw pixels. Two textures that differ pixel by pixel but look alike get a small loss.
- A patch-based discriminator \(D\) looks at overlapping patches and says “real” or “reconstructed” for each. The decoder is rewarded for fooling it, so it must draw one sharp, plausible detail instead of the blurry average.
- A single attention layer at the lowest resolution lets each code use context from the whole image.
Here \(\mathcal{L}_{\text{VQ}}\) is (224.4) with the perceptual reconstruction loss, \(\mathcal{Z}\) is the codebook and \(\hat{\mathbf{x}} = G(\mathbf{z}_q)\). The weight \(\lambda\) is not a fixed number. It is set at every step so that the two losses push the decoder with similar strength:
- Read the symbols
- \(G_L\)
- the weights of the last layer of the decoder
- \(\nabla_{G_L}\mathcal{L}\)
- gradient of a loss with respect to those weights; its norm says how hard that loss pulls on the output
- \(\delta\)
- a tiny constant to avoid division by zero
If the GAN gradient is weak, \(\lambda\) grows; if it is strong, \(\lambda\) shrinks. This removes one hard-to-tune hyper-parameter. In practice the discriminator is usually switched on only after the autoencoder has trained for a while (LlamaGen enables it after 20k iterations, with adversarial weight 0.5 and commitment weight 0.25). [Certain for LlamaGen]
How much does this help? Reconstruction FID on the ImageNet validation set, from the VQGAN paper (Table 5) [Certain]:
| Model | Latent grid | Codebook size | rFID (val) |
|---|---|---|---|
| DALL·E dVAE (no perceptual or GAN loss) | 32 × 32 | 8,192 | 32.01 |
| VQGAN | 16 × 16 | 1,024 | 7.94 |
| VQGAN | 16 × 16 | 16,384 | 4.98 |
| VQGAN (Gumbel-softmax) | 32 × 32 | 8,192 | 1.49 |
With 4 times fewer tokens than DALL·E’s dVAE (256 vs 1,024), VQGAN reconstructs far better. And the table shows the two knobs of every tokenizer: more codes (1,024 → 16,384) and more tokens (16 × 16 → 32 × 32) both improve reconstruction. LlamaGen (2024) later reached rFID 2.19 at 16 × 16 with 16,384 codes of dimension 8, and 0.59 at 32 × 32. [Certain]
Stage two: generating tokens
Once the tokenizer is trained, it is frozen. Every training image becomes a grid of integers, flattened into a sequence \(s_1, \dots, s_N\) (usually row by row, the raster order). A transformer then learns the distribution of these sequences exactly like a language model learns text:
Here \(c\) is the condition: a class label, a text prompt, or any earlier tokens of a conversation. The loss is the ordinary cross-entropy over a vocabulary of \(K\) image codes. Everything from LLM training carries over: the same transformer, the same sampling with temperature, top-k and top-p, the same KV cache, and classifier-free guidance on the logits. [Certain that LlamaGen uses a plain Llama-style transformer]
- input: condition \(c\) (prompt tokens), trained transformer \(T\), frozen codebook and decoder \(G\), grid size \(h \times w\)
- \(\text{seq} \leftarrow [c, \texttt{<boi>}]\) // begin-of-image token
- for \(i = 1\) to \(N = h \cdot w\):
- \(\boldsymbol{\ell} \leftarrow T(\text{seq})\) restricted to the \(K\) image codes // one forward pass with KV cache
- \(s_i \sim \softmax(\boldsymbol{\ell} / \tau)\), with top-k or top-p // formula (224.10)
- append \(s_i\) to seq
- reshape \(s_1, \dots, s_N\) to an \(h \times w\) grid; look up codes \(\mathbf{e}_{s_i}\)
- return \(G(\text{grid of codes})\) // the decoder draws the pixels
The cost is \(N\) sequential forward passes: 256 for a VQGAN grid, 4,096 for Emu3 at 512 × 512. This is why later work reduces the number of sequential steps: MaskGIT predicts many masked tokens in parallel over a few rounds; VAR predicts whole coarse-to-fine scales at once; residual quantization (RQ-VAE) stacks several codes per position so the grid can be coarser; TiTok uses a 1-D sequence of only 32 tokens for a 256 × 256 image. [Certain for the ideas; details in the papers]
Quantizers without a learned lookup: FSQ and LFQ
Codebook collapse, commitment losses and restarts all come from one design: a learned table looked up by nearest neighbor. Two newer methods remove the table.
Finite scalar quantization (FSQ) (Mentzer et al., 2023). Project the encoder output to a few channels \(d\) (for example 4 to 6). Squash each channel into a fixed range and round it to one of \(L_i\) integer levels:
(This form is for odd \(L_i\); even \(L_i\) needs a half-step shift, as in the code below.) The rounding uses the STE. The “codebook” is the grid of all level combinations, so it is never learned and no code can die. With levels \([8, 5, 5, 5]\), \(|\mathcal{C}| = 1{,}000\), close to \(2^{10}\); with \([8, 8, 8, 5, 5, 5]\), \(64{,}000\), close to \(2^{16}\). These are the paper’s recommended level sets. [Certain] The token is the integer whose mixed-radix digits are the levels.
Lookup-free quantization (LFQ) (MAGVIT-v2, Yu et al., 2023) goes further: \(d = \log_2 K\) channels, each quantized to \(\{-1, +1\}\) by its sign. The token is the binary number formed by the signs:
The entropy term asks each single prediction to be confident (low first term) while the average over the batch uses all codes (high second term). MAGVIT-v2 uses \(K = 2^{18} = 262{,}144\). [Certain] Its paper reports that with ordinary VQ, generation quality got worse as the vocabulary grew, while with LFQ both reconstruction and generation kept improving up to the largest vocabularies they tried. [Certain, their Figure 1, at 128 × 128]
Discrete vs continuous visual tokens
Not every “visual token” is a codebook index. There are two families, and they serve different jobs.
| Continuous tokens | Discrete tokens | |
|---|---|---|
| What they are | Real-valued vectors: ViT / SigLIP patch features (understanding), or VAE latents (diffusion) | Integers from a codebook (VQ, FSQ, LFQ) |
| Trained for | Semantics (contrastive image–text loss), or reconstruction with a smooth latent | Reconstruction of pixels, under a hard bit budget \(N \log_2 K\) |
| Into the LLM | Through a projector, as input embeddings (session 179) | As new vocabulary entries, through the ordinary embedding table |
| Out of the LLM | Needs a regression or diffusion head (Transfusion, MAR) | Ordinary softmax and cross-entropy, ordinary sampling |
| Information loss | Small (no rounding) | Hard limit; fine text and small faces suffer |
For understanding, continuous semantic features win. A VQ tokenizer is trained to redraw pixels, so its codes describe colors and textures, not “this is a stop sign”. The Janus ablation measures this with the same 1.3B LLM [Certain, Janus Table 5]:
| Visual input for understanding | POPE ↑ | MMBench ↑ | SEED ↑ | MMMU ↑ |
|---|---|---|---|---|
| VQ tokenizer for both tasks (Chameleon-style) | 60.1 | 35.0 | 34.9 | 24.7 |
| Semantic tokenizer distilled from SigLIP, for both tasks | 82.4 | 52.7 | 54.9 | 26.6 |
| SigLIP for understanding + VQ for generation (Janus) | 87.0 | 69.4 | 63.7 | 30.5 |
Janus therefore decouples the two paths: SigLIP features go in for understanding, and LlamaGen’s VQ tokenizer (16,384 codes, \(f = 16\)) is used for generation. Chameleon, which uses one VQ tokenizer for both, says in its paper that its tokenizer is weak at reconstructing images with a lot of text, which caps its OCR ability. [Certain]
For generation, both work. Discrete tokens let you reuse the LLM machinery unchanged. Continuous latents keep more detail but need a different output head: diffusion in latent space (Stable Diffusion, session 228), diffusion inside the same transformer (Transfusion), or a small per-token diffusion head (MAR, “autoregressive image generation without vector quantization”). Which is better is still an open research question. [Likely]
Why this matters for any-to-any models
An any-to-any model reads and writes several modalities in one sequence. With discrete tokenizers this becomes simple bookkeeping: every modality gets its own range of integer ids in one shared vocabulary, plus special tokens such as begin-of-image and end-of-image. Then one transformer with one cross-entropy loss handles all of them.
- Chameleon (Meta, 2024): a 512 × 512 image is 1,024 tokens from 8,192 codes, inside a 65,536-entry BPE vocabulary. Text and images are freely mixed in one sequence (“early fusion”). [Certain]
- Emu3 (BAAI, 2024): one tokenizer for images and video, based on SBER-MoVQGAN, 32,768 codes, 8 × 8 spatial and 4 × temporal compression. A 512 × 512 image, or a 4-frame 512 × 512 clip, becomes 4,096 tokens. Text, images and video are all trained with next-token prediction only. [Certain]
- Audio follows the same pattern with neural audio codecs that turn sound into discrete tokens (session 184).
The price is sequence length. A 10-second video at 24 frames per second with Emu3-style compression is 60 latent frames of 4,096 tokens: 245,760 tokens before any text. That is why video tokenizers compress time and why session 183 is its own topic.
A visual tokenizer in PyTorch
The code below implements the quantizer of (224.1), the STE (224.3), the commitment term of (224.4), the EMA update (224.6) with optional dead-code restarts, and perplexity (224.7). It trains a tiny VQ autoencoder on synthetic 16 × 16 images twice, without and with restarts, and then checks FSQ (224.11). It runs on a CPU in about 20 seconds and prints the same numbers every time.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) class VectorQuantizerEMA(nn.Module): """Nearest-code quantizer with EMA codebook, straight-through gradient and dead-code restart.""" def __init__(self, K=64, D=16, beta=0.25, gamma=0.99, restart=False): super().__init__() self.K, self.beta, self.gamma, self.restart = K, beta, gamma, restart e = torch.randn(K, D) self.register_buffer("codebook", e) # (K, D): updated by EMA, not by the optimizer self.register_buffer("N", torch.ones(K)) # EMA of how many vectors chose each code self.register_buffer("m", e.clone()) # EMA of the sum of those vectors def forward(self, z): # z: (B, D, H, W) encoder output B, D, H, W = z.shape flat = z.permute(0, 2, 3, 1).reshape(-1, D) # one row per grid position d = (flat.pow(2).sum(1, keepdim=True) # squared distance to every code (224.1) - 2 * flat @ self.codebook.t() + self.codebook.pow(2).sum(1)) idx = d.argmin(1) # the tokens zq = self.codebook[idx].view(B, H, W, D).permute(0, 3, 1, 2) if self.training: with torch.no_grad(): # EMA update (224.6) onehot = F.one_hot(idx, self.K).type(flat.dtype) self.N.mul_(self.gamma).add_(onehot.sum(0), alpha=1 - self.gamma) self.m.mul_(self.gamma).add_(onehot.t() @ flat, alpha=1 - self.gamma) self.codebook.copy_(self.m / self.N.clamp(min=1e-5).unsqueeze(1)) if self.restart: # move dead codes onto random encoder outputs dead = self.N < 0.03 if dead.any(): pick = flat[torch.randint(0, flat.shape[0], (int(dead.sum()),))] self.codebook[dead] = pick; self.m[dead] = pick; self.N[dead] = 1.0 commit = self.beta * F.mse_loss(z, zq.detach()) # commitment term of (224.4) zq = z + (zq - z).detach() # straight-through estimator (224.3) p = torch.bincount(idx, minlength=self.K).float() / idx.numel() ppl = torch.exp(-(p * torch.log(p + 1e-10)).sum()) # perplexity (224.7) return zq, idx.view(B, H, W), commit, ppl def fsq(z, levels=(8, 5, 5, 5)): # finite scalar quantization (224.11) L = torch.tensor(levels, dtype=z.dtype) half_l = (L - 1) * (1 - 1e-3) / 2 offset = torch.where(L % 2 == 1, 0.0, 0.5) # even L needs a half-step shift bounded = torch.tanh(z + torch.tan(offset / half_l)) * half_l - offset q = bounded + (bounded.round() - bounded).detach() # round with straight-through gradient digits = (q + L // 2).long() # each channel -> 0 .. L_i - 1 basis = torch.cumprod(torch.cat([torch.ones(1), L[:-1]]), 0).long() return q / (L // 2), (digits * basis).sum(-1) # code in [-1, 1] and its integer token # 1) the straight-through gradient is the identity vq = VectorQuantizerEMA() z = torch.randn(2, 16, 4, 4, requires_grad=True) zq, idx, commit, ppl = vq(z); zq.sum().backward() print("STE gradient is all ones:", torch.allclose(z.grad, torch.ones_like(z))) # 2) a tiny VQ autoencoder on 16x16 toy images (f = 4, a 4x4 token grid) def toy_batch(n, g): x = torch.zeros(n, 3, 16, 16) for i in range(n): for _ in range(3): # three random colored squares r, c = torch.randint(0, 12, (2,), generator=g).tolist() x[i, :, r:r + 4, c:c + 4] = torch.rand(3, 1, 1, generator=g) return x def train(restart): torch.manual_seed(0) enc = nn.Sequential(nn.Conv2d(3, 32, 4, 2, 1), nn.ReLU(), nn.Conv2d(32, 16, 4, 2, 1)) dec = nn.Sequential(nn.ConvTranspose2d(16, 32, 4, 2, 1), nn.ReLU(), nn.ConvTranspose2d(32, 3, 4, 2, 1)) vq = VectorQuantizerEMA(K=64, D=16, restart=restart) opt = torch.optim.Adam([*enc.parameters(), *dec.parameters()], lr=2e-3) g = torch.Generator().manual_seed(1) for step in range(400): x = toy_batch(32, g) zq, idx, commit, ppl = vq(enc(x)) rec = F.mse_loss(dec(zq), x) loss = rec + commit # (224.4); the codebook term is replaced by EMA opt.zero_grad(); loss.backward(); opt.step() used = int((torch.bincount(idx.flatten(), minlength=64) > 0).sum()) print(f"restart={restart}: recon MSE {rec.item():.4f}, perplexity {ppl.item():.1f}, codes used in last batch {used}/64") train(restart=False) # codes start far from the encoder outputs: most of them die train(restart=True) # dead codes are moved onto real encoder outputs # 3) FSQ: 4 channels with levels (8, 5, 5, 5) give an implicit codebook of 1000 codes code, tok = fsq(torch.tensor([[0.3, -2.0, 0.0, 5.0]])) print("FSQ code", code.tolist(), "token", tok.item()) allz = torch.randn(200000, 4) * 2 print("distinct FSQ tokens seen:", fsq(allz)[1].unique().numel())
Output (PyTorch 2.x, CPU):
STE gradient is all ones: True restart=False: recon MSE 0.0324, perplexity 1.8, codes used in last batch 2/64 restart=True: recon MSE 0.0218, perplexity 36.1, codes used in last batch 55/64 FSQ code [[0.25, -1.0, 0.0, 1.0]] token 885 distinct FSQ tokens seen: 1000
Without restarts the codebook collapses to 2 codes (perplexity 1.8 out of 64): the toy model is the real-world failure in miniature. With restarts it uses 55 codes in a batch and its reconstruction error is about 33% lower. The FSQ token 885 is the mixed-radix number \(5 + 0 \cdot 8 + 2 \cdot 40 + 4 \cdot 200\).
Where this is used
| Application | Tokenizer | Why tokens |
|---|---|---|
| Class- and text-to-image with an LLM-style model (LlamaGen, DALL·E 1; Parti [Likely]) | VQGAN-style, 16 × 16 or 32 × 32, codebooks of 8,192 to 16,384 | Reuse the LLM architecture, its scaling recipes and its serving stack |
| Native multimodal chat that can answer with images (Chameleon, Emu3) | One shared vocabulary of text and image codes | One model, one loss, any order of text and images |
| Unified understanding + generation (Janus) | SigLIP in, VQ out | Each path gets the token type it needs |
| Video generation and video prediction | Spatio-temporal tokenizers (MAGVIT-v2, Emu3: 4× in time) | A video is a 3-D token grid; next-token or masked-token prediction generates it (session 231) |
| Video compression | MAGVIT-v2 tokens as a codec | The paper reports user-study quality better than HEVC and on par with VVC at similar bit rates [Certain] |
| Video near-duplicate detection and retrieval | Histograms or n-grams of visual tokens per shot | Integers are cheap to index and compare, like text search; this is an idea, not a standard method [Speculation] |
| Masked image modeling for pretraining (BEiT-style) | A frozen dVAE or VQ tokenizer gives the prediction targets | Cross-entropy on discrete targets instead of pixel regression [Likely] |
Common mistakes
- Not monitoring codebook usage. The reconstruction loss can look fine while 90% of the codes are dead. Log usage and perplexity (224.7) from the first step.
- Forgetting the stop-gradients. Writing \(\lVert \mathbf{z}_e - \mathbf{e} \rVert^2\) with no \(\text{sg}\) lets encoder and codes chase each other with the same weight; writing the STE as
zq = zq + (zq - z).detach()instead ofz + (zq - z).detach()gives the encoder no gradient at all. - Expecting the reconstruction loss to train the codebook. Under the STE it does not; you need the codebook term or EMA.
- Mixing up the two loss weights. The codebook term has weight 1, the commitment term weight \(\beta \approx 0.25\). Swapping them makes the encoder chase a slow codebook.
- Huge code dimension with a huge codebook. \(D = 256\) with 16,384 codes gave 0.29% usage in LlamaGen. Use low-dimensional normalized codes, FSQ, or LFQ.
- Judging a tokenizer only by PSNR. Generation quality depends on rFID, on usage, and on how easy the tokens are to predict; a bigger vocabulary can reconstruct better but generate worse (MAGVIT-v2, Figure 1, for plain VQ).
- Using reconstruction tokens for understanding. VQ codes lack semantics; a VLM fed only VQ tokens loses a lot on understanding benchmarks (Janus ablation).
- Ignoring the sequence length. Halving \(f\) multiplies tokens by 4 and attention cost by 16; at video scale this dominates everything (Figure 3).
Interview questions
What is a visual tokenizer, and why does an LLM-based image generator need one?basic
How does VQ-VAE backpropagate through the argmin?core
zq = z + (zq - z).detach(). It works because \(\mathbf{z}_e\) and \(\mathbf{z}_q\) are close and in the same space, so the decoder’s gradient at \(\mathbf{z}_q\) is a good direction for \(\mathbf{z}_e\). The estimator is biased; alternatives are Gumbel-softmax relaxations (used by DALL·E’s dVAE) or FSQ/LFQ with rounding plus STE.Write the VQ-VAE loss and say what each term trains.core
What is codebook collapse, how do you detect it, and how do you fix it?core
Why does VQGAN add a perceptual loss and a GAN loss?core
How is the adaptive GAN weight in VQGAN computed, and why?advanced
How many tokens and bits does a 512 × 512 image take with \(f = 16\) and \(K = 8{,}192\)? What changes at \(f = 8\)?basic
Why do understanding-focused VLMs use SigLIP or CLIP features instead of VQ tokens?core
Explain FSQ and LFQ. What problem do they solve?advanced
Discrete tokens or continuous latents for image generation inside an LLM?advanced
Why is a larger codebook not always better for generation?advanced
Exercises
- In Figure 1 with \(K = 4\), move \(\mathbf{z}_e\) to \((0.4, 0.7)\). Which token is chosen? Compute the codebook and commitment terms with \(\beta = 0.25\).
- A tokenizer has \(f = 8\) and \(K = 16{,}384\). For a 384 × 384 image, give the grid, the number of tokens, the bits, and the compression ratio against 8-bit RGB.
- Code frequencies on a validation set are \((0.7, 0.1, 0.1, 0.1, 0, 0, 0, 0)\). What are usage and perplexity?
- Prove that the codebook term \(\sum_i \lVert \mathbf{z}_{e,i} - \mathbf{e}_k \rVert^2\) over the vectors assigned to code \(k\) is minimized by their mean. Why does this justify EMA?
- FSQ with levels \([7, 5, 5, 5, 5]\): how big is the implicit codebook, and which power of two is it meant to match?
- A 5-second, 24 fps, 512 × 512 video goes through an Emu3-style tokenizer (\(f = 8\), 4× in time). How many tokens? How many with \(f = 16\) instead?
Answerscheck
Key takeaways
- A visual tokenizer = encoder + nearest-code quantizer (224.1) + decoder. The code indices are the tokens; an image costs \(N = HW/f^2\) tokens and at most \(N\log_2 K\) bits (224.2).
- The argmin has no gradient; the straight-through estimator (224.3) copies the decoder’s gradient to the encoder. The codebook learns from a separate term or from EMA (224.4, 224.6); the commitment term (\(\beta = 0.25\)) keeps the encoder near its codes.
- Codebook collapse is the classic failure; watch usage and perplexity (224.7) and fix it with data init, dead-code restarts, low-dimensional normalized codes, or FSQ/LFQ (224.11, 224.12).
- VQGAN’s perceptual loss, patch discriminator and adaptive weight (224.8, 224.9) make \(f = 16\) (256 tokens for 256 × 256) sharp: rFID 7.94 vs 32.01 for DALL·E’s dVAE with 4 times more tokens.
- Stage two is a plain language model over token sequences (224.10). Discrete tokens are natural for generation and for any-to-any models (Chameleon, Emu3); understanding works much better with continuous semantic features (Janus).
Key formulas
References for this session
- van den Oord et al. (2017). Neural Discrete Representation Learning
- Esser et al. (2020). Taming Transformers for High-Resolution Image Synthesis
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models
- van den Oord et al. (2017). Neural Discrete Representation Learning
- Esser et al. (2020). Taming Transformers for High-Resolution Image Synthesis
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models
- Bengio, Léonard & Courville (2013). Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Ramesh et al. (2021). Zero-Shot Text-to-Image Generation
- Razavi, van den Oord & Vinyals (2019). Generating Diverse High-Fidelity Images with VQ-VAE-2
- Yu et al. (2021). Vector-quantized Image Modeling with Improved VQGAN
- Zhang, Isola, Efros, Shechtman & Wang (2018). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Sun et al. (2024). Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
- Chang, Zhang, Jiang, Liu & Freeman (2022). MaskGIT: Masked Generative Image Transformer
- Mentzer, Minnen, Agustsson & Tschannen (2023). Finite Scalar Quantization: VQ-VAE Made Simple
- Yu et al. (2023). Language Model Beats Diffusion: Tokenizer is Key to Visual Generation
- Lee, Kim, Kim, Cho & Han (2022). Autoregressive Image Generation using Residual Quantization
- Tian, Jiang, Yuan, Peng & Wang (2024). Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
- Yu et al. (2024). An Image is Worth 32 Tokens for Reconstruction and Generation
- Li, Tian, Li, Deng & He (2024). Autoregressive Image Generation without Vector Quantization
- Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models
- Wang et al. (2024). Emu3: Next-Token Prediction is All You Need
- Wu et al. (2024). Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
- Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models