Native multimodal and any-to-any models
A VLM reads images and a diffusion model draws them. A native multimodal model does both with one transformer, in one sequence. This session builds the three main designs step by step: one shared vocabulary of image codes (Chameleon, Emu3), next-token text plus diffusion images (Transfusion), and separate encoders for reading and drawing (Janus), then shows how to choose the output head and balance the tasks in the data.
What you will learn
- What native multimodal and any-to-any mean, and the four questions every design must answer: image in, image out, loss, and shared weights
- The shared-vocabulary family: one softmax over text and image codes, Chameleon's 1,024 codes per image and its stability fixes, and Emu3's 4,096 codes, row markers and video
- The hybrid family: Transfusion's loss L_LM + λ·L_DDPM with λ = 5, its causal mask with bidirectional image blocks, and why it matched Chameleon's FID with about 34× less compute
- The decoupled family: why Janus uses SigLIP for reading and VQ codes for drawing, read from its ablation (MMBench 35.0 vs 69.4)
- Monolithic vs factorized output heads, modality masking, and classifier-free guidance on logits
- How to balance text and image tasks with loss weights and data ratios, and how to compute the image share of the loss
- The inference cost of autoregressive vs diffusion image generation: sequential passes vs processed positions
Why this matters
A VLM such as LLaVA can read an image but cannot draw one. A diffusion model such as Stable Diffusion can draw but cannot hold a conversation about what it drew. A native multimodal model is one network that reads and writes text and images (and sometimes video and audio) in the same sequence. An any-to-any model is one where any mix of modalities can be the input and any mix can be the output.
Why would you want one model instead of a VLM plus a separate image generator? Three reasons. First, interleaved output: a recipe with a picture after each step, or a story with illustrations, needs a model that decides on its own when to switch from words to pixels. Second, shared knowledge: a model that understands "a red cube on top of a blue sphere" should also be better at drawing it, and the other way round. Third, editing and multi-turn work: "make the sky darker" needs a model that sees its own previous image as context. GPT-4o image generation and Gemini's native image output are products built on this idea; their exact architectures are not published. [Certain that they are not published]
In session 224 we built the part that turns an image into a grid of integers. In session 228 we saw how to generate continuous image latents with a transformer. In session 179 we saw how images enter a VLM. This session puts these pieces together. We study three families, each represented by one paper: Chameleon and Emu3 (one shared vocabulary of discrete tokens), Transfusion (next-token prediction for text plus diffusion for images, in one transformer), and Janus (separate image encoders for understanding and for generation). Then we look at the two engineering questions every family must answer: what the output head looks like, and how to balance the tasks in the data.
Three ways to build one model for reading and drawing
Every design has to answer four questions:
- Image in: how does an image enter the transformer? As discrete codes (integers), as continuous latent patches, or as features from a vision encoder?
- Image out: how does the model produce an image? By predicting the next code from a softmax, or by removing noise from continuous patches?
- Loss: one cross-entropy for everything, or cross-entropy for text plus a regression loss for images?
- Weights: are all weights shared by all modalities, or does each modality have its own encoder, adaptor or head?
Figure 1 shows the answers for the three families, plus the older "pipeline" design, in which an LLM only writes a prompt and a separate diffusion model draws the picture (NExT-GPT works roughly this way). The pipeline is not native: the LLM never sees or produces the pixels itself.
How to read Figure 1
Click each design. Blue boxes carry text, orange boxes carry images, and the green box is the transformer that all modalities share. Look at the right side: the shared-vocabulary design has one output head, the others have two output paths. The readout lists how one image is produced and which models use the design. In the pipeline design, notice that the image never goes back into the transformer as pixels: the LLM only sends a text prompt or an embedding to a separate generator.
Family 1: one shared vocabulary of discrete tokens
The simplest idea: if an image is a grid of integers (from a VQ tokenizer), then an image is just more "words". Add the image codes to the text vocabulary, and train an ordinary decoder-only transformer with next-token prediction on sequences that mix both. This is early fusion in its purest form: there is no separate vision encoder in the model at all, and all weights are shared by all modalities.
Let \(\mathcal{V}_T\) be the text vocabulary and \(\mathcal{V}_I\) the image codebook. The model sees one sequence \(x_1, \dots, x_n\) where each token comes from the joint vocabulary, and it learns the usual autoregressive factorization:
- Read the symbols
- \(\mathcal{V}_T\)
- text tokens from the BPE tokenizer
- \(\mathcal{V}_I\)
- image codes from the VQ tokenizer (one integer per image patch)
- \(\mathcal{V}_S\)
- special tokens such as begin-of-image
<boi>, end-of-image<eoi>, and end-of-line markers - \(x_{\lt i}\)
- all tokens before position \(i\), of any modality
Because the loss is the same cross-entropy for all tokens, one training run teaches captioning (image first, then text), text-to-image (text first, then image codes), and interleaved documents. Which task a sequence teaches depends only on the order of its pieces.
Chameleon
Chameleon (Meta, 2024) is the reference model for this family. Its image tokenizer turns a 512 × 512 image into 1,024 codes from a codebook of 8,192 entries. Its BPE vocabulary has 65,536 entries, and these 8,192 image codes are part of it, so one eighth of the vocabulary is image codes. [Certain] It comes in 7B and 34B sizes. The pretraining data has three parts: 2.9T tokens of text only, 1.5T tokens from 1.4B image–text pairs, and 400B tokens of interleaved text and images; in total the models saw about 9.2T tokens. [Certain]
Training such a model is harder than it looks. Image codes and text tokens have very different statistics, but they share every weight. The Chameleon team reports that the norms of activations slowly grow until training diverges, and that they needed QK-norm (a normalization on queries and keys before the dot product, for both sizes), a z-loss term \(10^{-5} \log^2 Z\) that keeps the softmax normalizer \(Z\) small, dropout 0.1 for the 7B model, and a changed norm placement for the 34B model. [Certain] Their ablations without image generation did not diverge, which points to the mixed modalities as the cause. (session 179 covers this from the VLM side.)
Emu3
Emu3 (BAAI, 2024) pushes the same idea further: 8B parameters, trained from scratch on text, images and video with next-token prediction only. Its tokenizer compresses 8 × 8 pixels into one code (and 4 frames into one in time), with 32,768 codes, so a 512 × 512 image is \(64 \times 64 = 4{,}096\) tokens. [Certain] The total vocabulary is 184,622 entries. The context is 5,120 tokens in the first stage (text and images) and 131,072 tokens in the second stage, when video is added. [Certain] Each sample has a fixed layout:
[BOS] {caption} [SOV] {meta: resolution, fps, duration} [SOT] {vision tokens with [EOL] and [EOF]} [EOV] [EOS]
Here [EOL] marks the end of each row of codes and [EOF] the end of each frame. These markers let the model know the 2D (and 3D) shape of the grid, because the flat sequence alone does not say where a row ends. For understanding data, the caption is moved after [EOV], so the same format teaches captioning. [Certain] Emu3 reports GenEval 0.54 and DPG-Bench 80.60 for image generation and a VBench total of 80.96 for video. [Certain]
- Tokenize: text \(\to\) BPE ids; each image \(\to\) VQ codes, offset by \(|\mathcal{V}_T|\) so they get their own ids
- build one sequence:
[BOS]text<boi>codes (with row markers)<eoi>more text … - Train: next-token cross-entropy on all positions, with a weight \(w_m\) per modality (formula 232.2)
- Sample: generate text tokens as usual
- if the model emits
<boi>(or the user asks for an image): switch to image mode - \(\quad\) for \(k = 1 \dots N\): mask all non-image logits (formula 232.3), apply CFG (formula 232.7), sample one code
- \(\quad\) append
<eoi>; decode the \(N\) codes with the VQ decoder into pixels - continue generating text; the generated codes stay in the context as input for later turns
Step 3 uses a weighted loss. Image codes are many and hard to predict, so their raw loss can drown the text loss. Emu3 multiplies the loss of vision tokens by 0.5. [Certain] In general:
- Read the symbols
- \(m(i)\)
- the modality of target token \(i\): text, image or special
- \(w_m\)
- the loss weight of that modality, for example \(w_{\text{text}} = 1\), \(w_{\text{image}} = 0.5\)
Restricting the output to one modality
Once the model is inside an image, every next token must be an image code. A text token in the middle of the grid would break the decoder. So at sampling time we set the logits of every token outside \(\mathcal{V}_I\) to \(-\infty\) and renormalize. Chameleon describes exactly this: tokens outside the wanted modality are masked. [Certain]
Take a tiny vocabulary of two text tokens and three image codes, with logits \(z = (2.0,\ 1.0 \mid 0.5,\ 3.0,\ 1.5)\). The full softmax gives \((0.203,\ 0.075 \mid 0.045,\ 0.553,\ 0.123)\). The image codes together get 0.722 of the mass, so without the mask the model would put a text token into the grid 27.8% of the time. With the mask (232.3), the three image probabilities are divided by 0.722 and become \((0.063,\ 0.766,\ 0.171)\). The order between image codes does not change; only the mass that leaked to text is removed.
Family 2: next-token text plus diffusion images (Transfusion)
Discrete codes have a cost. The VQ step throws information away (Chameleon's paper notes its tokenizer is weak on images with a lot of text), and generating a 4,096-code image takes 4,096 sequential steps. Diffusion on continuous latents keeps more detail and needs fewer steps. Transfusion (Meta, 2024) asks: why not do both in one transformer? Text tokens are predicted with a softmax as usual. Image patches are continuous vectors, and the same transformer is trained to predict the noise that was added to them, as in session 225.
A 256 × 256 image goes through a VAE (86M parameters) to a \(32 \times 32 \times 8\) latent. Groups of \(2 \times 2\) latent pixels become one patch, so the image is \(16 \times 16 = 256\) patch vectors. A small linear layer (or a few U-Net down blocks) maps each patch to the model width, and the mirror layer maps the output back. [Certain] The image is wrapped in <BOI> and <EOI> and placed inside the text sequence. The loss is a sum:
- Read the symbols
- \(\mathcal{L}_{\text{LM}}\)
- next-token cross-entropy, computed on text tokens only
- \(\mathcal{L}_{\text{DDPM}}\)
- noise-prediction error, computed on image patches only
- \(x_t\)
- the image latent after adding noise of level \(t\) (one \(t\) per image)
- \(c\)
- everything before the image in the sequence: the caption, earlier images, earlier turns
- \(\lambda\)
- a balancing weight; the paper uses \(\lambda = 5\)
The attention mask
Text must stay causal: a text token may not look at later tokens, or next-token prediction becomes cheating. But an image is not produced left to right; the denoiser updates all patches together, so each patch should see all other patches of the same image. Transfusion therefore uses causal attention everywhere, plus full (bidirectional) attention inside each image. [Certain]
Here \(\operatorname{img}(i)\) is the index of the image that position \(i\) belongs to, or \(\varnothing\) for text. \(M_{ij} = 1\) means token \(i\) may attend to token \(j\). Figure 2 lets you build this mask and compare it to the plain causal mask of Chameleon.
How to read Figure 2
Click any cell to see why it is allowed or blocked. With "Caption → image" there are 9 tokens: 45 of the 81 pairs are allowed by the causal rule, and the Transfusion mask adds 6 more (the upper triangle of the 4 × 4 image block). Switch to "Two images": the patches of image 2 can see image 1 (it is in the past), but no patch of image 1 can see image 2. Each image block is bidirectional on its own; between blocks the rule is still causal.
Training and sampling
- input: a document with text and images; VAE encoder \(E\); noise schedule \(\bar\alpha_t\)
- for each image: \(x_0 \leftarrow\) patchify\((E(\text{image}))\); draw \(t \sim U\{1, \dots, 1000\}\) and \(\varepsilon \sim \mathcal{N}(0, I)\)
- \(\quad x_t \leftarrow \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon\) // noisy patches go in the sequence
- run the transformer once with the mask of (232.5)
- \(\mathcal{L} \leftarrow\) cross-entropy on text positions \(+\ \lambda \cdot\) MSE between \(\varepsilon\) and the predicted noise on image positions
- Decoding: sample text tokens one by one
- if the model samples
<BOI>: append \(N\) patches of pure noise - \(\quad\) for \(s = S, \dots, 1\): one forward pass over the patches (text prefix cached), one denoising update
- \(\quad\) append
<EOI>; decode the clean latent with the VAE; continue sampling text
A detail that matters: when an image comes before its caption, the model must read the image to write the text. If the image were very noisy, the caption would have nothing to learn from. So for the 20% of training examples with the image first, Transfusion limits the noise to at most \(t = 500\) of 1,000. This raised the captioning score (CIDEr) from 33.7 to 35.2 for the 7B model. [Certain]
What the comparison showed
The Transfusion paper trained Chameleon-style models with the same data and compute as a control. For text-to-image, Transfusion beat the Chameleon approach at less than a third of the compute, and reached the same FID with about 34 times less compute. [Certain] Their 7B model, trained on 2T tokens (about half text, half image patches), reached GenEval 0.63 and COCO FID 6.78. [Certain] A main reason is that a diffusion step updates all 256 patches at once and keeps continuous information, while the discrete model must predict 1,024 codes one at a time through a lossy tokenizer. [Likely]
For an image of \(H \times W\) pixels, down-sampling factor \(f\) and patch size \(p\) (in latent pixels):
Chameleon: \(512/16 = 32\), so \(N = 32^2 = 1{,}024\). Emu3: \(512/8 = 64\), so \(N = 4{,}096\), plus one [EOL] per row, 4,160 positions in total. [Likely for the exact marker count] Janus: \(384/16 = 24\), so \(N = 576\). Transfusion: \(256/(8 \cdot 2) = 16\), so \(N = 256\). The same 256 × 256 image at Emu3 settings would be \(32^2 = 1{,}024\) codes, four times more positions than Transfusion's patches.
Family 3: separate encoders for understanding and for generation (Janus)
Both families above use one image representation for reading and for drawing. Janus (DeepSeek, 2024) argues that these two jobs want different things. Understanding wants semantic features: "this is a dog, it is jumping". A CLIP or SigLIP encoder gives exactly that. Generation wants features that can be turned back into exact pixels: colors, textures, edges. A VQ tokenizer gives that. One representation is a compromise for both.
So Janus decouples the image input: SigLIP-Large (384 × 384, 576 patch features) plus a 2-layer MLP adaptor for understanding, and LlamaGen's VQ tokenizer (16,384 codes, \(f = 16\), so 576 codes for 384 × 384) plus another 2-layer MLP adaptor for generation. Both feed the same 1.3B DeepSeek-LLM. A separate, randomly initialized image head predicts the next VQ code; the normal text head predicts text. [Certain] The released config shows the LLM width (2,048), the text vocabulary (102,400) and the image head over 16,384 codes. [Certain]
The paper backs this with an ablation, all at the same small scale. Figure 3 shows it.
How to read Figure 3
Choose a metric. Compare A with D first: same data, same LLM, only the understanding encoder changes, and MMBench goes from 35.0 to 69.4. Then compare D with E: training generation together with understanding costs only 1.2 MMBench points against the understanding-only model. On COCO FID, D (8.53) is even slightly better than the generation-only model F (8.92). Grey bars mean the model was not trained for that task.
Janus trains in three stages. Stage I freezes the LLM and both encoders and trains only the two adaptors and the image head. Stage II unfreezes the LLM for unified pretraining. Stage III fine-tunes everything except the generation encoder. [Certain] Janus-Pro (2025) kept the design, scaled to 7B, changed the data mix and added about 72 million synthetic aesthetic images; Janus-Pro-7B reports GenEval 0.80 and MMBench 79.2. [Certain] Show-o is a related single-transformer design: it predicts text autoregressively and generates image codes with discrete diffusion (it fills in masked codes over a few rounds). [Certain] JanusFlow keeps the decoupled encoders but replaces VQ-code prediction with rectified flow inside the LLM, which is a step toward the hybrid family. [Certain]
Janus generates images with classifier-free guidance (CFG, see session 227) applied directly to the logits of the image head. During training, the caption is replaced by a pad token 10% of the time, so the same model also learns the unconditional distribution. At sampling time:
- Read the symbols
- \(\ell_c\)
- logits over the 16,384 image codes, given the caption
- \(\ell_u\)
- logits given only the pad token (no caption)
- \(s\)
- guidance scale; Janus uses \(s = 5\); Emu3 uses 5.0 to 5.5 depending on the benchmark
CFG doubles the work per image: every step runs the model twice, once with the caption and once without (in practice as a batch of two). [Certain]
Monolithic vs factorized output heads
Look at the right side of Figure 1 again. There are two ways to make the output.
A monolithic head is one linear layer and one softmax over the whole joint vocabulary. Chameleon and Emu3 do this. It is simple, and it lets the model decide on its own to switch modality (by emitting <boi>). The costs: the head is large (Emu3: \(184{,}622 \times 4{,}096 \approx 756\)M weights if it is a plain linear layer), probability mass can leak across modalities (so you need the mask of 232.3), and text and image tokens compete for the same output space.
A factorized head gives each modality its own output layer. Janus has a text head over 102,400 tokens and a separate image head over 16,384 codes. Transfusion has the LM head for text and a linear or U-Net up layer that outputs noise vectors for patches. Each head can have its own loss (cross-entropy vs MSE), its own size, and its own learning speed. The model still needs a way to decide which head to use; usually the special token <boi> (from the text head) or the user's request switches modes. [Certain for Janus and Transfusion]
| Monolithic (Chameleon, Emu3) | Factorized (Janus, Transfusion) | |
|---|---|---|
| Output layer | one softmax over \(\mathcal{V}_T \cup \mathcal{V}_I \cup \mathcal{V}_S\) | one head per modality |
| Loss | cross-entropy only | cross-entropy for text; cross-entropy or MSE for images |
| Modality switch | the model samples <boi> from the same softmax | a special token or the request selects the head |
| Main risk | leakage across modalities; one modality dominates the gradient | more parts to tune; heads can learn at very different speeds |
A further step is to split the transformer body too. BAGEL (ByteDance, 2025) uses separate weight sets for understanding tokens and generation tokens inside each layer (a "mixture of transformers"), while all tokens still attend to each other in one sequence. [Likely] This keeps one context but stops the two tasks from fighting over the same weights.
Balancing tasks in the data
A unified model is trained on a mix: pure text, image understanding (image then text), image generation (text then image), and interleaved documents. Two numbers control the balance: how often each kind of sample appears (the data ratio), and how much each token counts in the loss (the loss weight). Images make this tricky, because one image is hundreds or thousands of tokens while its caption is often a few dozen.
Take a batch where a fraction \(p\) of samples is text-only documents of length \(L_t\), and the rest are caption–image pairs with \(N_c\) caption tokens and \(N_i\) image tokens. With image loss weight \(w\), the share of the total loss weight that comes from image tokens is:
The gradient follows the loss weight, so \(\rho_I\) tells you roughly how much of each update is spent on drawing images. [Likely as an approximation: it ignores that some tokens are harder than others]
With \(N_i = 4{,}096\) codes, a 64-token caption, no text-only data (\(p = 0\)) and \(w = 1\): \(\rho_I = 4096/4160 = 0.985\). Only 1.5% of the loss is about text. Emu3's weight \(w = 0.5\) gives \(2048/2112 = 0.970\): still 97% image. With a 200-token caption it is \(2048/2248 = 0.911\). So the weight alone is not enough; the data must also contain plenty of text-heavy samples (pure text, captioning, instruction data), or the language skills fade.
How to read Figure 4
Start at the Emu3 preset: the bottom bar is almost all orange. Lower \(w\) and watch the orange shrink, slowly, because there are 64 times more image tokens than caption tokens. Then raise \(p\), the fraction of text-only documents: this moves the balance much faster. Click the Janus preset: with 576 image tokens, the same caption is a much larger share. This is why models with short image sequences are easier to balance.
Real recipes do both. Janus states its mixes as understanding : pure text : text-to-image samples. Stage I uses 1:0:1, stage II 2:3:5 and stage III 7:3:10. [Certain] Janus-Pro changed stage III to 5:1:4 (half of all samples are understanding data). [Certain] Chameleon's stage 2 (the last 20% of training) lowers the weight of stage 1 data by 50% and mixes in higher-quality data while keeping the image–text proportion similar. [Certain] Transfusion uses about 1:1 text tokens to image patches and balances the two losses with \(\lambda = 5\). [Certain]
The cost of generating one image
The families differ a lot at inference time. An autoregressive model with a KV cache runs one forward pass per image token: \(N\) sequential passes, each one processing a single new position. A diffusion model runs \(S\) sampling steps, each one a forward pass over all \(N\) patches at once:
With CFG, both double the positions (the conditional and unconditional runs are batched, so the number of sequential passes stays the same). The two families stress different hardware limits. One decode step of an LLM is memory-bound: its time is set by reading all weights from memory, not by the few FLOPs for one token (session 78). So 4,096 sequential AR passes are slow even though each one does little math. Diffusion does fewer passes, but each one processes many positions and is compute-bound. Figure 5 compares them.
How to read Figure 5
Left bars: sequential forward passes (latency). Right bars: positions processed (total compute, about \(2P\) FLOPs each for a model with \(P\) parameters). At 512 pixels and 250 steps, the diffusion model needs only 250 passes but processes 512,000 positions with CFG; Emu3-style needs 4,096 passes but only 8,192 positions. Lower the steps to 25 (a fast sampler from session 226): the diffusion model now processes 51,200 positions, still more than Emu3-style, but it needs about 164 times fewer sequential passes. Increase the side to 1,024 pixels: AR passes grow four times.
Beyond images: any-to-any
The same recipes extend to other modalities. Anything with a tokenizer can join the shared vocabulary: AnyGPT turns speech, images and music into discrete tokens and trains one LLM on them. [Certain] Unified-IO 2 tokenizes images, text, audio and actions for one encoder–decoder model. [Certain] Emu3 handles video by compressing 4 frames into one latent frame and adding [EOF] markers. [Certain] The pipeline style also scales to many outputs: NExT-GPT attaches separate diffusion decoders for image, video and audio to an LLM. [Certain] The trade-off is the same as for images: shared tokens are simple and fully native, but long and lossy; continuous outputs with diffusion or flow are higher quality but need a second kind of loss and sampler. The session 184 session follows the audio side.
A tiny unified model in PyTorch
The code builds a joint vocabulary, trains-step with the weighted loss (232.2), samples image codes with the modality mask (232.3) and CFG (232.7), builds the Transfusion mask (232.5), and computes the hybrid loss (232.4). It is deterministic and runs on a CPU in a few seconds.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # ---- 1) one shared vocabulary: text ids, then image codes, then special tokens ---- N_TEXT, N_IMG = 1000, 256 # toy sizes (Chameleon: 57,344 + 8,192) BOI, EOI = N_TEXT + N_IMG, N_TEXT + N_IMG + 1 V = N_TEXT + N_IMG + 2 IMG = torch.zeros(V, dtype=torch.bool); IMG[N_TEXT:N_TEXT + N_IMG] = True def attn_mask(is_img, img_id): """Formula (232.5): causal, plus full attention inside the same image.""" T = len(is_img) causal = torch.ones(T, T).tril().bool() same = (img_id[:, None] == img_id[None, :]) & is_img[:, None] & is_img[None, :] return causal | same # True = may attend class TinyUnified(nn.Module): def __init__(self, d=64, layers=2): super().__init__() self.emb = nn.Embedding(V, d) self.pos = nn.Embedding(512, d) layer = nn.TransformerEncoderLayer(d, 4, 4 * d, dropout=0.0, batch_first=True, norm_first=True) self.body = nn.TransformerEncoder(layer, layers, enable_nested_tensor=False) self.head = nn.Linear(d, V) # monolithic head: one softmax over all of V def forward(self, ids, allow): h = self.emb(ids) + self.pos(torch.arange(ids.shape[1])) h = self.body(h, mask=~allow) # PyTorch: True = blocked return self.head(h) model = TinyUnified() # a caption of 6 text tokens, then <boi> 16 image codes <eoi> text = torch.randint(0, N_TEXT, (6,)) img = torch.randint(N_TEXT, N_TEXT + N_IMG, (16,)) ids = torch.cat([text, torch.tensor([BOI]), img, torch.tensor([EOI])])[None] # ---- 2) weighted next-token loss, formula (232.2); Emu3 uses w_img = 0.5 ---- causal = torch.ones(ids.shape[1], ids.shape[1]).tril().bool() # discrete tokens: plain causal logits = model(ids, causal) tgt = ids[0, 1:] ce = F.cross_entropy(logits[0, :-1], tgt, reduction="none") w = torch.where(IMG[tgt], 0.5, 1.0) loss = (w * ce).sum() / w.sum() print("weighted loss:", round(loss.item(), 3), " image share of weight:", round((w * IMG[tgt]).sum().item() / w.sum().item(), 3)) # 16*0.5 / (16*0.5 + 7) = 0.533 # ---- 3) sample image codes with a modality mask (232.3) and CFG on logits (232.7) ---- @torch.no_grad() def sample_image(prompt, n=16, s=3.0, temp=1.0): g = torch.Generator().manual_seed(1) cond = torch.cat([prompt, torch.tensor([BOI])])[None] unc = torch.tensor([[BOI]]) # unconditional: the prompt is dropped for _ in range(n): lc = model(cond, torch.ones(cond.shape[1], cond.shape[1]).tril().bool())[0, -1] lu = model(unc, torch.ones(unc.shape[1], unc.shape[1]).tril().bool())[0, -1] l = lu + s * (lc - lu) # classifier-free guidance on logits l = l.masked_fill(~IMG, float("-inf")) # only image codes are allowed now nxt = torch.multinomial(F.softmax(l / temp, -1), 1, generator=g) cond = torch.cat([cond, nxt[None]], 1); unc = torch.cat([unc, nxt[None]], 1) return cond[0, len(prompt) + 1:] codes = sample_image(text) print("all sampled ids are image codes:", bool(IMG[codes].all())) # True # ---- 4) Transfusion-style mask: 3 text tokens, one image of 4 patches, 2 text tokens ---- is_img = torch.tensor([0, 0, 0, 1, 1, 1, 1, 0, 0]).bool() img_id = torch.tensor([-1, -1, -1, 0, 0, 0, 0, -1, -1]) M = attn_mask(is_img, img_id) print("allowed pairs:", int(M.sum()), "of", M.numel()) # 51 of 81 (causal alone: 45) # ---- 5) the hybrid loss, formula (232.4): LM loss + lambda * noise-prediction MSE ---- eps, eps_hat = torch.randn(4, 32), torch.randn(4, 32) * 0.1 lm = torch.tensor(2.0) # stand-in for the text cross-entropy print("L_total =", round((lm + 5.0 * F.mse_loss(eps_hat, eps)).item(), 3))
On PyTorch 2.8 (CPU) this prints a weighted loss of 7.593 (an untrained model, close to \(\ln 1258 = 7.137\) plus noise), an image share of 0.533, True for the masked sampling, 51 allowed pairs, and \(\mathcal{L}_{\text{total}} = 8.656\). In a real Transfusion model the image patches are continuous vectors that enter through a linear layer instead of nn.Embedding, and the mask of part 4 is passed to every attention layer.
Where this is used
| Application | Which family fits | Why |
|---|---|---|
| Interleaved documents (recipes, tutorials, illustrated stories) | Shared vocabulary or hybrid | The model itself decides when an image goes in the text |
| Image editing by conversation ("make the sky darker") | Native models of any family | The previous image is in the context, so the edit can keep what should stay [Likely] |
| Strong understanding plus decent generation in a small model | Decoupled (Janus, Janus-Pro) | Semantic encoder for reading; VQ codes for drawing |
| High-quality text-to-image inside an LLM | Hybrid (Transfusion, JanusFlow, BAGEL) | Continuous latents and few steps beat long discrete code sequences |
| Video generation and video captioning in one model | Shared vocabulary with a video tokenizer (Emu3) | Same next-token loss for frames and words; long contexts are needed |
| Video key-moment detection with a visual summary | Any native model | Find the moment (understanding), then generate a thumbnail or an edited frame (generation) |
| Speech in, speech out assistants | Shared vocabulary of audio tokens (AnyGPT-style) or omni models | Audio codes are just more tokens; see session 184 |
| Quick prototype of "chat that can draw" | Pipeline (LLM + separate diffusion model) | No joint training needed, but no shared knowledge either |
Common mistakes
- Sampling from the full vocabulary inside an image. One text token in a code grid breaks the decoder or shifts every later row. Mask the logits (232.3) and count the codes; force
<eoi>after exactly \(N\) codes. - Forgetting the row markers. A flat list of 4,096 codes does not say where a row ends. Without
[EOL]markers or 2D position encodings, the model must learn the grid width from data, and other resolutions fail. - Using a causal mask for diffusion patches. A denoiser that cannot see later patches of the same image works worse; Transfusion needs the bidirectional block (232.5). The opposite mistake (letting text see the future) leaks the answer.
- Letting image tokens take the whole gradient. With thousands of image tokens per sample, text skills fade. Watch \(\rho_I\) (232.8) and a text-only benchmark.
- Captioning from a very noisy image. In a hybrid model, a caption that follows an image needs a readable image; cap the noise (Transfusion: at most \(t = 500\)).
- Ignoring training stability. Shared weights across modalities can make norms grow until training diverges. Use QK-norm and z-loss from the start, as Chameleon did.
- Using a VQ tokenizer for understanding. Pixel-faithful codes carry weak semantics; the Janus ablation shows MMBench 35.0 vs 69.4 with SigLIP.
- Comparing models without counting CFG. CFG doubles the compute per image; a fair cost comparison must include it.
Interview questions
What does "native multimodal" mean, and how is it different from LLaVA?basic
How does Chameleon generate an image with a plain decoder-only LLM?basic
Why is training an early-fusion model like Chameleon unstable, and what fixes did they use?core
Explain Transfusion's loss and attention mask.core
Why did Transfusion beat the Chameleon approach in compute efficiency?advanced
Why does Janus use two image encoders?core
Monolithic vs factorized output heads: what are the trade-offs?core
How would you balance text and image losses in a unified model?core
Compare the inference cost of AR image generation and diffusion in one model.advanced
Why does Transfusion cap the noise level for image-to-text examples?advanced
How do you apply classifier-free guidance to an autoregressive image model?core
Exercises
- How many tokens does a 768 × 768 image take with Chameleon's tokenizer (\(f = 16\))? With Emu3's (\(f = 8\))?
- Logits over two text tokens and two image codes are \((1, 1 \mid 2, 0)\). What are the image probabilities after the mask (232.3)?
- In Figure 2's "Two images" preset (1 text, BOI, 3 patches, EOI, 1 text, BOI, 3 patches, EOI: 12 tokens), how many pairs does the Transfusion mask allow?
- With \(N_i = 1{,}024\), \(N_c = 32\), \(w = 1\) and half of the samples text-only with \(L_t = 2{,}048\), what is \(\rho_I\)?
- A Transfusion-style model makes a 512 × 512 image with 50 steps and CFG. How many sequential passes and positions? Compare with Chameleon-style with CFG.
Answerscheck
Key takeaways
- Native multimodal models read and write several modalities with one transformer. There are three families: shared discrete vocabulary (Chameleon, Emu3), hybrid next-token plus diffusion (Transfusion), and decoupled encoders (Janus).
- Shared vocabulary (232.1): images are VQ codes in the same softmax; simple and fully native, but long sequences (1,024 to 4,096 per image), a lossy tokenizer, and training stability issues (QK-norm, z-loss).
- Hybrid (232.4, 232.5): cross-entropy for text, noise-prediction MSE for continuous patches, causal mask with bidirectional image blocks; it matched Chameleon's FID with about 34× less compute.
- Decoupled (Janus): SigLIP for reading, VQ for drawing; MMBench 35.0 → 69.4 against a single VQ encoder, at no cost in FID.
- Output heads are monolithic (one softmax, needs masking 232.3) or factorized (one head per modality). Balance tasks with loss weights and data ratios, and measure the image share \(\rho_I\) (232.8).
Key formulas
References for this session
- Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models
- Wang et al. (2024). Emu3: Next-Token Prediction is All You Need
- Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
- Wu et al. (2024). Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models
- Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models
- Wang et al. (2024). Emu3: Next-Token Prediction is All You Need
- Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
- Wu et al. (2024). Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models
- Esser et al. (2020). Taming Transformers for High-Resolution Image Synthesis
- Ho et al. (2020). Denoising Diffusion Probabilistic Models
- Ho & Salimans (2022). Classifier-Free Diffusion Guidance
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Chen et al. (2025). Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Ma et al. (2024). JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
- Sun et al. (2024). Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
- Xie et al. (2024). Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
- Deng et al. (2025). Emerging Properties in Unified Multimodal Pretraining (BAGEL)
- Zhan et al. (2024). AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
- Lu et al. (2023). Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
- Wu et al. (2023). NExT-GPT: Any-to-Any Multimodal LLM