Read straight down: explanation → mathematics → code → lab → review are now visible on one page, without separate tabs. Content status: this is still a concise legacy draft, not yet the complete research-level rewrite. Read the rebuilt Session 01 as the depth benchmark.
Interactive architecture
Click each block; the detail appears in the right panel. This is a conceptual process diagram.
Simple English explanation
A common generative VLM uses a pretrained image encoder, a learned vision-to-text projector and an autoregressive text model. Fusion can use concatenation, cross-attention or resampling.
Open technical explanation
Distinguish image encoder features, projector output, language embeddings and the autoregressive decoder. Compare early fusion to cross-attention.
Trace the input and output, the trainable variables, the loss or objective, and what differs during inference. Study failure modes and measure the trade-off against a simple baseline.
Where this is useful
Distinguish image encoder features, projector output, language embeddings and the autoregressive decoder. Compare early fusion to cross-attention.
Mathematics
Work through each variable, its shape, and units. Derive the formula from its inputs, then test extreme parameter values. The formula above is an essential starting point, not every derivation in the field.
Derivation checklist
State assumptions; identify normalization axes; check dimensions; locate the learned map; calculate compute and memory complexity; evaluate a numerical example by hand.
Minimal Python / PyTorch
import torch
visual=torch.randn(2,196,768)
project=torch.nn.Linear(768,1024)
print(project(visual).shape)Runnable toy or conceptual snippet; this is not an end-to-end model training pipeline.
Make this production-ready
Fix seeds, add shape assertions and unit tests, gather GPU timing and memory, add relevant task metrics and compare with a published baseline.
Interactive budget exploration
Illustrative information-budget slider; it does not run a real model. Use it to reason about efficiency-quality trade-offs.
Experiment
Distinguish image encoder features, projector output, language embeddings and the autoregressive decoder. Compare early fusion to cross-attention.
Record the experimental method
Choose one variable, keep the rest fixed, report quality, latency, GPU peak memory, five failures and a hypothesis for each failure.
Review and self-check
What are the inputs and outputs?
Identify representations and shapes, including time units for video.
What is optimized during training?
Specify data, objective, trainable parameters and any frozen encoders.
How would you falsify a performance claim?
Construct hard negatives and ablations; compare on fixed resources and a meaningful benchmark.
Review after 1, 3, 7, 14 and 30 days.