LLM & VLM courseBlog
1Track 1: Foundations · session 1 of 15

What is a large language model? The big picture

One idea runs through this whole course: a model that reads tokens and predicts the next one. We follow one sentence through a real LLM, from text to the next word. Then we see how the model learns, how it writes, and how one model can do many tasks.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: Beginner · start hereBefore this: Basic Python and high-school algebra

What you will learn

  • What a language model computes, written as one probability formula, and why “predict the next token” teaches so much
  • The full path inside an LLM: text, tokens, vectors, transformer blocks, scores, probabilities, the next token, with real tensor shapes
  • How softmax and temperature turn scores into choices
  • How the model learns (cross-entropy loss, perplexity), and why training is parallel but writing is one token at a time
  • The life of a model, the three transformer families, and how one model serves many tasks, from sentiment to video search
  • The numbers interviewers ask about: parameters, training compute, memory, tokens per second

Why this matters

Ten years ago, every language task had its own model. One model found spam. Another model translated. Another one wrote summaries. Each needed its own labeled data, and each was trained from zero.

Large language models changed this with one simple bet: train one big model on one simple task, using a huge amount of text, and then reuse it for almost everything. The simple task is predict the next piece of text. That is all. No labels from humans are needed, because every text already contains its own answers: the next word is right there.

Why does such a simple task teach so much? Look at what you must know to guess the next word in each of these sentences:

Text so farGood next wordWhat you need to know
“She opened the door and walked …”inGrammar
“The capital of France is …”ParisA fact about the world
“17 + 25 = …”42Arithmetic
“The keeper dives to his left and saves the …”shot / penaltyHow football works
“All the clues point to the butler, so the killer is …”the butlerSimple reasoning over the story

To make these guesses well across trillions of words, the model has to store a lot about language and the world inside its weights. Later we add a little extra training so the model follows instructions and answers questions. That gives you a chat assistant.

This session gives you the whole picture at once. Every later session zooms into one box of this picture. When a later topic feels confusing, come back here and find where it sits.

The one formula: predict the next token

An LLM does not read letters or whole words. It reads tokens: small pieces of text such as keeper or the. A token can be a whole word, part of a word, a space plus a word, or a punctuation mark. Session 8 shows exactly how text is cut into tokens. For now, think “token ≈ a common piece of a word”.

We write a text as a list of tokens \(x_1, x_2, \dots, x_T\). The model gives a probability to every possible next token, given all the tokens before it:

\[ p_\theta\left(x_t \mid x_1, x_2, \dots, x_{t-1}\right) \;=\; p_\theta\left(x_t \mid x_{\lt t}\right) \tag{1.1} \]
Read the symbols
\(x_t\)
the token at position \(t\) (the one we want to predict)
\(x_{\lt t}\)
all tokens before position \(t\): the context
\(p_\theta(\cdot \mid \cdot)\)
a probability that depends on the model’s numbers (weights) \(\theta\). The bar “\(\mid\)” means “given”.
\(\theta\)
all the learned weights of the network, often billions of numbers

So the output of the model is not one word. It is a list of probabilities, one for each token in the vocabulary. If the vocabulary has 128,256 tokens, the model outputs 128,256 numbers that add up to 1.

From next-token probabilities to the probability of a whole text

If we can predict each next token, we can give a probability to a whole sentence. We use the chain rule of probability: the chance of the whole sentence is the chance of the first token, times the chance of the second token given the first, and so on.

\[ p_\theta(x_1, \dots, x_T) \;=\; \prod_{t=1}^{T} p_\theta\left(x_t \mid x_{\lt t}\right) \tag{1.2} \]

The symbol \(\prod\) means “multiply all of these together”. This formula is exact; it is not an approximation. It is the reason a model that predicts “the next token” is a full model of language.

Worked example

Suppose a model gives these probabilities for the sentence “The keeper saves the shot”:

\(p(\text{keeper}\mid\text{The}) = 0.02\), \(p(\text{saves}\mid\text{The keeper}) = 0.30\), \(p(\text{the}\mid\dots) = 0.60\), \(p(\text{shot}\mid\dots) = 0.40\).

Then the sentence probability (after “The”) is

\[ 0.02 \times 0.30 \times 0.60 \times 0.40 = 0.00144 \]

With 50 tokens instead of 4, the product would be something like \(10^{-60}\). Computers cannot store such tiny numbers well. So we always work with logarithms instead.

The log turns a product into a sum. This is the same formula, but safe for computers:

\[ \log p_\theta(x_1, \dots, x_T) \;=\; \sum_{t=1}^{T} \log p_\theta\left(x_t \mid x_{\lt t}\right) \tag{1.3} \]

In the example:

\[ \log 0.02 + \log 0.30 + \log 0.60 + \log 0.40 = -3.91 - 1.20 - 0.51 - 0.92 = -6.54, \qquad e^{-6.54} \approx 0.00144 \]

Same answer, no tiny numbers. (We use the natural log, written \(\log\), everywhere in this course.)

Writing text: predict, add, repeat

To write, the model uses formula (1.1) again and again. It predicts a probability list, picks one token, adds it to the input, and repeats. This loop is called autoregressive generation: “auto” because the model feeds on its own outputs. Play with it in Figure 1.

Figure 1. A tiny language model about football. It knows only a few hundred word pairs, but it works exactly like a real LLM: it gives probabilities to the next token, one token is chosen, and the loop repeats. You can also click any bar to choose that token yourself.
How to read Figure 1

The grey box is the text so far. The small number under each token is the probability the model gave it when it was chosen. The bars are the probabilities for the next token. “Add the most likely token” is greedy decoding. “Sample” draws at random, using the bars as chances. Move the temperature to the left and the tallest bar grows (safer, more repetitive text). Move it right and the bars get flatter (more surprising text). The line at the bottom adds up the log-probabilities, which is formula (1.3) in action.

Inside the machine: from text to the next token

Now let us open the box. Every decoder LLM (GPT, Llama, Qwen, Gemma, Mistral) follows the same path. Figure 2 walks through it one stage at a time, with the real tensor shapes. We use a batch of 1 sentence with 4 tokens, and the sizes of Llama 3 8B.

Figure 2. The path of one sentence through a decoder LLM. Click any box, or press Play. The shape shows the size of the data at that point: \(B\) is the batch size, \(T\) the number of tokens, \(d\) the width of each vector, \(|V|\) the vocabulary size.
How to read Figure 2

Read left to right. Text becomes token IDs (integers). Each ID picks one row of a big table, the embedding matrix, so each token becomes a vector of \(d = 4096\) numbers. The vectors pass through 32 transformer blocks; in each block, tokens share information (attention) and then each token is processed alone (MLP). Only the last token’s final vector is used to predict the next token: it is compared with every vocabulary vector to get one score per token (logits). Softmax turns scores into probabilities, and one token is chosen.

Stage 1: text to token IDs

A tokenizer cuts the text into tokens and replaces each token by its number in a fixed list, the vocabulary. The vocabulary size is written \(|V|\). Llama 3 uses \(|V| = 128{,}256\). After this stage the sentence is just a list of integers, shape \((B, T)\).

Stage 2: token IDs to vectors (embedding)

The model has a big table \(E\) with one row per token. Each row is a vector of \(d\) numbers. Looking up the row for token \(x_t\) gives its first vector:

\[ \mathbf{h}^{(0)}_t = E[x_t], \qquad E \in \R^{|V| \times d} \tag{1.4} \]

This is the same as multiplying a one-hot vector (all zeros except a 1 at position \(x_t\)) by the matrix \(E\). The lookup is just a fast way to do that multiplication. These rows are learned during training, so tokens with similar use end up with similar vectors (session 11). The model also needs to know the order of tokens; modern LLMs add it inside attention with rotary position embeddings, RoPE (session 13 and session 14).

Stage 3: transformer blocks

The vectors pass through \(L\) identical blocks (\(L = 32\) in Llama 3 8B). Each block has two parts:

  • Attention: every token looks at the tokens before it and pulls in useful information. This is how “it” can learn that it means “the ball”. Track 2 is all about this.
  • MLP (a small feed-forward network): each token’s vector is processed alone, like each token “thinking” about what it just collected.

Each part adds its result to the vector instead of replacing it. This running vector is called the residual stream. With \(\mathbf{H}\) as the matrix of all token vectors (shape \(T \times d\)), one modern block is:

\[ \mathbf{U} = \mathbf{H} + \operatorname{Attn}\big(\operatorname{Norm}(\mathbf{H})\big), \qquad \mathbf{H}' = \mathbf{U} + \operatorname{MLP}\big(\operatorname{Norm}(\mathbf{U})\big) \tag{1.5} \]

The shape stays \((B, T, d)\) through all blocks. Only the content changes: the vectors become richer and more about meaning in context.

Stage 4: the last vector becomes scores (logits)

To predict the next token, we take the final vector of the last position, \(\mathbf{h}^{(L)}_T\), and multiply by an output matrix \(W_U\) (the “unembedding”). This gives one score per vocabulary token:

\[ \mathbf{z} = \mathbf{h}^{(L)}_T \, W_U, \qquad W_U \in \R^{d \times |V|}, \qquad \mathbf{z} \in \R^{|V|} \tag{1.6} \]

Each score \(z_i\) is a dot product between the final vector and the column for token \(i\). A dot product is large when two vectors point the same way. So the score answers: “how well does token \(i\) fit here?” These raw scores are called logits. They can be any real number, positive or negative.

Stage 5: scores to probabilities (softmax)

We need probabilities: numbers between 0 and 1 that add up to 1. The softmax function does this:

\[ p_i = \softmax(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{|V|} e^{z_j}} \tag{1.7} \]

Why this formula? The exponential \(e^{z}\) makes every score positive and keeps the order (a bigger score gives a bigger number). Dividing by the sum makes everything add up to 1. It is also smooth, so we can train it with gradients. Figure 3 lets you change the scores and watch the probabilities.

Worked example

Scores \(\mathbf{z} = (2.0,\ 1.0,\ 0.1)\). Then \(e^{2.0} = 7.389\), \(e^{1.0} = 2.718\), \(e^{0.1} = 1.105\). The sum is \(11.212\). So the probabilities are \((0.659,\ 0.242,\ 0.099)\). Notice: a gap of 1 in score means a factor of \(e \approx 2.7\) in probability.

Temperature \(\tau\) divides the scores before softmax:

\[ p_i(\tau) = \frac{e^{z_i/\tau}}{\sum_{j} e^{z_j/\tau}} \tag{1.8} \]

With \(\tau = 0.5\), the same scores become \((4, 2, 0.2)\) and the probabilities become \((0.864,\ 0.117,\ 0.019)\): sharper. With \(\tau = 2\) they become \((0.502,\ 0.304,\ 0.194)\): flatter. As \(\tau \to 0\), all the probability goes to the top score (greedy). As \(\tau \to \infty\), all tokens become equally likely.

TokenScore \(z_i\) (drag)\(z_i/\tau\)\(e^{z_i/\tau - m}\)\(p_i\)
Figure 3. Softmax with temperature, computed live. The column \(e^{z_i/\tau - m}\) uses the stable trick from the tip: \(m\) is the largest value of \(z/\tau\), so the top token always gets exactly 1 before we divide by the sum.
How to read Figure 3

Drag a score and watch only the probabilities change; their sum stays 1. Raise one score by 1 and its probability grows by about a factor of \(e\) (before the sum is recomputed). Then move the temperature: low temperature makes one winner; high temperature spreads the probability. The entropy in the readout measures how spread out the choice is: 0 means certain.

Stage 6: choosing the next token

Now we pick one token from the probabilities. Greedy takes the largest. Sampling draws at random using the probabilities. Better methods (top-k, top-p, min-p) cut off the long tail of unlikely tokens before sampling; session 79 covers them. The chosen token is added to the input, and the whole path runs again.

Real sizes: Llama 3 8B

Here are the real numbers for one popular model, and where its 8 billion parameters live. Counting parameters like this is a classic interview question.

PartShapeParameters
Embedding table \(E\)\(128{,}256 \times 4{,}096\)525 M
Attention, per block (\(W_Q, W_K, W_V, W_O\); 32 query heads, 8 key/value heads, head size 128)\(4096{\times}4096 + 2 \cdot 4096{\times}1024 + 4096{\times}4096\)41.9 M
MLP, per block (SwiGLU, hidden size 14,336: three matrices)\(3 \times 4{,}096 \times 14{,}336\)176.2 M
All 32 blocks\(32 \times (41.9 + 176.2)\) M6.98 B
Output matrix \(W_U\) (not shared with \(E\) in this model)\(4{,}096 \times 128{,}256\)525 M
Total (norm weights are tiny)about 8.03 B

How an LLM learns: training

Where do the right weights \(\theta\) come from? We start from random numbers and improve them, step by step, so that the model gives high probability to real text.

The labels are free

Take any text. The input is the text; the target is the same text shifted left by one token. Position 1 must predict token 2, position 2 must predict token 3, and so on. Nobody has to label anything. This is called self-supervised learning.

InputThekeepersavestheshot
Targetkeepersavestheshot.

The loss: cross-entropy

For each position, we look at the probability the model gave to the correct next token, take its log, and make it negative. Then we average over all positions. This is the cross-entropy loss:

\[ \mathcal{L}(\theta) = -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta\left(x_t \mid x_{\lt t}\right) \tag{1.9} \]

It is just formula (1.3) with a minus sign, divided by \(T\). So minimizing the loss is the same as making real text as likely as possible (this is called maximum likelihood).

Worked example

If the model gives the correct token probability 0.9, the loss for that position is \(-\log 0.9 = 0.105\). For 0.25 it is \(1.386\). For 0.01 it is \(4.61\). Being confidently wrong costs a lot. Before training, a model with \(|V| = 128{,}256\) gives every token about \(1/|V|\), so the loss starts near \(\log 128{,}256 \approx 11.8\).

A friendlier number is perplexity, the exponential of the loss:

\[ \operatorname{PPL} = e^{\mathcal{L}} \tag{1.10} \]

A loss of 2.0 gives perplexity \(e^2 \approx 7.4\). You can read it as: “on average, the model is as unsure as if it were choosing between about 7 equally likely tokens.” Lower is better. Perplexity is only fair to compare between models that use the same tokenizer and the same test text.

Figure 4. The loss for one position is \(-\log p\). It is 0 when the model is sure and right (\(p = 1\)), and it grows without limit as \(p \to 0\).
How to read Figure 4

The curve is steep on the left: going from \(p = 0.01\) to \(p = 0.1\) lowers the loss by 2.3, the same as going from 0.1 to 1. So training spends a lot of effort fixing tokens the model finds very surprising. The dashed lines show where your current \(p\) sits.

Improving the weights

Backpropagation computes the gradient \(\nabla_\theta \mathcal{L}\): for every weight, which direction makes the loss smaller. Then we take a small step that way:

\[ \theta \leftarrow \theta - \eta \, \nabla_\theta \mathcal{L}(\theta) \tag{1.11} \]

Here \(\eta\) is the learning rate (the step size). Real LLMs use a smarter version called AdamW (session 4), but the idea is the same. Training repeats this for millions of steps, each on a batch of millions of tokens.

Why training is parallel but writing is not

During training we already know the whole text. So we can compute the predictions for all positions in one forward pass. We only need to stop each position from seeing the future (that would be cheating). A causal mask does this inside attention. Feeding the true previous tokens, not the model’s own guesses, is called teacher forcing.

During writing we do not know the future, because the model creates it. Token 6 depends on token 5, which the model has just picked. So generation must go one token at a time. Figure 5 shows both.

Figure 5. The causal mask. Each row is a position that is predicting; each column is a token it may look at. Colored cells are allowed, grey cells are hidden. In training, every row is computed in the same pass. In generation, only one new row is computed per step, and the rows already done are kept in the KV cache.
How to read Figure 5

Row 3 (“saves”) may look at columns 1 to 3 (“The”, “keeper”, “saves”) and must predict “the”. It may not look at column 4, because that is the answer. In the training view, all five predictions and all five losses come from one pass. Switch to generation and press “Next step”: each step adds one row and one new token. The rows above are not recomputed; their keys and values are reused from the cache. That is why generation needs a KV cache (session 80).

How much compute does training need?

A useful rule: one forward pass costs about \(2N\) floating-point operations (FLOPs) per token, where \(N\) is the number of parameters (one multiply and one add per weight). The backward pass costs about twice that. So training costs about:

\[ C_{\text{train}} \approx 6 \, N \, D \tag{1.12} \]

where \(D\) is the number of training tokens.

Worked example

An 8B model trained on 15 trillion tokens:

\[ C \approx 6 \times (8 \times 10^{9}) \times (15 \times 10^{12}) = 7.2 \times 10^{23} \text{ FLOPs} \]

If one GPU really delivers \(4 \times 10^{14}\) FLOP/s, this takes

\[ \frac{7.2 \times 10^{23}}{4 \times 10^{14}} = 1.8 \times 10^{9} \text{ GPU-seconds,} \]

which is about 500,000 GPU-hours, or about 3 weeks on 1,000 GPUs. Real runs need more, because GPUs are never fully busy. (This is an estimate for practice, not a reported number.)

How an LLM writes: inference

Writing has two phases, and they behave very differently on a GPU:

  • Prefill: read the whole prompt in one pass (like training, all positions at once). It does a lot of math per byte read, so it is limited by compute.
  • Decode: make one new token per step. Each step does little math but must read all the weights from memory. So it is limited by memory speed.
Worked example: why decoding is slow

An 8B model in BF16 has \(8 \times 10^9 \times 2 = 16\) GB of weights. An NVIDIA L4 reads memory at about 300 GB/s. At batch size 1, every new token reads all 16 GB, so one token takes at least \(16 / 300 \approx 0.053\) s. That is at most about 19 tokens per second, no matter how fast the math units are. With 4-bit weights (4 GB) the limit rises to about 75 tokens per second. On an H100 (about 3,350 GB/s) the same model could reach about 200 tokens per second. These are upper bounds; real systems also read the KV cache and have overheads.

Three numbers describe serving speed: TTFT (time to first token, mostly prefill), TPOT (time per output token, the decode speed), and throughput (tokens per second for all users together). The whole of Track 5 is about making these better.

The life of a model

An LLM product is built in stages. Each stage has its own data, loss and track in this course.

1. Pretrainingtrillions of tokensnext-token loss→ base model · Track 3 2. SFTquestion–answer pairsloss on answers only→ follows instructions 3. Preferences, RL“A is better than B”checkable rewardsDPO, PPO, GRPO · Track 4 4. Evaluationbenchmarks, judges,your own test setTrack 6 5. Servingquantize, batch, cachevLLM, TensorRT-LLMTrack 5 fix problems, train again
Figure 6. The life of a model. Pretraining gives a base model that continues text. Supervised fine-tuning (SFT) and preference training turn it into an assistant that answers. Evaluation finds problems; serving makes it fast and cheap.

The difference between a base model and an assistant is easy to see. Ask a base model “What is the capital of France?” and it may continue with “What is the capital of Germany? What is the capital of Spain?”, because lists of questions are common on the web. An assistant answers “Paris.” Same architecture, different training data at the end.

Three families of transformers

All LLMs use transformer blocks, but they differ in which tokens may look at which. That is set by the attention mask. Figure 7 shows the three main families.

Figure 7. Attention masks. A colored cell at row \(i\), column \(j\) means “token \(i\) may look at token \(j\)”.
FamilyWho sees whomTrained withGood atExamples
Encoder-onlyEvery token sees every tokenFill in hidden words (masked LM)Understanding: classification, embeddings for searchBERT, RoBERTa
Decoder-onlyEach token sees only the pastNext-token predictionWriting, chat, almost all modern LLMs and VLMsGPT, Llama, Qwen, Gemma
Encoder–decoderEncoder: all; decoder: past + all of the encoderRebuild hidden spans; translateInput-to-output tasks: translation, speech to textT5, BART, Whisper

Why did decoder-only win for large models? One model and one loss do everything; every token in the training text gives a training signal; and the same model can both read and write. Session 23 discusses the trade-offs in depth.

One model, many tasks

A trained LLM is a general engine. We use three different “exits” from it, shown in Figure 8.

Input tokenstext, image, video Transformer trunkL blocks of attention + MLPhidden vectors (B, T, d) Exit 1: next-token head → generatechat, summarization, QA, captions, video descriptions Exit 2: pool the vectors → one embeddingtext search, video retrieval, duplicate finding, clustering Exit 3: small classifier head → a labelsentiment, topic, brand safety, action or shot type
Figure 8. Three ways to use one trained model. The trunk is shared; only the small exit changes.

Classification by asking for a label word

To decide if a review is positive or negative, write a prompt like “Review: … Sentiment:” and compare the probability of the two label words:

\[ \hat{y} = \argmax_{c \in \{\text{positive},\,\text{negative}\}} \; \log p_\theta\left(\text{word}_c \mid \text{prompt}\right) \tag{1.13} \]

No training is needed. With labeled data, a small classifier on top of the hidden vectors is cheaper and often more accurate (session 55 and session 144).

Retrieval by comparing vectors

Average the final hidden vectors of a text into one vector \(\mathbf{e}\) (an embedding). Two texts with similar meaning get vectors that point the same way. We compare them with cosine similarity:

\[ \cos(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert} \tag{1.14} \]

The value is 1 for the same direction, 0 for unrelated, and \(-1\) for opposite. Search engines store millions of such vectors and return the closest ones to the query (session 150 and session 151).

Images and video become tokens too

A vision-language model (VLM) cuts an image into small patches, turns each patch into a vector with a vision encoder, and puts these vectors in front of the text tokens. The LLM then reads one long sequence:

\[ \big[\, \underbrace{\mathbf{v}_1, \dots, \mathbf{v}_m}_{\text{image patches}},\; \underbrace{\mathbf{e}_1, \dots, \mathbf{e}_n}_{\text{text tokens}} \,\big] \tag{1.15} \]

A video is many frames, so it becomes many such patch groups along time. The same next-token engine can then describe a scene, answer a question about a match, or write a timestamp.

Video taskInput → outputWhich exitWhere in the course
Text-to-video retrieval“corner kick that leads to a goal” → ranked clipsEmbeddings (exit 2)Session 208
Action recognitionclip → “shot”, “pass”, “foul”Classifier (exit 3) or label words (exit 1)Session 203
Key-moment detectionfull match → start and end times of goals and savesClassifier per segment, or timestamps as textSession 207
Video fingerprintingclip → “same broadcast as clip 4,127?”Special copy-detection embeddings, not meaningSession 209
Video segmentationclip + a click on a player → a mask in every frameA separate mask decoder with memory (SAM 2)Session 204
Video question answeringclip + “who touched the ball last?” → answerGenerate (exit 1), often with retrieval firstSession 210

Pseudo-code

Here is the whole session as two short algorithms. Read them slowly: each line is a box from Figure 2.

Algorithm 1: one training step of a language model
  1. input: a batch of token sequences \(X \in \{1..|V|\}^{B \times (T+1)}\), weights \(\theta\), learning rate \(\eta\)
  2. \(X_{\text{in}} \leftarrow X[:, 1{:}T]\), \(\;Y \leftarrow X[:, 2{:}T{+}1]\) // targets are the inputs shifted by one
  3. \(H \leftarrow E[X_{\text{in}}]\) // embedding lookup, shape \((B, T, d)\)
  4. for \(\ell = 1\) to \(L\) do
  5. \(H \leftarrow H + \operatorname{Attn}_\ell(\operatorname{Norm}(H))\) // with causal mask: no peeking
  6. \(H \leftarrow H + \operatorname{MLP}_\ell(\operatorname{Norm}(H))\)
  7. \(Z \leftarrow \operatorname{Norm}(H)\, W_U\) // logits for every position, shape \((B, T, |V|)\)
  8. \(\mathcal{L} \leftarrow -\frac{1}{BT}\sum_{b,t} \log \softmax(Z_{b,t})[Y_{b,t}]\) // cross-entropy (1.9)
  9. \(g \leftarrow \nabla_\theta \mathcal{L}\) // backpropagation
  10. \(\theta \leftarrow \theta - \eta\, g\) // in practice: AdamW with clipping and a schedule
Algorithm 2: generate text with temperature
  1. input: prompt tokens \(x_{1..n}\), temperature \(\tau\), max new tokens \(K\)
  2. for \(k = 1\) to \(K\) do
  3. \(\mathbf{z} \leftarrow \text{model}(x_{1..n})[\text{last position}]\) // one score per vocabulary token
  4. if \(\tau = 0\) then \(x_{n+1} \leftarrow \argmax_i z_i\) // greedy
  5. else \(x_{n+1} \sim \softmax(\mathbf{z}/\tau)\) // sample, formula (1.8)
  6. \(n \leftarrow n + 1\)
  7. if \(x_n\) is the end-of-text token then break
  8. return \(x_{1..n}\)

And the same ideas in real PyTorch. The first block computes the training loss with the shift by one; the second is a simple generation loop.

import torch
import torch.nn.functional as F

torch.manual_seed(0)
V, d, T = 1000, 64, 8                      # vocabulary size, vector width, sequence length
tokens = torch.randint(0, V, (2, T + 1))   # 2 sequences of T+1 token ids
x, y = tokens[:, :-1], tokens[:, 1:]       # input and target, shifted by one

emb = torch.nn.Embedding(V, d)             # the table E, formula (1.4)
unemb = torch.nn.Linear(d, V, bias=False)  # W_U, formula (1.6)
h = emb(x)                                 # (2, T, d); a real model runs L blocks here
logits = unemb(h)                          # (2, T, V)
loss = F.cross_entropy(logits.reshape(-1, V), y.reshape(-1))   # formula (1.9)
loss.backward()
print(round(loss.item(), 2), round(torch.exp(loss).item()))    # about 7.0 and about 1100: close to log V and V
@torch.no_grad()
def generate(model, ids, max_new=20, temperature=1.0, eos_id=None):
    for _ in range(max_new):
        logits = model(ids)[:, -1, :]                  # scores for the next token only
        if temperature == 0:
            nxt = logits.argmax(dim=-1, keepdim=True)  # greedy
        else:
            probs = torch.softmax(logits / temperature, dim=-1)
            nxt = torch.multinomial(probs, num_samples=1)
        ids = torch.cat([ids, nxt], dim=1)             # add it and repeat
        if eos_id is not None and nxt.item() == eos_id:
            break
    return ids
# This loop re-reads the whole sequence every step. Real systems keep a KV cache (session 80).

Common mistakes and failure modes

  • “Tokens are words.” No. One word can be several tokens, and token counts differ a lot between languages. Cost, speed and context limits are all counted in tokens.
  • “High probability means correct.” No. The model learns likely text. Wrong but typical answers can get high probability (hallucination).
  • “Greedy decoding is always best.” Greedy text often repeats itself in loops. For open writing, sampling with a cut-off works better; for exact answers (math, code, labels), low temperature is good.
  • “Temperature 0 gives the same output every time.” Usually, but not always: in large batched serving, tiny floating-point differences can change which token is top when two scores are almost equal. [Likely]
  • Exposure bias. In training the model always sees true previous tokens; in generation it sees its own, sometimes wrong, tokens. Small errors can grow over long outputs.
  • Context limit. The model only sees the last \(T_{\max}\) tokens (for example 8,192 for Llama 3 8B). Anything earlier is simply not there (Track 2 and Track 5 discuss longer context).

Interview questions

Explain an LLM in two sentences.basic
An LLM is a decoder-only transformer trained to predict the next token on trillions of tokens of text, by minimizing cross-entropy. It generates text autoregressively: predict a distribution over the vocabulary, pick a token, append it, repeat.
Why do we work with log-probabilities?basic
The probability of a long text is a product of many numbers below 1, which quickly becomes too small for floating point (underflow). The log turns the product into a sum (formula 1.3), which is stable. It also gives a simple loss whose gradient is well behaved, and maximizing the log-likelihood is the same as maximizing the likelihood because log is increasing.
What is the difference between logits and probabilities? Why softmax?basic
Logits are raw scores in \((-\infty, \infty)\), one per token, from \(\mathbf{h} W_U\). Probabilities are in \([0,1]\) and sum to 1. Softmax maps one to the other: the exponential keeps order and makes values positive, and the normalization makes them sum to 1. It is smooth (good gradients), and adding the same constant to all logits does not change the result, which allows the stable max-subtraction trick.
Training is parallel over positions but generation is sequential. Why?core
In training the full target text is known, so with teacher forcing and a causal mask we compute all \(T\) predictions in one forward pass. In generation, token \(t+1\) depends on token \(t\), which the model has just produced, so steps cannot run in parallel. Prefill (the prompt) is still parallel; only decode is sequential. Speculative decoding (session 128) is a trick to get some parallelism back.
An 8B model in BF16: how much memory to serve it, and how many FLOPs per token?core
Weights: \(8\times10^9 \times 2\) bytes = 16 GB, plus the KV cache, which grows with context length and batch size (for Llama 3 8B about 128 KiB per token in BF16, so 1 GiB for 8,192 tokens), plus activations and runtime overhead. Compute: about \(2N = 16\) GFLOPs per generated token for the matrix multiplies, plus a smaller attention term that grows with context length.
Estimate the compute to pretrain an 8B model on 15T tokens.core
Use \(C \approx 6ND\): \(6 \times 8\times10^9 \times 15\times10^{12}\) \(= 7.2\times10^{23}\) FLOPs. The 6 is 2 for the forward pass and 4 for the backward pass. Then divide by the real throughput per GPU (peak × utilization) to get GPU-hours.
Why is decoding memory-bound at batch size 1, and how do you fix it?advanced
Each decode step performs about \(2N\) FLOPs while reading about \(2N\) bytes of BF16 weights: about 1 FLOP per byte. GPUs need hundreds of FLOPs per byte to be compute-bound, so the step time is set by memory bandwidth: time ≥ weight bytes / bandwidth. Fixes: batch many requests (weights read once for many tokens), quantize weights (fewer bytes), speculative decoding (verify several tokens per weight read), and use GQA or MLA to shrink the KV cache that is also read every step.
What does temperature do? What happens as it goes to 0 and to infinity?basic
It divides the logits before softmax (formula 1.8). Small \(\tau\) makes the distribution sharper; as \(\tau \to 0\) it becomes greedy (all mass on the top token). Large \(\tau\) flattens it; as \(\tau \to \infty\) it becomes uniform over the vocabulary. It does not change the order of tokens, only how strongly the top ones are preferred.
Encoder-only or decoder-only: when would you choose each?core
Encoder-only (bidirectional attention) gives strong representations per token and per sentence at small size and low latency: good for classification, tagging and embeddings for retrieval. Decoder-only is needed when you must generate, and at large scale it also does understanding well through prompting; it is the standard for chat and VLMs. Encoder–decoder fits clear input-to-output tasks such as translation and speech recognition, where the input is read once and the output is generated.
What is perplexity, and is lower always better?core
Perplexity is \(e^{\mathcal{L}}\), the exponential of the average next-token cross-entropy: roughly the number of equally likely choices the model is torn between. Lower is better on the same test text and the same tokenizer. It is not comparable across tokenizers (different token units), and it does not measure helpfulness, truthfulness or instruction following, which is why chat models are evaluated with tasks and judges.
Using only the training objective, explain why LLMs hallucinate.advanced
The objective rewards text that is likely under the training distribution, not text that is true. When the model lacks a fact, a fluent, typical-looking answer still gets higher likelihood than “I don’t know”, which is rare in web text. Sampling can also pick a wrong low-probability token, and later tokens then build on it. Fixes include retrieval (RAG), training to abstain, and calibration (session 152 and session 156).
How can a generative LLM do classification or retrieval?core
Classification: compare the probabilities of label words after a prompt (formula 1.13), or train a small head on the hidden states. Retrieval: pool the hidden states into one embedding and compare embeddings with cosine similarity (formula 1.14); embedding models are usually fine-tuned with a contrastive loss so that similar meanings get close vectors (session 60).

Exercises

  1. Compute \(\softmax(1, 2, 3)\) by hand.
  2. The model gives the correct token probability 0.5. What is the loss? A model has average loss 3.0. What is its perplexity?
  3. How many parameters are in the Llama 3 8B embedding table?
  4. Give an upper bound on decode speed for an 8B model on an L4 (300 GB/s) at batch 1, in BF16 and in 4-bit.
  5. In Figure 1, set the temperature to its lowest value and press “Add the most likely token” until the sentence ends. Then do it again. Why is the sentence the same both times? Now use “Sample” at temperature 2. What changes?
Answerscheck
1. \(e^1 = 2.718\), \(e^2 = 7.389\), \(e^3 = 20.086\), sum \(30.193\); so \((0.090,\ 0.245,\ 0.665)\). 2. \(-\log 0.5 = 0.693\); \(e^{3} \approx 20.1\). 3. \(128{,}256 \times 4{,}096 = 525{,}336{,}576 \approx 0.53\)B. 4. BF16: \(300/16 \approx 19\) tokens/s; 4-bit: \(300/4 = 75\) tokens/s (upper bounds). 5. Greedy has no randomness, so the same probabilities always give the same choices. At temperature 2 the bars are flatter and samples differ more, sometimes in odd ways (“the keeper sings”).

Key takeaways

  • An LLM models \(p_\theta(x_t \mid x_{\lt t})\). The chain rule makes this a full model of text (1.2), and we use logs for stability (1.3).
  • The path is: tokens → embedding rows → \(L\) blocks of attention + MLP on a residual stream → last vector × \(W_U\) → logits → softmax → choose a token → repeat.
  • Training minimizes cross-entropy (1.9) with targets shifted by one; it is parallel thanks to the causal mask. Perplexity is \(e^{\mathcal{L}}\).
  • Training costs about \(6ND\) FLOPs; generating costs about \(2N\) FLOPs per token, but decode speed is limited by memory bandwidth.
  • The same trunk serves generation, embeddings (search) and classification, and with image or video tokens it becomes a VLM.

Key formulas

\[ p_\theta(x_{1:T}) = \prod_t p_\theta(x_t \mid x_{\lt t}), \quad \mathcal{L} = -\tfrac{1}{T}\sum_t \log p_\theta(x_t \mid x_{\lt t}), \quad \operatorname{PPL} = e^{\mathcal{L}}, \quad p_i = \frac{e^{z_i/\tau}}{\sum_j e^{z_j/\tau}}, \quad C \approx 6ND \]

References for this session

  1. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 1: Introduction
  2. Zhao et al. (2023). A Survey of Large Language Models
  3. Raschka, Build a Large Language Model (From Scratch), chapter 1: Understanding large language models
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language