What is a large language model? The big picture
One idea runs through this whole course: a model that reads tokens and predicts the next one. We follow one sentence through a real LLM, from text to the next word. Then we see how the model learns, how it writes, and how one model can do many tasks.
What you will learn
- What a language model computes, written as one probability formula, and why “predict the next token” teaches so much
- The full path inside an LLM: text, tokens, vectors, transformer blocks, scores, probabilities, the next token, with real tensor shapes
- How softmax and temperature turn scores into choices
- How the model learns (cross-entropy loss, perplexity), and why training is parallel but writing is one token at a time
- The life of a model, the three transformer families, and how one model serves many tasks, from sentiment to video search
- The numbers interviewers ask about: parameters, training compute, memory, tokens per second
Why this matters
Ten years ago, every language task had its own model. One model found spam. Another model translated. Another one wrote summaries. Each needed its own labeled data, and each was trained from zero.
Large language models changed this with one simple bet: train one big model on one simple task, using a huge amount of text, and then reuse it for almost everything. The simple task is predict the next piece of text. That is all. No labels from humans are needed, because every text already contains its own answers: the next word is right there.
Why does such a simple task teach so much? Look at what you must know to guess the next word in each of these sentences:
| Text so far | Good next word | What you need to know |
|---|---|---|
| “She opened the door and walked …” | in | Grammar |
| “The capital of France is …” | Paris | A fact about the world |
| “17 + 25 = …” | 42 | Arithmetic |
| “The keeper dives to his left and saves the …” | shot / penalty | How football works |
| “All the clues point to the butler, so the killer is …” | the butler | Simple reasoning over the story |
To make these guesses well across trillions of words, the model has to store a lot about language and the world inside its weights. Later we add a little extra training so the model follows instructions and answers questions. That gives you a chat assistant.
This session gives you the whole picture at once. Every later session zooms into one box of this picture. When a later topic feels confusing, come back here and find where it sits.
The one formula: predict the next token
An LLM does not read letters or whole words. It reads tokens: small pieces of text such as keeper or the. A token can be a whole word, part of a word, a space plus a word, or a punctuation mark. Session 8 shows exactly how text is cut into tokens. For now, think “token ≈ a common piece of a word”.
We write a text as a list of tokens \(x_1, x_2, \dots, x_T\). The model gives a probability to every possible next token, given all the tokens before it:
- Read the symbols
- \(x_t\)
- the token at position \(t\) (the one we want to predict)
- \(x_{\lt t}\)
- all tokens before position \(t\): the context
- \(p_\theta(\cdot \mid \cdot)\)
- a probability that depends on the model’s numbers (weights) \(\theta\). The bar “\(\mid\)” means “given”.
- \(\theta\)
- all the learned weights of the network, often billions of numbers
So the output of the model is not one word. It is a list of probabilities, one for each token in the vocabulary. If the vocabulary has 128,256 tokens, the model outputs 128,256 numbers that add up to 1.
From next-token probabilities to the probability of a whole text
If we can predict each next token, we can give a probability to a whole sentence. We use the chain rule of probability: the chance of the whole sentence is the chance of the first token, times the chance of the second token given the first, and so on.
The symbol \(\prod\) means “multiply all of these together”. This formula is exact; it is not an approximation. It is the reason a model that predicts “the next token” is a full model of language.
Suppose a model gives these probabilities for the sentence “The keeper saves the shot”:
\(p(\text{keeper}\mid\text{The}) = 0.02\), \(p(\text{saves}\mid\text{The keeper}) = 0.30\), \(p(\text{the}\mid\dots) = 0.60\), \(p(\text{shot}\mid\dots) = 0.40\).
Then the sentence probability (after “The”) is
With 50 tokens instead of 4, the product would be something like \(10^{-60}\). Computers cannot store such tiny numbers well. So we always work with logarithms instead.
The log turns a product into a sum. This is the same formula, but safe for computers:
In the example:
Same answer, no tiny numbers. (We use the natural log, written \(\log\), everywhere in this course.)
Writing text: predict, add, repeat
To write, the model uses formula (1.1) again and again. It predicts a probability list, picks one token, adds it to the input, and repeats. This loop is called autoregressive generation: “auto” because the model feeds on its own outputs. Play with it in Figure 1.
How to read Figure 1
The grey box is the text so far. The small number under each token is the probability the model gave it when it was chosen. The bars are the probabilities for the next token. “Add the most likely token” is greedy decoding. “Sample” draws at random, using the bars as chances. Move the temperature to the left and the tallest bar grows (safer, more repetitive text). Move it right and the bars get flatter (more surprising text). The line at the bottom adds up the log-probabilities, which is formula (1.3) in action.
Inside the machine: from text to the next token
Now let us open the box. Every decoder LLM (GPT, Llama, Qwen, Gemma, Mistral) follows the same path. Figure 2 walks through it one stage at a time, with the real tensor shapes. We use a batch of 1 sentence with 4 tokens, and the sizes of Llama 3 8B.
How to read Figure 2
Read left to right. Text becomes token IDs (integers). Each ID picks one row of a big table, the embedding matrix, so each token becomes a vector of \(d = 4096\) numbers. The vectors pass through 32 transformer blocks; in each block, tokens share information (attention) and then each token is processed alone (MLP). Only the last token’s final vector is used to predict the next token: it is compared with every vocabulary vector to get one score per token (logits). Softmax turns scores into probabilities, and one token is chosen.
Stage 1: text to token IDs
A tokenizer cuts the text into tokens and replaces each token by its number in a fixed list, the vocabulary. The vocabulary size is written \(|V|\). Llama 3 uses \(|V| = 128{,}256\). After this stage the sentence is just a list of integers, shape \((B, T)\).
Stage 2: token IDs to vectors (embedding)
The model has a big table \(E\) with one row per token. Each row is a vector of \(d\) numbers. Looking up the row for token \(x_t\) gives its first vector:
This is the same as multiplying a one-hot vector (all zeros except a 1 at position \(x_t\)) by the matrix \(E\). The lookup is just a fast way to do that multiplication. These rows are learned during training, so tokens with similar use end up with similar vectors (session 11). The model also needs to know the order of tokens; modern LLMs add it inside attention with rotary position embeddings, RoPE (session 13 and session 14).
Stage 3: transformer blocks
The vectors pass through \(L\) identical blocks (\(L = 32\) in Llama 3 8B). Each block has two parts:
- Attention: every token looks at the tokens before it and pulls in useful information. This is how “it” can learn that it means “the ball”. Track 2 is all about this.
- MLP (a small feed-forward network): each token’s vector is processed alone, like each token “thinking” about what it just collected.
Each part adds its result to the vector instead of replacing it. This running vector is called the residual stream. With \(\mathbf{H}\) as the matrix of all token vectors (shape \(T \times d\)), one modern block is:
The shape stays \((B, T, d)\) through all blocks. Only the content changes: the vectors become richer and more about meaning in context.
Stage 4: the last vector becomes scores (logits)
To predict the next token, we take the final vector of the last position, \(\mathbf{h}^{(L)}_T\), and multiply by an output matrix \(W_U\) (the “unembedding”). This gives one score per vocabulary token:
Each score \(z_i\) is a dot product between the final vector and the column for token \(i\). A dot product is large when two vectors point the same way. So the score answers: “how well does token \(i\) fit here?” These raw scores are called logits. They can be any real number, positive or negative.
Stage 5: scores to probabilities (softmax)
We need probabilities: numbers between 0 and 1 that add up to 1. The softmax function does this:
Why this formula? The exponential \(e^{z}\) makes every score positive and keeps the order (a bigger score gives a bigger number). Dividing by the sum makes everything add up to 1. It is also smooth, so we can train it with gradients. Figure 3 lets you change the scores and watch the probabilities.
Scores \(\mathbf{z} = (2.0,\ 1.0,\ 0.1)\). Then \(e^{2.0} = 7.389\), \(e^{1.0} = 2.718\), \(e^{0.1} = 1.105\). The sum is \(11.212\). So the probabilities are \((0.659,\ 0.242,\ 0.099)\). Notice: a gap of 1 in score means a factor of \(e \approx 2.7\) in probability.
Temperature \(\tau\) divides the scores before softmax:
With \(\tau = 0.5\), the same scores become \((4, 2, 0.2)\) and the probabilities become \((0.864,\ 0.117,\ 0.019)\): sharper. With \(\tau = 2\) they become \((0.502,\ 0.304,\ 0.194)\): flatter. As \(\tau \to 0\), all the probability goes to the top score (greedy). As \(\tau \to \infty\), all tokens become equally likely.
| Token | Score \(z_i\) (drag) | \(z_i/\tau\) | \(e^{z_i/\tau - m}\) | \(p_i\) |
|---|
How to read Figure 3
Drag a score and watch only the probabilities change; their sum stays 1. Raise one score by 1 and its probability grows by about a factor of \(e\) (before the sum is recomputed). Then move the temperature: low temperature makes one winner; high temperature spreads the probability. The entropy in the readout measures how spread out the choice is: 0 means certain.
Stage 6: choosing the next token
Now we pick one token from the probabilities. Greedy takes the largest. Sampling draws at random using the probabilities. Better methods (top-k, top-p, min-p) cut off the long tail of unlikely tokens before sampling; session 79 covers them. The chosen token is added to the input, and the whole path runs again.
Real sizes: Llama 3 8B
Here are the real numbers for one popular model, and where its 8 billion parameters live. Counting parameters like this is a classic interview question.
| Part | Shape | Parameters |
|---|---|---|
| Embedding table \(E\) | \(128{,}256 \times 4{,}096\) | 525 M |
| Attention, per block (\(W_Q, W_K, W_V, W_O\); 32 query heads, 8 key/value heads, head size 128) | \(4096{\times}4096 + 2 \cdot 4096{\times}1024 + 4096{\times}4096\) | 41.9 M |
| MLP, per block (SwiGLU, hidden size 14,336: three matrices) | \(3 \times 4{,}096 \times 14{,}336\) | 176.2 M |
| All 32 blocks | \(32 \times (41.9 + 176.2)\) M | 6.98 B |
| Output matrix \(W_U\) (not shared with \(E\) in this model) | \(4{,}096 \times 128{,}256\) | 525 M |
| Total (norm weights are tiny) | about 8.03 B |
How an LLM learns: training
Where do the right weights \(\theta\) come from? We start from random numbers and improve them, step by step, so that the model gives high probability to real text.
The labels are free
Take any text. The input is the text; the target is the same text shifted left by one token. Position 1 must predict token 2, position 2 must predict token 3, and so on. Nobody has to label anything. This is called self-supervised learning.
| Input | The | keeper | saves | the | shot |
|---|---|---|---|---|---|
| Target | keeper | saves | the | shot | . |
The loss: cross-entropy
For each position, we look at the probability the model gave to the correct next token, take its log, and make it negative. Then we average over all positions. This is the cross-entropy loss:
It is just formula (1.3) with a minus sign, divided by \(T\). So minimizing the loss is the same as making real text as likely as possible (this is called maximum likelihood).
If the model gives the correct token probability 0.9, the loss for that position is \(-\log 0.9 = 0.105\). For 0.25 it is \(1.386\). For 0.01 it is \(4.61\). Being confidently wrong costs a lot. Before training, a model with \(|V| = 128{,}256\) gives every token about \(1/|V|\), so the loss starts near \(\log 128{,}256 \approx 11.8\).
A friendlier number is perplexity, the exponential of the loss:
A loss of 2.0 gives perplexity \(e^2 \approx 7.4\). You can read it as: “on average, the model is as unsure as if it were choosing between about 7 equally likely tokens.” Lower is better. Perplexity is only fair to compare between models that use the same tokenizer and the same test text.
How to read Figure 4
The curve is steep on the left: going from \(p = 0.01\) to \(p = 0.1\) lowers the loss by 2.3, the same as going from 0.1 to 1. So training spends a lot of effort fixing tokens the model finds very surprising. The dashed lines show where your current \(p\) sits.
Improving the weights
Backpropagation computes the gradient \(\nabla_\theta \mathcal{L}\): for every weight, which direction makes the loss smaller. Then we take a small step that way:
Here \(\eta\) is the learning rate (the step size). Real LLMs use a smarter version called AdamW (session 4), but the idea is the same. Training repeats this for millions of steps, each on a batch of millions of tokens.
Why training is parallel but writing is not
During training we already know the whole text. So we can compute the predictions for all positions in one forward pass. We only need to stop each position from seeing the future (that would be cheating). A causal mask does this inside attention. Feeding the true previous tokens, not the model’s own guesses, is called teacher forcing.
During writing we do not know the future, because the model creates it. Token 6 depends on token 5, which the model has just picked. So generation must go one token at a time. Figure 5 shows both.
How to read Figure 5
Row 3 (“saves”) may look at columns 1 to 3 (“The”, “keeper”, “saves”) and must predict “the”. It may not look at column 4, because that is the answer. In the training view, all five predictions and all five losses come from one pass. Switch to generation and press “Next step”: each step adds one row and one new token. The rows above are not recomputed; their keys and values are reused from the cache. That is why generation needs a KV cache (session 80).
How much compute does training need?
A useful rule: one forward pass costs about \(2N\) floating-point operations (FLOPs) per token, where \(N\) is the number of parameters (one multiply and one add per weight). The backward pass costs about twice that. So training costs about:
where \(D\) is the number of training tokens.
An 8B model trained on 15 trillion tokens:
If one GPU really delivers \(4 \times 10^{14}\) FLOP/s, this takes
which is about 500,000 GPU-hours, or about 3 weeks on 1,000 GPUs. Real runs need more, because GPUs are never fully busy. (This is an estimate for practice, not a reported number.)
How an LLM writes: inference
Writing has two phases, and they behave very differently on a GPU:
- Prefill: read the whole prompt in one pass (like training, all positions at once). It does a lot of math per byte read, so it is limited by compute.
- Decode: make one new token per step. Each step does little math but must read all the weights from memory. So it is limited by memory speed.
An 8B model in BF16 has \(8 \times 10^9 \times 2 = 16\) GB of weights. An NVIDIA L4 reads memory at about 300 GB/s. At batch size 1, every new token reads all 16 GB, so one token takes at least \(16 / 300 \approx 0.053\) s. That is at most about 19 tokens per second, no matter how fast the math units are. With 4-bit weights (4 GB) the limit rises to about 75 tokens per second. On an H100 (about 3,350 GB/s) the same model could reach about 200 tokens per second. These are upper bounds; real systems also read the KV cache and have overheads.
Three numbers describe serving speed: TTFT (time to first token, mostly prefill), TPOT (time per output token, the decode speed), and throughput (tokens per second for all users together). The whole of Track 5 is about making these better.
The life of a model
An LLM product is built in stages. Each stage has its own data, loss and track in this course.
The difference between a base model and an assistant is easy to see. Ask a base model “What is the capital of France?” and it may continue with “What is the capital of Germany? What is the capital of Spain?”, because lists of questions are common on the web. An assistant answers “Paris.” Same architecture, different training data at the end.
Three families of transformers
All LLMs use transformer blocks, but they differ in which tokens may look at which. That is set by the attention mask. Figure 7 shows the three main families.
| Family | Who sees whom | Trained with | Good at | Examples |
|---|---|---|---|---|
| Encoder-only | Every token sees every token | Fill in hidden words (masked LM) | Understanding: classification, embeddings for search | BERT, RoBERTa |
| Decoder-only | Each token sees only the past | Next-token prediction | Writing, chat, almost all modern LLMs and VLMs | GPT, Llama, Qwen, Gemma |
| Encoder–decoder | Encoder: all; decoder: past + all of the encoder | Rebuild hidden spans; translate | Input-to-output tasks: translation, speech to text | T5, BART, Whisper |
Why did decoder-only win for large models? One model and one loss do everything; every token in the training text gives a training signal; and the same model can both read and write. Session 23 discusses the trade-offs in depth.
One model, many tasks
A trained LLM is a general engine. We use three different “exits” from it, shown in Figure 8.
Classification by asking for a label word
To decide if a review is positive or negative, write a prompt like “Review: … Sentiment:” and compare the probability of the two label words:
No training is needed. With labeled data, a small classifier on top of the hidden vectors is cheaper and often more accurate (session 55 and session 144).
Retrieval by comparing vectors
Average the final hidden vectors of a text into one vector \(\mathbf{e}\) (an embedding). Two texts with similar meaning get vectors that point the same way. We compare them with cosine similarity:
The value is 1 for the same direction, 0 for unrelated, and \(-1\) for opposite. Search engines store millions of such vectors and return the closest ones to the query (session 150 and session 151).
Images and video become tokens too
A vision-language model (VLM) cuts an image into small patches, turns each patch into a vector with a vision encoder, and puts these vectors in front of the text tokens. The LLM then reads one long sequence:
A video is many frames, so it becomes many such patch groups along time. The same next-token engine can then describe a scene, answer a question about a match, or write a timestamp.
| Video task | Input → output | Which exit | Where in the course |
|---|---|---|---|
| Text-to-video retrieval | “corner kick that leads to a goal” → ranked clips | Embeddings (exit 2) | Session 208 |
| Action recognition | clip → “shot”, “pass”, “foul” | Classifier (exit 3) or label words (exit 1) | Session 203 |
| Key-moment detection | full match → start and end times of goals and saves | Classifier per segment, or timestamps as text | Session 207 |
| Video fingerprinting | clip → “same broadcast as clip 4,127?” | Special copy-detection embeddings, not meaning | Session 209 |
| Video segmentation | clip + a click on a player → a mask in every frame | A separate mask decoder with memory (SAM 2) | Session 204 |
| Video question answering | clip + “who touched the ball last?” → answer | Generate (exit 1), often with retrieval first | Session 210 |
Pseudo-code
Here is the whole session as two short algorithms. Read them slowly: each line is a box from Figure 2.
- input: a batch of token sequences \(X \in \{1..|V|\}^{B \times (T+1)}\), weights \(\theta\), learning rate \(\eta\)
- \(X_{\text{in}} \leftarrow X[:, 1{:}T]\), \(\;Y \leftarrow X[:, 2{:}T{+}1]\) // targets are the inputs shifted by one
- \(H \leftarrow E[X_{\text{in}}]\) // embedding lookup, shape \((B, T, d)\)
- for \(\ell = 1\) to \(L\) do
- \(H \leftarrow H + \operatorname{Attn}_\ell(\operatorname{Norm}(H))\) // with causal mask: no peeking
- \(H \leftarrow H + \operatorname{MLP}_\ell(\operatorname{Norm}(H))\)
- \(Z \leftarrow \operatorname{Norm}(H)\, W_U\) // logits for every position, shape \((B, T, |V|)\)
- \(\mathcal{L} \leftarrow -\frac{1}{BT}\sum_{b,t} \log \softmax(Z_{b,t})[Y_{b,t}]\) // cross-entropy (1.9)
- \(g \leftarrow \nabla_\theta \mathcal{L}\) // backpropagation
- \(\theta \leftarrow \theta - \eta\, g\) // in practice: AdamW with clipping and a schedule
- input: prompt tokens \(x_{1..n}\), temperature \(\tau\), max new tokens \(K\)
- for \(k = 1\) to \(K\) do
- \(\mathbf{z} \leftarrow \text{model}(x_{1..n})[\text{last position}]\) // one score per vocabulary token
- if \(\tau = 0\) then \(x_{n+1} \leftarrow \argmax_i z_i\) // greedy
- else \(x_{n+1} \sim \softmax(\mathbf{z}/\tau)\) // sample, formula (1.8)
- \(n \leftarrow n + 1\)
- if \(x_n\) is the end-of-text token then break
- return \(x_{1..n}\)
And the same ideas in real PyTorch. The first block computes the training loss with the shift by one; the second is a simple generation loop.
import torch import torch.nn.functional as F torch.manual_seed(0) V, d, T = 1000, 64, 8 # vocabulary size, vector width, sequence length tokens = torch.randint(0, V, (2, T + 1)) # 2 sequences of T+1 token ids x, y = tokens[:, :-1], tokens[:, 1:] # input and target, shifted by one emb = torch.nn.Embedding(V, d) # the table E, formula (1.4) unemb = torch.nn.Linear(d, V, bias=False) # W_U, formula (1.6) h = emb(x) # (2, T, d); a real model runs L blocks here logits = unemb(h) # (2, T, V) loss = F.cross_entropy(logits.reshape(-1, V), y.reshape(-1)) # formula (1.9) loss.backward() print(round(loss.item(), 2), round(torch.exp(loss).item())) # about 7.0 and about 1100: close to log V and V
@torch.no_grad() def generate(model, ids, max_new=20, temperature=1.0, eos_id=None): for _ in range(max_new): logits = model(ids)[:, -1, :] # scores for the next token only if temperature == 0: nxt = logits.argmax(dim=-1, keepdim=True) # greedy else: probs = torch.softmax(logits / temperature, dim=-1) nxt = torch.multinomial(probs, num_samples=1) ids = torch.cat([ids, nxt], dim=1) # add it and repeat if eos_id is not None and nxt.item() == eos_id: break return ids # This loop re-reads the whole sequence every step. Real systems keep a KV cache (session 80).
Common mistakes and failure modes
- “Tokens are words.” No. One word can be several tokens, and token counts differ a lot between languages. Cost, speed and context limits are all counted in tokens.
- “High probability means correct.” No. The model learns likely text. Wrong but typical answers can get high probability (hallucination).
- “Greedy decoding is always best.” Greedy text often repeats itself in loops. For open writing, sampling with a cut-off works better; for exact answers (math, code, labels), low temperature is good.
- “Temperature 0 gives the same output every time.” Usually, but not always: in large batched serving, tiny floating-point differences can change which token is top when two scores are almost equal. [Likely]
- Exposure bias. In training the model always sees true previous tokens; in generation it sees its own, sometimes wrong, tokens. Small errors can grow over long outputs.
- Context limit. The model only sees the last \(T_{\max}\) tokens (for example 8,192 for Llama 3 8B). Anything earlier is simply not there (Track 2 and Track 5 discuss longer context).
Interview questions
Explain an LLM in two sentences.basic
Why do we work with log-probabilities?basic
What is the difference between logits and probabilities? Why softmax?basic
Training is parallel over positions but generation is sequential. Why?core
An 8B model in BF16: how much memory to serve it, and how many FLOPs per token?core
Estimate the compute to pretrain an 8B model on 15T tokens.core
Why is decoding memory-bound at batch size 1, and how do you fix it?advanced
What does temperature do? What happens as it goes to 0 and to infinity?basic
Encoder-only or decoder-only: when would you choose each?core
What is perplexity, and is lower always better?core
Using only the training objective, explain why LLMs hallucinate.advanced
How can a generative LLM do classification or retrieval?core
Exercises
- Compute \(\softmax(1, 2, 3)\) by hand.
- The model gives the correct token probability 0.5. What is the loss? A model has average loss 3.0. What is its perplexity?
- How many parameters are in the Llama 3 8B embedding table?
- Give an upper bound on decode speed for an 8B model on an L4 (300 GB/s) at batch 1, in BF16 and in 4-bit.
- In Figure 1, set the temperature to its lowest value and press “Add the most likely token” until the sentence ends. Then do it again. Why is the sentence the same both times? Now use “Sample” at temperature 2. What changes?
Answerscheck
Key takeaways
- An LLM models \(p_\theta(x_t \mid x_{\lt t})\). The chain rule makes this a full model of text (1.2), and we use logs for stability (1.3).
- The path is: tokens → embedding rows → \(L\) blocks of attention + MLP on a residual stream → last vector × \(W_U\) → logits → softmax → choose a token → repeat.
- Training minimizes cross-entropy (1.9) with targets shifted by one; it is parallel thanks to the causal mask. Perplexity is \(e^{\mathcal{L}}\).
- Training costs about \(6ND\) FLOPs; generating costs about \(2N\) FLOPs per token, but decode speed is limited by memory bandwidth.
- The same trunk serves generation, embeddings (search) and classification, and with image or video tokens it becomes a VLM.
Key formulas
References for this session
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 1: Introduction
- Zhao et al. (2023). A Survey of Large Language Models
- Raschka, Build a Large Language Model (From Scratch), chapter 1: Understanding large language models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language