The PEFT family: adapters, BitFit, prefix tuning, prompt tuning, P-Tuning v2
Full fine-tuning makes a new copy of every weight for every task. Parameter-efficient fine-tuning (PEFT) freezes the pretrained model and trains a very small set of new or chosen numbers instead. This session builds the first PEFT family from the ground up: adapters, BitFit, prompt tuning, prefix tuning, P-Tuning and P-Tuning v2, with their math, their real costs, and a clear way to choose between prompting, a linear probe, PEFT and full fine-tuning.
What you will learn
- Why PEFT exists: storage, memory and serving costs when one base model must serve many tasks
- Adapters (Houlsby): a small bottleneck with a skip connection inside each block, started as the identity, and its parameter count 2md + d + m
- BitFit: train only the bias terms, why it works on BERT, and why it cannot be used on Llama-style models that have no biases
- Prompt tuning and prefix tuning: learned virtual tokens at the input versus learned keys and values at every layer, and the exact attention identity that explains prefix tuning
- P-Tuning (a prompt encoder) and P-Tuning v2 (deep prompts plus a classification head) and why they close the gap at small model sizes
- How to choose between prompting, a linear probe, PEFT and full fine-tuning using data size, number of tasks, latency and model size
Why this matters
In session 55 we trained every weight of a model for one task. That works, but it has a price: each task gets its own full copy of the model. An 8B model in bf16 is about 16 GB. One hundred tasks are about 1.6 TB of weights, and each one needs its own GPU memory when it serves traffic. Parameter-efficient fine-tuning (PEFT) keeps one frozen copy of the pretrained model and trains only a tiny set of numbers per task, often less than 1% of the model.
This session covers the first PEFT family, the methods that came before LoRA. They are still asked about in interviews, they are still used, and they explain why LoRA was designed the way it was (session 57):
- Adapters (Houlsby et al., 2019): add a small bottleneck network inside every block and train only that.
- BitFit (Ben Zaken et al., 2021): add nothing; train only the bias terms that already exist.
- Prompt tuning (Lester et al., 2021) and P-Tuning (Liu et al., 2021): learn a few “virtual tokens”, vectors placed in front of the input.
- Prefix tuning (Li & Liang, 2021) and P-Tuning v2 (Liu et al., 2021): learn virtual keys and values at every layer.
The three families change three different places in the network: new layers (adapters), existing parameters (BitFit), or the input and the attention context (prompts and prefixes). The last section answers the practical question an engineer actually faces: should I write a better prompt, train a linear probe, use PEFT, or fine-tune everything?
Real systems: the Hugging Face peft library implements prompt tuning, prefix tuning and P-Tuning next to LoRA, so one base model can load many small task files. [Likely] Prompt-based PEFT also lets one batch mix requests for different tasks, because the shared weights do not change; only the prepended vectors differ per request. [Certain] The VLM book (Noyan et al., chapter 5) uses PEFT for vision-language post-training for the same reason: the base model is large and the task data is small.
The PEFT idea: freeze the base, train a small piece
Full fine-tuning starts from the pretrained weights \(\theta_0\) and changes all of them. PEFT freezes \(\theta_0\) and trains a small set of task parameters \(\phi\). The model for task \(n\) is the pair \((\theta_0, \phi_n)\). The storage cost for \(N\) tasks then changes like this:
- Read the symbols
- \(\theta_0\)
- the pretrained weights; \(|\theta_0|\) is how many numbers they contain
- \(\phi\)
- the small set of task parameters that PEFT trains (adapters, biases, prompts, prefixes)
- \(N\)
- number of tasks (or customers, or languages) that share the base model
There are three ways to choose \(\phi\), and this session follows them in order:
- Add new modules to the network and train only them (adapters).
- Select a small subset of the existing weights and train only that subset (BitFit trains the biases).
- Add new inputs: learned vectors that the frozen network reads like tokens (prompt tuning, P-Tuning, prefix tuning, P-Tuning v2).
Almost every method also trains a new task head when the task is classification, as in session 55. For example, Houlsby et al. train the adapters, the layer-norm parameters and the final classification layer. [Certain] When we count “trained parameters” below, we count the backbone part and say “plus the head”.
Figure 1 shows where each method lives in one transformer block, and how many numbers it trains for three real models. Choose a method and a model.
- \(n \leftarrow 0\) // full fine-tuning: every box counts
- input: if prompt: \(n \mathrel{+}= p\,d\) // (56.5)
- for \(\ell = 1, \dots, L\):
- LN 1: if BitFit: \(n \mathrel{+}= \) its bias
- attention: if BitFit: \(n \mathrel{+}= \mathbf{b}_Q, \mathbf{b}_K, \mathbf{b}_V, \mathbf{b}_O\); if prefix: \(n \mathrel{+}= 2\,p\,d_{kv}\) // (56.6)
- if adapters: \(n \mathrel{+}= 2md + d + m\) // (56.3)
- LN 2 and MLP: if BitFit: \(n \mathrel{+}= \) their biases
- if adapters: \(n \mathrel{+}= 2md + d + m\)
- output side: other biases (BitFit); the new task head is counted apart
How to read Figure 1
Start with “Full fine-tuning”: every box is trained, and the per-task file is the whole model (16 GB for Llama 3.1 8B). Click “Adapters”: only the two small orange boxes in each block are trained, about 0.4% of Llama 3.1 8B at \(m = 64\). Click “BitFit” with BERT-base: about 0.09% (the small dots). Now switch the model to Llama 3.1 8B: the count drops to zero, because Llama has no bias vectors at all. Qwen2.5 has biases only on the query, key and value projections. Click “Prompt tuning” and “Prefix”: move the length slider and compare. Prompt tuning trains \(p \times d\) numbers; prefix tuning trains keys and values in every layer, so it is about \(2L\) times larger (less with grouped-query attention). Press Play with “BitFit” and BERT-base: layer 1 adds 768 (norm 1), 3,072 (Q, K, V, O biases) and 4,608 (norm 2 and the two MLP biases), 8,448 in total; the other 11 layers add 92,928 more, and the embedding norm and pooler biases bring it to 102,912. With “Full fine-tuning” on Llama 3.1 8B, one layer is 218,112,000 numbers and the embeddings plus output layer are 1,050,677,248.
Adapters: a small bottleneck inside each block
An adapter is a tiny two-layer network: it projects the hidden vector of size \(d\) down to a small size \(m\) (the bottleneck), applies a nonlinearity, and projects back up to \(d\). It has its own skip connection, so its output is “input plus a small correction”.
- Read the symbols
- \(\mathbf{h} \in \R^{d}\)
- the output of a sub-layer (attention or MLP) for one token
- \(W_{\text{down}} \in \R^{m \times d}\)
- the down-projection to the bottleneck size \(m\), with bias \(\mathbf{b}_{\text{down}} \in \R^{m}\)
- \(W_{\text{up}} \in \R^{d \times m}\)
- the up-projection back to size \(d\), with bias \(\mathbf{b}_{\text{up}} \in \R^{d}\)
- \(f\)
- a nonlinearity such as GELU or ReLU
- \(m\)
- the bottleneck size, much smaller than \(d\) (Houlsby et al. chose from 8, 64 and 256)
Where it goes. Houlsby et al. add the adapter twice to each layer: once after the attention sub-layer and once after the feed-forward sub-layer. Each adapter is applied to the sub-layer output after its projection back to size \(d\), and before the residual addition. [Certain] So the block becomes \(\mathbf{x} \leftarrow \mathbf{x} + \mathrm{Adapter}(\mathrm{Attn}(\mathrm{LN}(\mathbf{x})))\), and the same for the MLP. (The original BERT is post-LN; the placement relative to the skip connection is the part that matters.)
How many numbers. One adapter has two matrices and two bias vectors:
The first formula is from the paper. [Certain] The second counts two adapters in each of \(L\) layers. Houlsby et al. used adapters with about 0.5 to 8% of the original parameters, and the bottleneck \(m\) is the single knob that trades quality for size. [Certain]
Start as the identity. Because of the inner skip connection, an adapter with near-zero weights returns \(\mathbf{h}\) almost unchanged. So at the start of training, the network with adapters computes the same function as the pretrained network, and training only has to learn a small correction. Houlsby et al. draw the adapter weights from a zero-mean Gaussian with standard deviation \(10^{-2}\), truncated at two standard deviations, and found quality is robust below \(10^{-2}\) but drops when the initialization is too large. [Certain] Many later implementations simply set \(W_{\text{up}} = 0\), which makes the start exactly the identity (the same trick appears in LoRA). Figure 2 shows why the scale matters.
How to read Figure 2
At the paper's setting (\(\sigma = 10^{-2}\), \(m = 64\)) the correction is about 1% of the input: the network starts almost exactly as the pretrained one. Move \(\sigma\) to \(10^{-1}\): the correction becomes as large as the input itself, so the adapter destroys the pretrained features before training even begins. Because both projections scale with \(\sigma\), the correction grows like \(\sigma^2\): each step of 0.5 on the slider multiplies it by about 10. The bottleneck slider changes the parameter count (formula 56.3) roughly in proportion to \(m\). Houlsby et al. found mean validation accuracy almost flat across sizes 8, 64 and 256 (86.2%, 85.8%, 85.7%), so small adapters are often enough. [Certain]
BERT-base (\(d = 768\), \(L = 12\), 109,482,240 parameters). With \(m = 64\), one adapter has \(2 \cdot 64 \cdot 768 + 768 + 64 = 99{,}136\) parameters. Two per layer and 12 layers give \(24 \times 99{,}136 = 2{,}379{,}264\), or 2.17% of the model.
Llama 3.1 8B (\(d = 4096\), \(L = 32\), 8,030,261,248 parameters). With \(m = 64\): \(64 \times (2 \cdot 64 \cdot 4096 + 4096 + 64) = 33{,}820{,}672\) parameters, 0.42% of the model. In bf16 that is a 67.6 MB file per task, compared with 16.06 GB for a full copy.
Results. On GLUE with BERT-large, adapters with sizes 8 to 256 scored 80.0 against 80.4 for full fine-tuning, training 3.6% of the parameters per task, and needing 1.3 times the parameters of one model for all 9 tasks instead of 9 times. [Certain]
The cost: latency. Adapters are extra layers in series. They cannot be merged into the frozen weights, because of the nonlinearity \(f\). At large batch sizes the extra work is small, but at batch size 1 it shows. The LoRA paper measured one forward pass of GPT-2 medium on one GPU: 19.8 ms without adapters and 23.9 to 25.8 ms with two adapter variants (+20.7% and +30.3%) at batch size 1 and length 128, but only +2.2% and +3.0% at batch size 32 and length 512. [Certain] This latency is the main reason LoRA was designed as a linear update that can be merged (session 57).
Which layers matter? Houlsby et al. removed trained adapters one layer at a time. Removing adapters from the lower layers (0 to 4) on MNLI barely changed the result; the upper layers carried most of the effect. Removing all adapters dropped MNLI to 37%, the majority-class score. [Certain] Later adapter variants put one adapter only after the MLP (Lin et al., 2020, called AdapterL in the LoRA paper) to save latency. [Certain]
BitFit: train only the biases
Every linear layer computes \(\mathbf{y} = W\mathbf{x} + \mathbf{b}\). BitFit (“bias-terms fine-tuning”) freezes every weight matrix \(W\) and every layer-norm gain, and trains only the additive bias vectors \(\mathbf{b}\): the biases of the query, key, value and output projections, of the two MLP layers, and of the layer norms. [Certain] Nothing new is added to the network, so inference cost does not change at all.
- Read the symbols
- \(W\)
- a frozen weight matrix (no gradient is stored for it)
- \(\mathbf{b}\)
- the bias vector, the only trained part of the layer
- \(\partial \mathcal{L} / \partial \mathbf{y}_t\)
- the gradient arriving from above at token \(t\); the bias gradient is just its sum over tokens
A bias adds the same shift to every token. So BitFit can move each feature up or down and change which neurons in the MLP are active, but it cannot rotate or mix features. The surprise is how far that goes. On BERT-base, biases are 0.09% of the parameters, and the GLUE validation average was 82.4 for BitFit against 82.3 for full fine-tuning; a frozen model with only a trained head scored 62.1. [Certain] On BERT-large (biases 0.08%) BitFit scored 84.2 against 84.1. [Certain] The authors read this as evidence that fine-tuning mostly exposes knowledge the model already learned in pretraining, more than it teaches new knowledge. [Certain]
Two more findings are useful. Training only two bias vectors per layer, the query bias \(\mathbf{b}_q\) and the bias of the first MLP layer (\(\mathbf{b}_{m2}\) in the paper's notation, intermediate.dense.bias in Hugging Face BERT), keeps most of the quality with 0.04% of the parameters: 81.1 on average. [Certain] And training the same number of randomly chosen weights instead of biases is clearly worse (78.5). [Certain] BitFit also needs a larger learning rate: the paper used \(10^{-4}\) to \(10^{-3}\) for bias-only training against \(10^{-5}\) to \(5 \times 10^{-5}\) for full fine-tuning, and reports that optimization was more stable. [Certain]
Training cost. BitFit saves optimizer memory (no Adam states for \(W\)) and the weight-gradient matrix multiply in every layer. But the backward pass must still carry \(\partial \mathcal{L} / \partial \mathbf{x}\) down through every layer, because every layer has a trained bias. So activation memory is about the same as full fine-tuning. This is true for all methods here: if the first trained parameter sits in layer 1, backpropagation must reach layer 1.
Prompt tuning: learn the input, not the model
With prompting, we write words in front of the input (“Classify the sentiment:”). Each word becomes an embedding vector, and the model reads those vectors. Prompt tuning skips the words: it learns the vectors directly. These learned vectors are called a soft prompt or virtual tokens, because they live in the same space as token embeddings but do not need to be the embedding of any real token.
- Read the symbols
- \(P\)
- the soft prompt: \(p\) trainable vectors of size \(d\), the only trained numbers
- \(E(x_t)\)
- the frozen embedding of real input token \(x_t\)
- \([\,\cdot\,;\,\cdot\,]\)
- concatenation along the sequence axis: the prompt vectors come first
- \(p\)
- prompt length (number of virtual tokens)
The model is not changed at all. Training sends gradients back through the whole frozen network into \(P\) only. Lester et al. used T5 models up to XXL (11B parameters), a default of 100 virtual tokens, a learning rate of 0.3 with Adafactor, and 30,000 steps per task. [Certain] That learning rate is huge compared with fine-tuning (about \(10^{-5}\)); the soft prompt is a small set of free vectors and needs large steps to move.
The scale result. The key finding: prompt tuning gets closer to full model tuning as the model grows, and at T5-XXL (11B) it matches it on SuperGLUE. [Certain] With 5 tokens on T5-XXL (\(d = 4096\)) the prompt has only 20,480 parameters, over five orders of magnitude fewer than the 11B model. [Certain] For small models the gap is large, as Figure 4 shows.
Details that matter (all from the paper's ablations [Certain]):
- Length. Going from 1 token to more is critical for most sizes; beyond 20 tokens gains are small. The XXL model works even with a single token.
- Initialization. Random initialization is worst. Initializing from embeddings of real vocabulary words, or of the class label words, is better, and the difference disappears at XXL size.
- The base model's objective. T5 was pretrained with span corruption, which is a poor fit. Continuing T5's training with a plain language-model objective for up to 100K steps (“LM adaptation”) made prompt tuning work much better.
- Mixed-task batches. Because the model is shared, one batch can hold examples for many tasks, each with its own prompt. The paper also ensembles five prompts on one frozen model.
P-Tuning (v1). Liu et al. (2021, “GPT Understands, Too”) had the same idea for knowledge probing and SuperGLUE, with two changes. First, the virtual tokens are placed inside a template next to real “anchor” words, for example around “The capital of [Britain] is [MASK]”. Second, the vectors are produced by a small prompt encoder (an LSTM or an MLP) instead of being free parameters, which makes training more stable. [Certain] They motivate it with how fragile manual prompts are: changing a single word in a prompt could drop LAMA accuracy by 20 points. [Certain] Unlike Lester et al., P-Tuning was also used together with fine-tuning the model.
Prefix tuning: virtual keys and values at every layer
Prompt tuning changes only the input layer. Its vectors must travel through all the frozen layers before they affect the output, and each layer can transform them in ways the prompt cannot control. Prefix tuning gives every layer its own learned vectors. In each attention layer \(\ell\), it adds \(p\) learned keys and \(p\) learned values in front of the real ones:
- Read the symbols
- \(X_\ell\)
- the hidden states of the real tokens entering layer \(\ell\)
- \(W_Q, W_K, W_V\)
- the frozen query, key and value projections of that layer
- \(P^{K}_{\ell}, P^{V}_{\ell} \in \R^{p \times d_{kv}}\)
- the learned prefix keys and values of layer \(\ell\); \(d_{kv}\) is the total key width (\(n_{kv}\) heads times head size \(d_h\))
Every real token can attend to the prefix as if \(p\) extra tokens came first, but the prefix itself has no query and is not computed from anything: it is just stored. The trained size is \(|\phi| = 2\,L\,p\,d_{kv}\). For Llama 3.1 8B with grouped-query attention (8 key-value heads of size 128, so \(d_{kv} = 1024\)) and \(p = 20\), this is \(2 \cdot 32 \cdot 20 \cdot 1024 = 1{,}310{,}720\) parameters, 0.016% of the model. A model with full multi-head attention would need 4 times more. At inference the prefix is exactly a pre-filled part of the KV cache (session 26): 2.5 MiB per sequence in bf16 for this example.
Li & Liang applied it to GPT-2 for table-to-text and to BART for summarization. Training about 0.1% of the parameters matched full fine-tuning with all the data, and did better in low-data settings and on topics not seen in training. [Certain] Best prefix lengths were about 10 for table-to-text and 200 for summarization; longer prefixes gave lower training loss but slightly worse test scores. [Certain]
Why a prefix works: one exact identity
He et al. (2021) showed that adding a prefix to attention is the same as mixing the original attention output with an attention over the prefix alone. Take one query \(\mathbf{q}\), real keys and values \(K, V\), and prefix keys and values \(P^K, P^V\). Split the softmax sum into its prefix part and its real part:
- Read the symbols
- \(\lambda(\mathbf{q})\)
- the share of attention that this query gives to the prefix (between 0 and 1); it depends on the query, so each token mixes differently
- \(\mathbf{p}^K_j\)
- prefix key \(j\); \(\mathbf{k}_i\) is real key \(i\)
- \(d_h\)
- the head size used to scale scores
The proof is two lines: the softmax denominator over all keys is the sum of the two partial sums, so the weights on the real keys are the old weights times \(1 - \lambda\), and the weights on the prefix are the prefix-only weights times \(\lambda\). So a prefix adds a vector \(\mathrm{Attn}(\mathbf{q}, P^K, P^V)\) to each token's attention output, with a gate \(\lambda\) chosen by the query. That is very close to what an adapter does: add a small learned correction to a sub-layer output. He et al. use this view to put adapters, prefix tuning and LoRA in one framework. [Certain] Figure 3 lets you check the identity.
How to read Figure 3
With \(p = 0\) the output is the original one. Add prefix keys: the real weights shrink by the same factor \(1 - \lambda\), so their ratios do not change, and the output slides toward the prefix output. A higher prefix score or a longer prefix both increase \(\lambda\). The readout computes the output twice, directly with one softmax over all keys and through formula (56.7), and prints the difference: it is at the level of rounding error. In a real model, the prefix keys are learned so that \(\lambda\) is large for the queries that need a change and small for the others.
A query has scores 2, 1 and 0 with three real keys, whose (1-D) values are 1, 0 and −1. Softmax weights: \(e^2, e^1, e^0\) divided by \(e^2 + e + 1 = 11.107\), so 0.665, 0.245 and 0.090. The original output is \(0.665 - 0.090 = 0.575\).
Add two prefix keys with score 1.5 each and value −2. Their partial sum is \(2e^{1.5} = 8.963\), so \(\lambda = 8.963 / (8.963 + 11.107) = 0.447\) by (56.8). The prefix-only output is −2. By (56.7) the new output is \((1 - 0.447)(0.575) + 0.447 \cdot (-2) = -0.575\). Two stored vectors flipped the sign of this token's attention output, and nothing in the frozen model changed.
Reparameterization. Li & Liang found that training \(P\) directly was unstable and very sensitive to the learning rate and initialization. [Certain] So during training they produce it from a smaller matrix through an MLP:
with \(k = 512\) for table-to-text and \(k = 800\) for summarization. After training, the MLP is thrown away and only the computed \(P\) is stored. [Certain] So the MLP changes how the optimizer sees the problem, not what is saved. P-Tuning's LSTM prompt encoder plays the same role. Li & Liang also tested “embedding-only” (prompt tuning style: learn only input vectors) on E2E: BLEU 62.2 with 10 tokens against 69.7 for the full prefix. [Certain] Deep prompts are much more expressive at the same model size.
P-Tuning v2: deep prompts that work at every size
By 2021 the picture was confusing. Prompt tuning matched fine-tuning at 10B parameters but was much worse on models of a few hundred million, and it failed on hard sequence labeling tasks such as named-entity recognition and extractive question answering, where every token needs its own label (session 55). P-Tuning v2 (Liu et al., 2021) is not a new mechanism. It is prefix tuning (“deep prompt tuning”) with a careful recipe for language understanding tasks [Certain]:
- Prompts at every layer, as in prefix tuning. This raises the trained fraction from about 0.01% to 0.1% to 3%, and puts the prompts close to the output where they have a direct effect.
- Reparameterization is optional. The MLP helped on some datasets (RTE, CoNLL04) and hurt on others (CoNLL12), so it is a hyperparameter.
- Prompt length depends on the task: simple classification prefers short prompts (under 20), hard sequence labeling prefers around 100.
- A plain classification head on the token representations, as in BERT, instead of a “verbalizer” (making the language-model head predict label words). Verbalizers do not work for sequence labeling.
- Optional multi-task pretraining of the prompts, which gives a better starting point.
- Read the symbols
- \(\mathbf{h}^{(L)}_t\)
- the frozen model's last hidden state for token \(t\), computed with the deep prompts in every layer
- \(W_c, \mathbf{b}_c\)
- a new, randomly initialized linear classification head (for sentence tasks, applied to one pooled or [CLS] vector)
With this recipe the paper reports results comparable to fine-tuning from 330M to 10B parameters, with only 0.1% to 3% of the parameters tuned, on SuperGLUE and on sequence labeling. [Certain] Figure 4 shows the SuperGLUE numbers from the paper.
How to read Figure 4
On RTE, prompt tuning on the input only is almost 28 points behind fine-tuning on RoBERTa-large (58.8 versus 86.6), while P-Tuning v2 is ahead (89.5). At 10B the shallow method has almost caught up (89.9 versus 93.1). Click through the tasks: the blue bars climb with model size, which is Lester et al.'s scale result; the orange bars stay near the grey bars at every size, which is P-Tuning v2's result. A few tasks (WiC, WSC at 10B) are slightly below fine-tuning: “comparable” does not mean “always equal”.
Comparing the family
Before the table, run all four methods through the same small block with real numbers. Figure 5 is one attention sub-layer with \(d = 4\), three frozen tokens and one head. The last token's output \(\mathbf{y}\) should become a target \(\mathbf{y}^*\) for a new task, with loss \(L = \lVert \mathbf{y} - \mathbf{y}^* \rVert^2\). Each method gets its trained numbers in a different place: BitFit in the bias rows, prompt tuning in new input rows, prefix tuning in new key and value rows, adapters in a new module after the output projection. Everything else is frozen. Step through the forward pass and watch where the orange numbers enter; then switch to “after training” to see what 300 Adam steps on those numbers alone can do.
- \(X \leftarrow [\mathbf{x}_1; \mathbf{x}_2; \mathbf{x}_3]\) // frozen embeddings
- prompt: \(X \leftarrow [P; X]\) // trained rows (56.5)
- \(\mathbf{q} \leftarrow \mathbf{x}_3 W_Q + \mathbf{b}_Q\)
- \(K \leftarrow X W_K + \mathbf{b}_K,\; V \leftarrow X W_V + \mathbf{b}_V\) // BitFit trains the \(\mathbf{b}\)
- prefix: \(K \leftarrow [P^K; K],\; V \leftarrow [P^V; V]\) // (56.6)
- \(s_j \leftarrow \mathbf{q} \cdot \mathbf{k}_j / \sqrt{d}\)
- \(\mathbf{w} \leftarrow \softmax(\mathbf{s})\) // prefix share \(= \lambda\) (56.8)
- \(\mathbf{a} \leftarrow \sum_j w_j \mathbf{v}_j\)
- \(\mathbf{o} \leftarrow \mathbf{a} W_O + \mathbf{b}_O\)
- adapter: \(\mathbf{o} \leftarrow \mathbf{o} + W_{\text{up}} f(W_{\text{down}} \mathbf{o} + \mathbf{b}_{\text{down}}) + \mathbf{b}_{\text{up}}\) // (56.2)
- \(\mathbf{y} \leftarrow \mathbf{x}_3 + \mathbf{o}\); \(L \leftarrow \lVert \mathbf{y} - \mathbf{y}^* \rVert^2\)
- \(\theta \leftarrow \theta - \eta\,\mathrm{Adam}(\partial L / \partial \theta)\) // \(\theta\) = the orange numbers only
How to read Figure 5
Press Play with “Adapter”: the forward pass runs line by line and the step box shows every formula with its numbers. The adapter's correction is exactly zero at the start, so its output equals the frozen block's output (loss 1.4890). Switch to “Prompt tuning”: two new orange rows appear in the input, they get their own keys and values through the frozen projections, and they already change the output before training (loss 2.7877), because the prompt starts from word embeddings. “Prefix tuning” adds orange rows directly to K and V; the readout shows the prefix share λ. Now choose “After 300 Adam steps”: BitFit and the adapter reach the target (loss near 0), prefix tuning with \(p = 2\) also does, and prompt tuning with \(p = 1\) (only 4 numbers, which must pass through the frozen \(W_K\) and \(W_V\)) stops at 0.0568. The last step shows the gradient: only the orange numbers have one.
This tiny example fits one output vector, so every method with enough numbers can reach it. It is a picture of where each method acts, not a benchmark: on real tasks the ranking depends on the data and the model size (Figure 4).
| Method | What is trained | Trained size | Inference cost | Mixed tasks in one batch |
|---|---|---|---|---|
| Full fine-tuning | Every weight | 100% per task | None extra | No (one model per task) |
| Linear probe | Only a new head on frozen features | \(d \times \text{classes}\) | None extra | Yes (heads are tiny) |
| Adapters | Bottleneck modules, 2 per layer (56.3) | 0.1 to 8% | Extra serial layers: +20 to 30% at batch 1 (GPT-2 medium) | Possible, but needs per-example routing of adapter weights |
| BitFit | Existing bias vectors | 0.04 to 0.09% (BERT); 0 for Llama | None | Possible (add per-example bias) |
| Prompt tuning / P-Tuning | \(p\) input vectors (56.5) | Under 0.01% for large models | \(p\) extra tokens in every layer | Yes, naturally |
| Prefix tuning / P-Tuning v2 | \(p\) keys and values per layer (56.6) | 0.01 to 3% | \(p\) extra KV-cache entries per sequence; shorter usable context | Yes (per-request prefix in the KV cache) |
Two costs are shared by all of them. Training memory: the frozen weights still sit on the GPU, and activations are still stored for backpropagation; PEFT removes only the optimizer states and gradients of the frozen weights. For an 8B model, full fine-tuning with AdamW in mixed precision needs about 16 bytes per parameter, 128 GB before activations (session 38); with PEFT it is about 16 GB of bf16 weights plus about 16 bytes per trained parameter (0.54 GB for the 33.8M adapter above). This is what lets an 8B model be adapted on one GPU. Optimization: prompt and prefix methods are harder to tune. The LoRA paper found prefix tuning “difficult to optimize”, with quality that changes non-monotonically with the number of trained parameters, and notes that the prefix takes up part of the sequence length. [Certain]
Putting it together: algorithms and code
- input: pretrained model \(\theta_0\), method, size (\(m\) or \(p\)), task head shape
- load \(\theta_0\); set
requires_grad = Falsefor every parameter // frozen and shared - if adapters: in each block insert Adapter after attention and after MLP, before the residual add; init \(W_{\text{up}} \approx 0\) // (56.2), identity at start
- if BitFit: set
requires_grad = Truefor every parameter whose name ends with “bias” // check that there are any - if prompt tuning: \(P \leftarrow\) embeddings of real words (or class labels), shape \(p \times d\) // (56.5)
- if prefix / P-Tuning v2: for each layer create \(P^K_\ell, P^V_\ell\) (optionally through an MLP) // (56.6), (56.9)
- add a new task head if the task needs one (classification, tagging) // (56.10)
- count trained parameters and print them // catch “0 trained” and “everything trained” bugs
- return model, list of trained parameters
- for each task \(n\):
- \(\text{optimizer} \leftarrow \mathrm{AdamW}\) over the trained parameters only // no states for \(\theta_0\)
- use a larger learning rate than full fine-tuning (about \(10^{-4}\) to \(10^{-3}\); up to 0.3 for prompt tuning with Adafactor)
- train with the task loss; choose the checkpoint on the validation set
- if reparameterized: compute \(P = \mathrm{MLP}(P')\) once and drop the MLP
- save \(\phi_n\) only (plus head) // kilobytes to megabytes, (56.1)
- serve: load \(\theta_0\) once; for each request, look up its task \(n\)
- prompt / prefix: prepend \(P_n\) to its input or its KV cache; batch requests of different tasks together
- adapters / BitFit: apply task \(n\)'s modules or biases (group requests by task if the kernel cannot mix)
The PyTorch program below has no downloads and runs on a CPU in about 10 seconds. It “pretrains” a tiny 2-layer encoder on one task (count six token types), then adapts it to a new task (is token 7 more frequent than token 11?) with each method, always with a new trained head. It implements adapters, BitFit, prompt tuning and prefix tuning by hand, in the same block, so you can see exactly where each one enters.
import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
V, D, H, L, T = 24, 32, 4, 2, 16 # vocab, width, heads, layers, length
class Block(nn.Module):
def __init__(s):
super().__init__()
s.ln1, s.ln2 = nn.LayerNorm(D), nn.LayerNorm(D)
s.qkv, s.o = nn.Linear(D, 3 * D), nn.Linear(D, D)
s.up, s.down = nn.Linear(D, 4 * D), nn.Linear(4 * D, D)
s.ad1 = s.ad2 = None # adapters (Houlsby): one per sub-layer
s.pk = s.pv = None # prefix keys / values for this layer
def attn(s, x):
B, N, _ = x.shape
q, k, v = s.qkv(x).view(B, N, 3, H, D // H).transpose(1, 3).unbind(2) # (B,H,N,dh)
if s.pk is not None: # prefix tuning: prepend learned K, V
k = torch.cat([s.pk.expand(B, -1, -1, -1), k], 2)
v = torch.cat([s.pv.expand(B, -1, -1, -1), v], 2)
y = F.scaled_dot_product_attention(q, k, v) # bidirectional encoder
return s.o(y.transpose(1, 2).reshape(B, N, D))
def forward(s, x):
a = s.attn(s.ln1(x)); a = a + s.ad1(a) if s.ad1 else a # adapter before the residual add
x = x + a
m = s.down(F.gelu(s.up(s.ln2(x)))); m = m + s.ad2(m) if s.ad2 else m
return x + m
class Net(nn.Module):
def __init__(s, n_out):
super().__init__()
s.emb, s.pos = nn.Embedding(V, D), nn.Parameter(0.02 * torch.randn(T, D))
s.blocks = nn.ModuleList(Block() for _ in range(L))
s.lnf, s.head = nn.LayerNorm(D), nn.Linear(D, n_out)
s.prompt = None # prompt tuning: learned input vectors
def forward(s, ids):
x = s.emb(ids) + s.pos
if s.prompt is not None:
x = torch.cat([s.prompt.expand(len(ids), -1, -1), x], 1)
for b in s.blocks: x = b(x)
return s.head(s.lnf(x).mean(1)) # mean-pool, then classify
def adapter(m): # down-project, nonlinearity, up-project
a = nn.Sequential(nn.Linear(D, m), nn.GELU(), nn.Linear(m, D))
nn.init.normal_(a[0].weight, std=1e-2); nn.init.zeros_(a[0].bias)
nn.init.zeros_(a[2].weight); nn.init.zeros_(a[2].bias) # starts as the identity
return a
def data(n, g, task): # task A: count of 6 tokens; task B: is count(7) > count(11)?
ids = torch.randint(0, V, (n, T), generator=g)
c = torch.stack([(ids == t).sum(1) for t in range(6)], 1).float()
return ids, (c if task == "A" else ((ids == 7).sum(1) > (ids == 11).sum(1)).long())
def train(net, params, task, steps, lr, seed):
g = torch.Generator().manual_seed(seed); opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
for _ in range(steps):
x, y = data(64, g, task); out = net(x)
loss = F.mse_loss(out, y) if task == "A" else F.cross_entropy(out, y)
opt.zero_grad(); loss.backward(); opt.step()
# 1) "pretrain" a backbone on task A (counting tokens 0..5)
base = Net(6); train(base, base.parameters(), "A", 600, 3e-3, seed=1)
sd = {k: v for k, v in base.state_dict().items() if not k.startswith("head")}
xt, yt = data(2000, torch.Generator().manual_seed(99), "B") # fixed test set for task B
def run(method):
torch.manual_seed(2); net = Net(2); net.load_state_dict(sd, strict=False)
for p in net.parameters(): p.requires_grad = False
if method == "adapter":
for b in net.blocks: b.ad1, b.ad2 = adapter(4), adapter(4)
if method == "prompt": net.prompt = nn.Parameter(net.emb.weight[:4].clone().unsqueeze(0)) # init from real tokens
if method == "prefix":
for b in net.blocks:
b.pk = nn.Parameter(0.5 * torch.randn(1, H, 4, D // H)); b.pv = nn.Parameter(0.5 * torch.randn(1, H, 4, D // H))
for n, p in net.named_parameters():
if method == "full" or n.startswith("head") or ".ad" in n or n in ("prompt",) or n.endswith((".pk", ".pv")) \
or (method == "bitfit" and n.endswith("bias")):
p.requires_grad = True
params = [p for p in net.parameters() if p.requires_grad]
lr = 3e-4 if method == "full" else 3e-3 # PEFT uses a larger learning rate
train(net, params, "B", 400, lr, seed=3)
with torch.no_grad(): acc = (net(xt).argmax(1) == yt).float().mean().item()
n_tr = sum(p.numel() for p in params) - sum(p.numel() for p in net.head.parameters())
return n_tr, acc
total = sum(p.numel() for k, p in base.state_dict().items() if not k.startswith("head"))
print("backbone parameters:", total)
for m in ["probe", "bitfit", "adapter", "prompt", "prefix", "full"]:
n, acc = run(m); print(f"{m:8s} trained (besides head): {n:6d} ({100*n/total:5.2f}%) test acc {acc:.3f}")
# backbone parameters: 26752
# probe trained (besides head): 0 ( 0.00%) test acc 0.751
# bitfit trained (besides head): 736 ( 2.75%) test acc 0.906
# adapter trained (besides head): 1168 ( 4.37%) test acc 0.923
# prompt trained (besides head): 128 ( 0.48%) test acc 0.758
# prefix trained (besides head): 512 ( 1.91%) test acc 0.803
# full trained (besides head): 26752 (100.00%) test acc 1.000
Read the output with care: this is a 27k-parameter toy, so percentages are much larger than in real models, and full fine-tuning wins easily because the model is tiny and data is unlimited. But the ordering repeats what the papers found for small models: input-only prompt tuning barely beats the linear probe (0.758 versus 0.751), deep prefixes help more (0.803), and adapters and BitFit, which change every layer's computation directly, get close to full fine-tuning. The bias count is checkable by hand: per block 96 + 32 + 128 + 32 (linear biases) + 32 + 32 (layer-norm biases), so \(2 \times 352 + 32 = 736\). In practice you would use the peft library (PromptTuningConfig, PrefixTuningConfig, PromptEncoderConfig for P-Tuning) instead of writing these by hand. [Likely]
How to choose: prompting, linear probe, PEFT or full fine-tuning
The methods in this session sit on a ladder. Each step up trains more, costs more, and can change the model more:
- Prompting (zero-shot or few-shot, session 143): no training at all. Best when you have very few labeled examples, the task is close to what the model already does, and you can afford long prompts at every request.
- Linear probe: freeze the model, train only a head on its features (as in session 55). Very cheap and fast to serve; works when the frozen features already separate the classes. The BitFit paper's frozen baseline (62.1 GLUE average versus 82.3) shows it often does not.
- PEFT: thousands to a few hundred thousand examples, many tasks or customers on one base model, limited GPU memory for training. Choose prompt or prefix methods when you need mixed-task batches; adapters or LoRA when quality matters more; avoid serial adapters when batch-1 latency is critical.
- Full fine-tuning: a large dataset, one or few tasks, a task far from pretraining (a new language, a new modality, a big behavior change), and the compute to train and store full copies. Also the reference you compare PEFT against.
These are rules of thumb, not laws. [Likely] The only reliable answer comes from a small experiment on your own data: try the cheaper option first and move up only if the validation score demands it. Figure 6 turns the rules into a small calculator.
How to read Figure 6
With 100 tasks on Llama 3.1 8B, full fine-tuning needs 1.61 TB of weights, while 100 prompt-tuned tasks add only 16 MB to one 16 GB base model. Drag the number of examples down below about 100: the suggestion becomes prompting, because there is too little data to train anything reliably. Drag it up to a million with one task and tick “far from pretraining”: full fine-tuning becomes worth its cost. Tick “batch-1 latency”: serial adapters are no longer suggested.
Where this is used
| Setting | Typical method | Why |
|---|---|---|
| Many customers or tasks on one hosted model | Prompt tuning, prefix tuning, LoRA | One base copy in GPU memory; tiny per-task files; prompt-style methods can mix tasks in one batch (56.1) |
| BERT-style classifiers on small GLUE-like data | BitFit, adapters | BitFit matched full fine-tuning on BERT with 0.08 to 0.09% of the parameters |
| Multilingual and multi-domain models | Adapters (one per language or domain) | New languages or domains are added without touching earlier ones; adapters can be stacked or swapped [Likely] |
| Sequence labeling: NER, extractive QA, tagging | P-Tuning v2, adapters | Deep prompts and a token-level head; input-only prompts fail here |
| Generation with little data (table-to-text, summaries) | Prefix tuning | Beat fine-tuning in low-data settings and on unseen topics in Li & Liang |
| VLMs: adapting to a new image domain or document type | Adapters or LoRA on the language model; sometimes learned visual prompts | The base VLM is large and the domain data is small (session 189) [Likely] |
| Video classification and action recognition with a frozen video encoder | Linear probe first, then adapters inside the encoder | Video backbones are expensive to fine-tune fully; adapters keep one shared backbone for many label sets [Likely] |
| Video retrieval and fingerprinting | Linear probe or small adapters on embeddings | The index is built from frozen embeddings; a full fine-tune would force re-embedding the whole library [Likely] |
| Key-moment detection and Video-RAG with a Video-LLM | Prefix or prompt tuning per domain; LoRA for bigger changes | Per-domain behavior (sports highlights, lectures) without separate model copies (session 205) [Likely] |
Common mistakes
- Using BitFit on a model without biases. Llama-style models have none, so “BitFit” trains only the head. Print the trained parameter count before training.
- Forgetting to freeze (or to unfreeze). A wrong
requires_gradloop either trains the whole model (no savings, a huge optimizer state) or trains nothing except the head. Count trained parameters. - Using the full fine-tuning learning rate. PEFT parameters usually need much larger rates: \(10^{-4}\) to \(10^{-3}\) for BitFit, 0.3 with Adafactor for prompt tuning in Lester et al. With \(10^{-5}\), prompt tuning seems not to learn.
- Initializing adapters too large. A large random \(W_{\text{up}}\) breaks the pretrained function at step 0 (Figure 2). Start at or near the identity.
- Random soft-prompt initialization on a small model. Initialize from real word embeddings or class-label embeddings, and use enough tokens (20 or more).
- Expecting input-only prompt tuning to work on a 300M model or on tagging tasks. Use deep prompts (P-Tuning v2), adapters or LoRA there.
- Ignoring the context cost of prompts and prefixes. \(p\) virtual tokens use \(p\) positions of the context window and \(p\) KV-cache entries per sequence; with long inputs, check that real tokens are not cut.
- Saving the reparameterization MLP instead of the final prefix, or forgetting the new task head in the saved file. The task file must contain everything that differs from the base model.
Interview questions
What is parameter-efficient fine-tuning and why do we need it?basic
Describe the Houlsby adapter: shape, placement, initialization and parameter count.core
Why do adapters add inference latency while LoRA does not?core
What does BitFit train, why does it work, and would you use it on Llama 3?core
Prompt tuning versus prefix tuning: what exactly differs?core
Show that a prefix in attention is a gated mix of two attention outputs.advanced
Why did prompt tuning work at 11B but not at 300M, and what did P-Tuning v2 change?advanced
What is the reparameterization trick in prefix tuning and P-Tuning, and what is saved?core
Does PEFT reduce activation memory in training? What does it reduce?core
A team wants to serve 500 customer-specific variants of an 8B model. What do you suggest?advanced
When would you choose a linear probe, and when full fine-tuning, over PEFT?basic
Exercises
- BERT-large has \(d = 1024\), \(L = 24\) and 335,141,888 parameters. How many parameters do Houlsby adapters with \(m = 64\) add, and what fraction of the model is that?
- Prompt tuning on T5-XXL (\(d = 4096\), about 11B parameters) with the default 100 tokens: how many parameters, and what percentage of the model?
- Prefix tuning on Llama 3.1 8B with \(p = 20\): how many trained parameters? How much extra KV cache in bf16 for a batch of 64 sequences? (Each token's K and V across all layers take \(2 \cdot 32 \cdot 1024\) numbers.)
- How many parameters does BitFit train in Llama 3.1 8B? In Qwen2.5 7B (\(d = 3584\), 28 layers, 4 key-value heads of size 128, biases on Q, K and V only)?
- A query has scores 2, 1, 0 with three real keys. You add two prefix keys with score 1.5. What is \(\lambda\)? What happens to the ratio of the weights on the first two real keys?
- You must store 50 task variants of Llama 3.1 8B in bf16. Compare full fine-tuning with Houlsby adapters at \(m = 64\).
Answerscheck
Key takeaways
- PEFT freezes the pretrained model and trains a small \(\phi\) per task, so \(N\) tasks cost \(|\theta_0| + N|\phi|\) instead of \(N|\theta_0|\) (56.1). It saves storage and optimizer memory, not activation memory.
- Adapters (56.2) are bottleneck MLPs with a skip connection, two per block, started near the identity; \(2md + d + m\) parameters each (56.3). Good quality, but extra serial latency at small batch.
- BitFit trains only biases (56.4): 0.09% of BERT-base and matching full fine-tuning on GLUE, but useless on bias-free models like Llama.
- Prompt tuning learns \(p\) input vectors (56.5); it is tiny and allows mixed-task batches, and it matches fine-tuning only for very large models.
- Prefix tuning learns keys and values in every layer (56.6); a prefix is a query-gated mix of the original attention output and a prefix output (56.7, 56.8). Reparameterization (56.9) stabilizes training and is dropped afterwards.
- P-Tuning v2 = deep prompts + a classification head + a per-task prompt length (56.10); comparable to fine-tuning from 330M to 10B parameters.
- Choose by data size, number of tasks, latency and distance from pretraining: prompting, then a linear probe, then PEFT, then full fine-tuning; test the cheaper option first.
Key formulas
References for this session
- Houlsby et al. (2019). Parameter-Efficient Transfer Learning for NLP
- Ben Zaken et al. (2021). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
- Lester et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning
- Li & Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Liu et al. (2021). GPT Understands, Too (P-Tuning)
- Liu et al. (2021). P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
- Houlsby et al. (2019). Parameter-Efficient Transfer Learning for NLP
- Ben Zaken et al. (2021). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
- Lester et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning
- Li & Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Liu et al. (2021). GPT Understands, Too (P-Tuning)
- Liu et al. (2021). P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
- He et al. (2021). Towards a Unified View of Parameter-Efficient Transfer Learning
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Hugging Face (code). PEFT library
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models