Forgetting, data mixing and model merging
When we fine-tune a model on a new task, it often gets worse at things it could already do. This session explains why this catastrophic forgetting happens, how to limit it (mix in general data, use a lower learning rate, train only a LoRA adapter, or pull the weights back), and how to combine several fine-tuned models by doing arithmetic on their weights: averaging, model soups, task vectors, TIES and DARE. It ends with how to measure a regression so that you can trust the result.
What you will learn
- What catastrophic forgetting is, and why gradient steps for a new task can raise the loss on old skills (first- and second-order view)
- Four ways to limit it: replay of general data, a lower learning rate or fewer steps, LoRA as a built-in regularizer, and a pull-back penalty such as EWC
- Why replaying general data and EWC are the same idea for a quadratic loss, and how to choose a mixing ratio
- Weight interpolation (WiSE-FT), model soups and linear mode connectivity: when averaging weights works and when it fails
- Task vectors and task arithmetic, plus TIES-Merging and DARE for merging several fine-tuned models
- How to measure a regression: a fixed evaluation suite, paired comparisons, confidence intervals and how many examples you need
Why this matters
In session 54 and session 57 we learned how to teach a model a new skill. This session is about the price. When you fine-tune a model on a narrow task, it often becomes worse at things it could do before: general knowledge, math, other languages, or even following the chat format. This is called catastrophic forgetting. Every team that fine-tunes models meets it, and interviewers ask about it often.
The name comes from McCloskey and Cohen (1989), who saw that a small neural network trained on a second list of facts lost the first list almost completely. [Certain] Large language models forget too. Luo et al. (2023) fine-tuned models from 1B to 7B parameters on a sequence of tasks and saw forgetting at all sizes; in that range, the larger models even forgot more, partly because they started from a higher level. [Certain] The InstructGPT paper calls the drop in general benchmark scores after alignment the alignment tax, and reduced it by mixing pretraining data into the RL updates (“PPO-ptx”). [Certain]
This session has three parts:
- Why forgetting happens, with a simple picture: the weights move to where the new task is good, and that place can be bad for the old skills.
- How to limit it: mix in general data (replay), use a lower learning rate or fewer steps, use LoRA as a regularizer, or add a penalty that pulls important weights back.
- Model merging: do arithmetic on the weights of several fine-tuned models. Weight averaging, model soups, task vectors, TIES and DARE. Merging can undo forgetting and combine skills without any extra training.
And one practical question ties the parts together: how do you know that you have a regression, and not just noise in the evaluation?
Real systems use these ideas every day. Llama 3 averaged models trained with different data or hyperparameters at each post-training stage (reward model, SFT and DPO). [Certain] Continual pretraining recipes mix a few percent of old data into the new data to avoid forgetting. [Certain] Open-source model hubs are full of merged models built with tools such as MergeKit, and a merged 7B model built with DARE once ranked first among 7B models on the Open LLM Leaderboard. [Certain]
Catastrophic forgetting: why fine-tuning can break general skills
A pretrained model has weights \(\theta_0\). These weights are good at many things at once, because pretraining saw a huge, mixed dataset. Fine-tuning then runs gradient descent on a new loss \(\mathcal{L}_{\text{task}}\) only. Nothing in that loss says “stay good at the old things”. The weights move to wherever the new loss is low, and the old skills are only safe if that place happens to be good for them too.
We measure forgetting on a held-out set of “general” evaluations that the fine-tuning data does not cover:
- Read the symbols
- \(A_{\text{gen}}(\theta)\)
- the score (for example accuracy) of the model with weights \(\theta\) on general evaluations: knowledge, math, code, other languages, chat behavior
- \(\theta_0\), \(\theta_{\text{ft}}\)
- the weights before and after fine-tuning
- \(F\)
- forgetting: how many points the general score dropped. \(F \gt 0\) is a regression; \(F/A_{\text{gen}}(\theta_0)\) is the relative drop
The first-order view: gradients that disagree
Take one small step of size \(\eta\) on the task loss: \(\Delta\theta = -\eta \nabla \mathcal{L}_{\text{task}}\). A Taylor expansion of the general loss gives its change:
If the two gradients point in similar directions (positive dot product), the task step also helps the general skills. If they point in opposite directions (negative dot product), the task step hurts the general skills. This is called gradient interference. The same weights store many skills, so the directions often overlap.
The second-order view: moving away from a good place
There is a catch. A well-pretrained model sits near a minimum of the general loss, so \(\nabla \mathcal{L}_{\text{gen}}(\theta_0) \approx 0\) and formula (61.2) says almost nothing. The real damage comes from the distance travelled. Near a minimum, the general loss grows like a bowl:
- Read the symbols
- \(\Delta\theta\)
- the total change of the weights during fine-tuning, \(\theta_{\text{ft}} - \theta_0\)
- \(H_{\text{gen}}\)
- the Hessian (matrix of second derivatives) of the general loss at \(\theta_0\). It says how fast the loss grows in each direction: large in “sharp” directions, small in “flat” ones
Formula (61.3) gives us the whole story of this section. Forgetting is large when the weights move far (large \(\Delta\theta\)) and in sharp directions of the general loss. So every fix does one of two things: move less, or move in flat directions.
Why LLM fine-tuning is especially at risk
- The data is narrow. A domain set (legal, medical, code in one language) covers a tiny part of what pretraining covered. Nothing in it reminds the model of the rest.
- Many steps on few examples. Several epochs on a small set move the weights far in the directions that only this set cares about.
- The learning rate is high relative to the change we need. A pretraining-size learning rate, or rewarming it to a high peak in continual pretraining, moves the weights a lot. Ibrahim et al. (2024) found that a higher rewarmed peak learning rate gives more adaptation and more forgetting. [Certain]
- Format and behavior are fragile. Fine-tuning on plain question–answer pairs without the chat template can break the chat behavior of an instruct model, even when its knowledge is fine. Some of what looks like lost knowledge is lost format, and a little training on the old format brings it back. [Likely]
Figure 1 makes formula (61.3) concrete with two weights, and runs the training loop one line at a time so you can see every number. The general loss is a narrow valley: flat along \(\theta_1\), sharp along \(\theta_2\). The task loss is a round bowl with its minimum somewhere else. Plain fine-tuning walks straight to the task minimum and climbs the steep side of the general valley.
- \(\theta \leftarrow \theta_0 = (0, 0)\)
- for \(t = 1, \dots, 60\):
- \(g \leftarrow (1-\alpha)(\theta - \theta^\ast) + (\alpha + \lambda)\,H\theta\) // gradient of (61.4) + (61.5)
- \(\theta \leftarrow \theta - \eta\, g\)
- record \(\mathcal{L}_{\text{gen}}(\theta)\) and \(\mathcal{L}_{\text{task}}(\theta)\)
- forgetting \(= \mathcal{L}_{\text{gen}}(\theta_{60}) - \mathcal{L}_{\text{gen}}(\theta_0)\) // like (61.1)
How to read Figure 1
Press Reset, set the speed to the left, and press Step. Line 3 computes the gradient at \(\theta_0\): with \(\alpha = 0\) it is \(g = (-2.5, -2.0)\), so line 4 moves the weights to \((0.25, 0.20)\), and line 5 shows that the general loss is already 0.088 after one step. Press Play to watch the rest: the left chart is the forgetting curve, the general loss climbing while the task loss falls. With all sliders at their start values, plain fine-tuning reaches the task minimum (task loss 0) but the general loss rises from 0 to 8.75. Move \(\alpha\) to 0.5: the path bends toward the flat direction \(\theta_1\) and ends near \((1.96, 0.40)\), where the general loss is only 0.80 and the task loss 1.43. Set \(\alpha\) back to 0 and \(\lambda\) to 1: the path ends at \((2.0, 0.4)\), the same minimum that replay with \(\alpha = 0.5\) is heading to (replay gets there more slowly, because it halves the task gradient). Then set both to 0 and lower \(\eta\) to 0.01: the path stops early on the straight line, with a worse task loss (1.53) and a worse general loss (1.80) than mixing. The grey curve is the set of best trade-offs; early stopping is not on it. Finally press Sweep \(\alpha\) (with \(\eta = 0.1\), \(\lambda = 0\)): the general loss after 60 steps falls from 8.75 at \(\alpha = 0\) to 4.57 at 0.1, 2.69 at 0.2 and 0.80 at 0.5, while the task loss rises only from 0 to 0.19, 0.51 and 1.43. The first 20% of general data removes more than two thirds of the forgetting.
Fix 1: mix in general data (replay)
The most direct fix is to keep reminding the model of the old skills. Instead of training on task data only, each batch contains a share \(\alpha\) of general data: a sample of pretraining text, or of the general SFT mix that made the model an assistant. This is called replay or data mixing. In expectation, the model minimizes a weighted sum of the two losses:
- Read the symbols
- \(\alpha\)
- the mixing ratio: the share of each batch (in tokens or examples) that comes from general data. \(\alpha = 0\) is plain fine-tuning
- \(\mathcal{L}_{\text{gen}}\)
- the loss on the replayed general data: next-token loss on pretraining text, or the SFT loss on general chats
Now the gradient of every step contains a part that says “stay good at general things”. In Figure 1 the path bends: it moves along the flat direction, where the general loss barely rises, and avoids the sharp one. For each \(\alpha\) the end point is the minimum of (61.4); all these minima together form the grey curve of best trade-offs.
How much to mix
- Continual pretraining (more pretraining on new text, such as a new language or new code): Ibrahim et al. (2024) used 5% replay of the old data for a weak shift (English to English data) and 25% for a strong shift (English to German), together with rewarming and re-decaying the learning rate, and this matched retraining from scratch on all the data. They tested this at 405M and 10B parameters. [Certain]
- SFT for a domain: mix the domain examples with the general instruction data of the original model. Llama 3’s final SFT mix was 52.66% general English examples, with the rest covering skills such as code, reasoning and tool use. [Certain] Luo et al. (2023) found that a general instruction-tuning stage before the narrow fine-tune also reduces forgetting. [Certain]
- RL and preference training: InstructGPT’s PPO-ptx adds a pretraining next-token loss to the RL objective. [Certain]
The general data should be close to what the model was trained on before. A random web crawl is a poor stand-in for the original SFT mix of an assistant. [Likely] If you do not have the original data (common for open-weight models), you can use an open instruction dataset of similar style, or generate replay data with the original model itself. [Likely]
You continue pretraining on 10B tokens of new data with 5% replay, measured as a share of all tokens. The new data is 95% of the total, so the total is \(10/0.95 = 10.53\)B tokens, and replay adds 0.53B tokens: about 5% more compute. With 25% replay, replay adds \(10/0.75 - 10 = 3.33\)B tokens, a third more.
For SFT: 40,000 domain examples and \(\alpha = 0.2\) means \(40000/0.8 = 50{,}000\) examples per epoch, so 10,000 of them general. Replay is cheap insurance compared with the cost of a model that forgets.
Fix 2: move less (lower learning rate, fewer steps, pull-back)
Formula (61.3) says forgetting grows with the size of \(\Delta\theta\). The simplest ways to move less are a lower learning rate, fewer steps (fewer epochs) and early stopping. They help, but they are blunt: they slow down movement in all directions, including the directions the task needs. In Figure 1 the low-learning-rate path stops early on the straight line toward the task minimum. It forgets less than plain fine-tuning, but it is worse than mixing on both losses at once.
A smarter version slows down only the directions that matter for the old skills. Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) adds a penalty that pulls each weight back toward its old value, with a strength that depends on how important the weight was for the old task: [Certain]
- Read the symbols
- \(\theta_{0,i}\)
- weight \(i\) before fine-tuning
- \(F_i\)
- the importance of weight \(i\) for the old skills: the diagonal of the Fisher information, \(F_i = \E\big[(\partial \log p_\theta(y \mid x) / \partial \theta_i)^2\big]\), averaged over old data. Near a minimum it approximates the diagonal of \(H_{\text{gen}}\)
- \(\lambda\)
- the pull-back strength. With all \(F_i = 1\) this is plain L2 regularization toward the starting point (sometimes called L2-SP)
Here is a useful connection. If the general loss is exactly the bowl of (61.3) and \(F\) equals its Hessian, then the EWC penalty is the general loss times \(\lambda\). Dividing (61.4) by \(1 - \alpha\) shows that mixing with ratio \(\alpha\) is the same objective as EWC with
\(\lambda = \alpha / (1 - \alpha)\),
so \(\alpha = 0.5\) matches \(\lambda = 1\). That is why Figure 1 ends at the same point for both. In real networks the two differ: replay measures the true general loss, while EWC uses a local, diagonal approximation that becomes wrong once the weights move far. This is one reason replay is the default for LLMs, and EWC is rarely used at LLM scale. [Likely] EWC also needs the Fisher values for every weight, which is one more copy of the model in memory.
Fix 3: LoRA as a regularizer
LoRA (session 57) freezes the pretrained weight matrix \(W_0\) and learns only a low-rank update:
- Read the symbols
- \(W_0\)
- the frozen pretrained matrix, of size \(d \times k\)
- \(B\), \(A\)
- the two small trained matrices; \(B\) starts at zero, so training starts exactly at the pretrained model
- \(r\)
- the rank, much smaller than \(d\) and \(k\) (often 8 to 64)
- \(\alpha_{\text{LoRA}}\)
- a fixed scale (not the mixing ratio \(\alpha\) of formula 61.4)
The rank limit is a strong constraint on \(\Delta\theta\): each matrix can change only inside an \(r\)-dimensional space of directions, and all other directions stay exactly where pretraining put them. Biderman et al. (2024) compared LoRA and full fine-tuning on code and math, for both instruction tuning and continued pretraining. Their title says it: LoRA learns less and forgets less. LoRA reached lower scores on the target domain but kept more of the base model’s skills outside it, and it forgot less than common regularizers such as weight decay and dropout. [Certain] They also found that full fine-tuning learns weight changes with a rank 10 to 100 times higher than typical LoRA settings. [Certain]
One attention projection in a 7B-class model is \(4096 \times 4096 = 16{,}777{,}216\) weights. LoRA with \(r = 16\) trains \(r(d + k) = 16 \times 8192 = 131{,}072\) numbers, which is 0.78% of the matrix. Every \(\Delta W\) it can produce has rank at most 16, out of a possible 4,096.
Two warnings. First, LoRA does not make forgetting impossible: a large rank, a high learning rate and many epochs can still move the model far, and in our PyTorch example below a rank-2 LoRA still forgets a lot on a toy task. Second, if your task truly needs a big change (a new language, a lot of new knowledge), the rank limit hurts learning; then full fine-tuning plus replay is usually the better choice. [Likely]
A useful bonus: a LoRA adapter can be switched off. The base model is still there, unchanged, so for requests outside the domain you can serve the base model and lose nothing. [Certain]
Model merging, part 1: averaging weights
Now a surprising idea. Instead of training one model that is good at everything, train several models and average their weights. Not their outputs (that would be an ensemble, which costs \(K\) forward passes), but the numbers inside the weight matrices, giving one model with the normal cost.
- Read the symbols
- \(\beta\)
- the interpolation weight: 0 gives the pretrained model, 1 the fine-tuned model, values in between lie on the straight line between them
- \(\theta_k\)
- \(K\) fine-tuned models, all started from the same pretrained weights (for example with different learning rates, seeds or data orders)
- \(\theta_{\text{soup}}\)
- a uniform model soup: the plain average of the \(K\) models
When does averaging work? Linear mode connectivity
Averaging weights sounds like it should fail. A neural network has many symmetries: you can swap two hidden units (and their weights in the next layer) and get the exact same function. Two networks trained from different random starts may store the same feature in different units, so the average mixes unrelated units and the result is broken.
The key fact is that this problem mostly goes away when both models start from the same weights. Frankle et al. (2020) showed that networks become “stable to SGD noise” early in training: after that point, two runs that differ only in data order end in minima connected by a straight line with no rise in error between them. This is called linear mode connectivity. [Certain] Fine-tuning from a shared pretrained model is the extreme case: all fine-tuned models start from the same point and usually stay in the same wide basin. [Likely] Ainsworth et al. (Git Re-Basin) showed that even independently trained ResNets on CIFAR-10 can be merged without a loss barrier if you first permute the units of one network to match the other. [Certain]
WiSE-FT: interpolate the base and the fine-tuned model
The first formula in (61.7) is WiSE-FT (Wortsman et al., 2022). They fine-tuned zero-shot models such as CLIP and then moved part of the way back toward the original weights. On ImageNet, this improved accuracy under distribution shift by 4 to 6 points over prior work while also raising ImageNet accuracy by 1.6 points, at no extra cost in training or inference. [Certain] For LLMs, the same trick trades task skill against general skill after training, with one knob \(\beta\) and no retraining.
Model soups
The second formula is a model soup (Wortsman et al., 2022). The usual practice is to run a hyperparameter sweep and keep the best model. A soup averages the models of the sweep instead. A uniform soup averages all of them. A greedy soup sorts the models by validation score and adds each one to the soup only if the averaged model gets better on held-out data. With ViT-G, a soup reached 90.94% top-1 accuracy on ImageNet. [Certain] Llama 3 used the same idea in post-training, averaging models from runs with different data or hyperparameters at the reward model, SFT and DPO stages. [Certain]
Figure 2 trains three small networks live in your browser and walks along straight lines between them. You can see both effects: the trade-off of WiSE-FT, and the barrier between two independently trained networks.
How to read Figure 2
In the first mode, \(\beta = 0\) is the base (general 97.7%, task about 52%, near chance) and \(\beta = 1\) is the fine-tuned model (task 96.0%, but general falls to 68.3%: forgetting). In between, you trade one for the other, and the curve is not a straight line, so some points are better than others. The second mode ends at a model that keeps 93.5% general and reaches 97.0% on the task: replay beats interpolation here. The third mode joins two good general models (97.7% and 97.3%); in the middle the general accuracy falls to about 56%. That dip is the barrier that appears when models do not share their starting weights.
Model merging, part 2: task vectors and their arithmetic
Ilharco et al. (2023) looked at fine-tuning as a vector. The task vector of a task is the change that fine-tuning made to the weights:
- Read the symbols
- \(\theta_t\)
- the model fine-tuned on task \(t\), started from \(\theta_0\)
- \(\tau_t\)
- its task vector: one number per weight, the same shape as the model
- \(\lambda\)
- a scaling factor, tuned on validation data. \(\lambda = 1/T\) is plain averaging of the \(T\) models
They showed three operations, across several models and tasks: [Certain]
- Addition: \(\theta_0 + \lambda(\tau_A + \tau_B)\) gives one model that is good at both tasks, without training on both.
- Negation: \(\theta_0 - \lambda \tau_A\) makes the model worse at task A while other tasks are mostly unaffected. One use is to remove an unwanted behavior (for example toxic text) by negating the task vector of a model fine-tuned to produce it.
- Analogies: if tasks are related as “A is to B as C is to D”, then \(\tau_C + (\tau_B - \tau_A)\) can improve task D without any data for D.
Note that WiSE-FT is task arithmetic with one task: \(\theta(\beta) = \theta_0 + \beta\,\tau\).
Figure 3 shows these merges in a picture with two weights, so you can see why they work and when they fail. Task A has its minimum at \(\theta_A = (2, 0)\), task B at \(\theta_B = (0, 2)\), and the base \(\theta_0 = (0, 0)\) is the minimum of a general loss. Each task loss is sharp in one direction and flat in the other (curvature 3 and 0.3). At interference angle \(\varphi = 0°\), task A is flat along \(\tau_B\) and task B is flat along \(\tau_A\), so adding the other task’s vector costs little. As \(\varphi\) grows, each task becomes sharp along the other task’s vector, and adding both vectors hurts both tasks. This matches the usual explanation of why task arithmetic works: the task vectors change directions that the other tasks do not care about much. [Likely] The algorithm next to the figure is the standard recipe: walk along a line in weight space, evaluate on validation data, keep the best knob value.
- \(\tau_A \leftarrow \theta_A - \theta_0\), \(\tau_B \leftarrow \theta_B - \theta_0\) // (61.8)
- for each knob value \(c = 0, 0.05, 0.10, \dots\):
- \(\theta(c) \leftarrow \theta_{\text{start}} + c\,\mathbf{d}\) // (61.7) or (61.8)
- evaluate \(\mathcal{L}_A\), \(\mathcal{L}_B\), \(\mathcal{L}_{\text{gen}}\) on validation data
- if the target loss is the lowest so far: \(c^\ast \leftarrow c\)
- return \(\theta(c^\ast)\)
How to read Figure 3
Start with task arithmetic at \(\varphi = 0°\) and press Reset, then Step. Line 1 builds \(\tau_A = (2, 0)\) and \(\tau_B = (0, 2)\); each later group of three lines moves \(\lambda\) by 0.05, computes the merged weights, evaluates the losses and updates the best value. At \(\lambda = 0.5\) (the plain average of the two fine-tunes) both task losses are 1.65. At \(\lambda = 1\) (the full sum) they are 0.6 each, because each task is flat along the other task’s vector. The best grid value is \(\lambda = 0.90\) with \(\mathcal{L}_A + \mathcal{L}_B = 1.092\); this is why task arithmetic tunes \(\lambda\) instead of fixing \(1/T\). Note the price: the general loss at \(\lambda = 1\) is 1.6, against 0.4 for the average. Now move \(\varphi\) to 45°: at \(\lambda = 1\) both task losses are 3.3, exactly as bad as the base model, and the best \(\lambda\) falls to 0.5. The “Average” mode walks the line from \(\theta_A\) to \(\theta_B\); its midpoint is the same point \((1, 1)\) as \(\lambda = 0.5\). In WiSE-FT mode at \(\varphi = 0°\), \(\beta = 0.5\) gives \(\mathcal{L}_A = 1.5\) with general loss 0.2, and the best value is \(\beta = 0.90\).
Interference: why plain addition loses information
When we average \(T\) task vectors, each one is divided by \(T\). If only one task changed a weight, that change is shrunk to \(1/T\) of its size. If two tasks changed the same weight in opposite directions, they cancel, and neither task gets what it needed. TIES-Merging (Yadav et al., 2023) names these two problems: redundant small values and sign disagreement. [Certain] It fixes them in three steps:
- Read the symbols
- \(\mathrm{top}_k(\tau_t)\)
- trim: keep the \(k\%\) of entries of \(\tau_t\) with the largest magnitude and set the rest to 0
- \(p\)
- one weight (parameter) position
- \(\gamma_p\)
- elect the sign: the sign with the larger total movement over all tasks at position \(p\)
- \(\mathcal{A}_p\)
- the tasks whose trimmed value at \(p\) is non-zero and has the sign \(\gamma_p\)
- \(\tau_{m,p}\)
- disjoint merge: average only over the agreeing tasks. The merged model is \(\theta_0 + \lambda \tau_m\)
When no validation data is available, the paper’s recipe keeps the top 20% of each task vector and uses \(\lambda = 1\). [Certain]
DARE: drop most of the delta and rescale
Yu et al. (2024) observed that SFT task vectors of LLMs are tiny and very redundant: the changes usually stay within 0.002 in absolute value. [Certain] DARE (Drop And REscale) randomly drops a share \(p\) of each task vector and scales up the rest so the expected value is unchanged:
- Read the symbols
- \(m_{t,i}\)
- 1 if entry \(i\) of task vector \(t\) is dropped (probability \(p\)), else 0
- \(\odot\)
- element-wise product
- \(1/(1-p)\)
- the rescale factor; it keeps \(\E[\hat\tau_t] = \tau_t\)
They report that DARE can drop 90% or even 99% of the delta with little loss, and that it helps when several models are merged. [Certain] In practice it is used as a first step before another merge method, such as task arithmetic or TIES. [Likely] It has a clear limit: after continued pretraining the deltas can reach about 0.03, and then DARE fails; for a code model built on extra code pretraining, removing just 10% of the delta dropped HumanEval and MBPP to 0. [Certain] Merging methods assume small, compatible task vectors from the same base.
Other methods you will meet in merge tools: SLERP (spherical interpolation between two models, which keeps the norm of the weights), and Fisher merging (a weighted average where each weight counts in proportion to its Fisher importance, like EWC). [Likely]
- input \(\tau_1, \tau_2, \tau_3\), with \(\tau_t = \theta_t - \theta_0\) // (61.8)
- DARE: \(\hat\tau_t \leftarrow (1 - m_t) \odot \tau_t / (1 - p)\) // (61.10)
- trim: \(\hat\tau_t \leftarrow \mathrm{top}_k(\hat\tau_t)\) // (61.9)
- elect: \(\gamma_p \leftarrow \mathrm{sgn}\big(\sum_t \hat\tau_{t,p}\big)\) for each entry \(p\)
- merge entry \(p\): task arithmetic \(\sum_t \tau_{t,p}\); TIES \(\tfrac{1}{|\mathcal{A}_p|}\sum_{t \in \mathcal{A}_p} \hat\tau_{t,p}\)
- \(\theta - \theta_0 \leftarrow \lambda\,\tau_m\)
How to read Figure 4
Choose TIES, press Reset and step through. Lines 3 trim each task to 4 entries; the dashed outlines are the values that were set to 0. Then line 4 runs once per entry. Look at entry 4: tasks 1, 2 and 3 want +0.6, −0.6 and −0.5. TIES elects the minus sign (sum −0.5) and line 5 averages only the two agreeing values, −0.55. Look at entry 6: only task 3 changed it (−0.8), and TIES keeps −0.8. The result keeps 0.48, 0.95 and 0.90 of the three task changes. Now choose Task arithmetic: \(\lambda\) jumps to 1/3, the plain average. Entry 4 becomes −0.167, which serves nobody, entry 6 shrinks to −0.257, and the kept shares fall to 0.36, 0.56 and 0.35. Set \(k = 8\) in TIES (no trimming): the kept shares are 0.48, 1.00 and 0.90, so here most of the gain comes from the sign election and the disjoint mean. Finally choose DARE + TIES with \(p = 0.5\) and seed 7. Task 2’s −0.6 at entry 4 is dropped, task 1’s +0.6 is doubled to +1.2 and task 3’s −0.5 to −1.0; the sum is +0.2, so the elected sign flips to + and the merged value is +1.2. With only 8 entries, one unlucky mask changes the answer; DARE relies on the millions of weights in a real model, where the drops average out. Change the seed to see other masks.
Task vectors (entries 1 to 8): \(\tau_1 = (0.9, -0.1, 0.05, 0.6, -0.7, 0.02, 0.3, -0.05)\), \(\tau_2 = (0.8, 0.5, -0.04, -0.6, -0.45, 0.01, -0.02, 0.4)\), \(\tau_3 = (-0.1, 0.6, 0.03, -0.5, 0.06, -0.8, 0.25, 0.02)\).
Average: \((0.533, 0.333, 0.013, -0.167, -0.363, -0.257, 0.177, 0.123)\).
Trim (keep 4 largest per task): \(\hat\tau_1 = (0.9, 0, 0, 0.6, -0.7, 0, 0.3, 0)\), \(\hat\tau_2 = (0.8, 0.5, 0, -0.6, -0.45, 0, 0, 0)\), \(\hat\tau_3 = (0, 0.6, 0, -0.5, 0, -0.8, 0.25, 0)\).
Elect: sums \((1.7, 1.1, 0, -0.5, -1.15, -0.8, 0.55, 0)\), so signs \((+, +, 0, -, -, -, +, 0)\).
Disjoint mean: \(\tau_m = (0.85, 0.55, 0, -0.55, -0.575, -0.8, 0.275, 0)\). Its length is 1.54, close to the length of a single task vector (1.15 to 1.33), while the plain average has length only 0.82: averaging shrinks every task’s change.
How to measure regressions
All the fixes above need a measurement: did the general skills drop, and by how much? This is harder than it looks, because an evaluation score is itself a noisy estimate. A drop from 70.0% to 68.0% can be a real regression or just noise, depending on the number of questions and on how you compare.
Build a regression suite before you fine-tune
- Cover the skills you must keep, not only the target task: knowledge (MMLU-style), math (GSM8K-style), code (HumanEval-style), instruction following, multi-turn chat, safety (harmful requests refused, harmless ones not over-refused), long context and other languages if you serve them (session 40, session 155).
- Check behavior, not only scores: does the model still stop at the end-of-turn token, use the chat template, keep its answer length, and call tools in the right format? Format breaks are among the most common regressions after fine-tuning. [Likely]
- Fix everything except the weights: same prompts, same chat template, same decoding settings (greedy or fixed seeds), same scoring code. Otherwise you measure the change of the setup, not of the model.
- Compare against two references: the model you started from, and the model currently in production.
- Decontaminate: the fine-tuning data must not contain the evaluation questions.
Is the drop real? Confidence intervals
For accuracy \(p\) on \(n\) independent questions, the standard error (SE) is \(\sqrt{p(1-p)/n}\). If we treat the two models as independent samples, the SE of the difference is:
But both models answer the same questions. Most questions are easy or hard for both; only the questions where they disagree carry information about the difference. The paired SE uses this:
- Read the symbols
- \(\hat p_A\), \(\hat p_B\)
- accuracy of the old and the new model on the same \(n\) questions
- \(b\)
- share of questions the old model got right and the new model got wrong (regressions)
- \(c\)
- share the old model got wrong and the new model got right (improvements)
- \(d\)
- share of questions where the two models disagree
A 95% confidence interval is the difference \(\pm 1.96\,\mathrm{SE}\). If it does not include 0, the change is unlikely to be noise. The paired SE is usually much smaller, because \(d\) is much smaller than \(2p(1-p)\). For a paired test on the counts \(b\) and \(c\), this is closely related to McNemar’s test. [Certain] A paired bootstrap over questions gives the same kind of interval for any metric, including judge scores. [Certain]
The old model scores 70.0%, the new one 68.0%, on \(n = 1000\) questions. The SE of one score is \(\sqrt{0.7 \times 0.3 / 1000} = 1.45\) points. Unpaired, the SE of the difference is \(\sqrt{(0.21 + 0.2176)/1000} = 2.07\) points, so the 95% interval is \(-2.0 \pm 4.05\), from −6.05 to +2.05 points: we cannot tell.
Now pair the answers. The models disagree on 10% of questions: 6% were right and became wrong, 4% the other way. Then \(\mathrm{SE}_{\text{paired}} = \sqrt{(0.10 - 0.0004)/1000} = 1.00\) point, and the interval is from −3.96 to −0.04 points. With the same data, the paired view says this is (just) a real regression.
How many questions do you need to detect a 1-point change? With \(d = 10\%\) and the paired test, the interval half-width \(1.96\sqrt{0.0999/n}\) reaches 1 point at \(n \approx 3{,}838\). Unpaired, at \(p = 0.7\), you would need about 16,135.
How to read Figure 5
At the start values (1,000 questions, 70% to 68%, 10% disagreement), the unpaired interval crosses 0 but the paired one does not. Raise \(d\) to 20%: the models disagree on more questions, the paired interval widens and now crosses 0 too. Lower \(n\) to 300 and both intervals become wide: a small benchmark cannot see a 2-point change. This is why regression suites need either many questions or several benchmarks for the same skill.
Two more traps. First, many comparisons: if you check 20 benchmarks at 95% confidence, about one will look significant by chance. Look for patterns (several benchmarks of the same skill moving together) or use a stricter threshold. [Certain] Second, seed noise: two fine-tunes with different seeds can differ by more than the effect you study. If you can, train with two or three seeds and compare the spread. [Likely]
Putting it together
- input: base \(\theta_0\), task data \(D_{\text{task}}\), general data \(D_{\text{gen}}\), mixing ratio \(\alpha\), regression suite \(S\), budget \(\epsilon\)
- score the base on \(S\) and on the task validation set; save per-question results // needed for paired tests
- for each step: build a batch with share \(\alpha\) from \(D_{\text{gen}}\) and \(1 - \alpha\) from \(D_{\text{task}}\) // formula (61.4)
- take an optimizer step with a low learning rate (or train a LoRA adapter, 61.6)
- at each checkpoint: score on \(S\) with the same prompts, template and decoding
- for each benchmark: paired difference and its 95% interval // formula (61.12)
- if some benchmark drops by more than \(\epsilon\) with an interval below 0: raise \(\alpha\), lower the learning rate or the rank, or interpolate back (61.7)
- return the checkpoint with the best task score among those that pass the gate
- input: base \(\theta_0\), fine-tuned models \(\theta_1, \dots, \theta_T\) from the same base, keep share \(k\), drop rate \(p\), scale \(\lambda\)
- for each task \(t\): \(\tau_t \leftarrow \theta_t - \theta_0\) // task vector, (61.8)
- optional DARE: drop each entry with probability \(p\), divide the rest by \(1 - p\) // (61.10)
- trim: keep the top \(k\%\) of entries by magnitude, set the rest to 0 // (61.9)
- for each parameter \(p\): \(\gamma_p \leftarrow \mathrm{sgn}(\sum_t \hat\tau_{t,p})\) // elect
- \(\tau_{m,p} \leftarrow\) mean of the non-zero \(\hat\tau_{t,p}\) with sign \(\gamma_p\) // disjoint merge
- \(\theta \leftarrow \theta_0 + \lambda\,\tau_m\); choose \(\lambda\) (and \(k\)) on validation data, or use \(k = 20\%\), \(\lambda = 1\)
- run the regression suite on \(\theta\) as in Algorithm 1 // merges can regress too
The PyTorch code below shows every idea of this session on a small model that runs in a few seconds on a CPU. A network learns a “general” skill, then is fine-tuned on a new task in four ways (plain, low learning rate, 20% replay, LoRA). Then it interpolates the weights (WiSE-FT) and adds two task vectors. The printed numbers are the real output.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) D, C, H = 16, 4, 64 # 1) three "skills": each is a labeling rule on its own region of input space def make_task(seed, shift, n): W = torch.randn(D, C, generator=torch.Generator().manual_seed(seed)) def sample(m, s): x = torch.randn(m, D, generator=torch.Generator().manual_seed(s)) + shift return x, (x @ W).argmax(1) return sample(n, seed + 1), sample(2000, seed + 2) # (train, test) def region(k): v = torch.zeros(D); v[k] = 6.0; return v GEN = make_task(10, 0.0, 4000) # the "general" skill, learned in pretraining TA = make_task(20, region(0), 1000) # new task A TB = make_task(30, region(1), 1000) # new task B def net(): return nn.Sequential(nn.Linear(D, H), nn.ReLU(), nn.Linear(H, H), nn.ReLU(), nn.Linear(H, C)) def acc(m, task): x, y = task[1] with torch.no_grad(): return (m(x).argmax(1) == y).float().mean().item() def clone(m): c = net(); c.load_state_dict(m.state_dict()); return c def train(m, sets, steps, lr, mix=None, seed=0): """Mini-batches of 32 rows; each row comes from set j with probability mix[j], formula (61.4).""" g = torch.Generator().manual_seed(seed) opt = torch.optim.AdamW([p for p in m.parameters() if p.requires_grad], lr=lr, weight_decay=0.0) mix = torch.tensor(mix or [1.0] * len(sets)) for _ in range(steps): src = torch.multinomial(mix, 32, replacement=True, generator=g) xs, ys = [], [] for j, (x, y) in enumerate(sets): i = torch.randint(0, len(x), (int((src == j).sum()),), generator=g) xs.append(x[i]); ys.append(y[i]) loss = F.cross_entropy(m(torch.cat(xs)), torch.cat(ys)) opt.zero_grad(); loss.backward(); opt.step() return m # 2) LoRA: freeze W, learn a rank-r update B @ A, formula (61.6) class LoRA(nn.Module): def __init__(s, lin, r=2, alpha=4): super().__init__(); s.lin, s.scale = lin, alpha / r for p in s.lin.parameters(): p.requires_grad = False s.A = nn.Parameter(torch.randn(r, lin.in_features) / lin.in_features ** 0.5) s.B = nn.Parameter(torch.zeros(lin.out_features, r)) # starts as an exact copy of the base def forward(s, x): return s.lin(x) + s.scale * (x @ s.A.T @ s.B.T) def with_lora(m): m = clone(m) for i in (0, 2, 4): m[i] = LoRA(m[i]) return m base = train(net(), [GEN[0]], 3000, 3e-3) # "pretraining" g0 = acc(base, GEN) print(f"base general {g0:.3f} task A {acc(base, TA):.3f}") # 3) fine-tune on task A: plain, low learning rate, 20% general replay, LoRA runs = {"plain FT": train(clone(base), [TA[0]], 600, 3e-3), "low LR": train(clone(base), [TA[0]], 600, 3e-4), "20% mix": train(clone(base), [TA[0], GEN[0]], 600, 3e-3, mix=[0.8, 0.2]), "LoRA r=2": train(with_lora(base), [TA[0]], 600, 3e-3)} for name, m in runs.items(): # forgetting = general accuracy before minus after, formula (61.1) print(f"{name:10s} general {acc(m, GEN):.3f} task A {acc(m, TA):.3f} forgetting {g0 - acc(m, GEN):+.3f}") # 4) WiSE-FT: walk on the straight line from base to the fine-tuned weights, formula (61.7) ft, sb = runs["plain FT"], base.state_dict() for a in (0.25, 0.5, 0.75): m = clone(base); m.load_state_dict({k: (1 - a) * sb[k] + a * ft.state_dict()[k] for k in sb}) print(f"alpha {a:.2f} general {acc(m, GEN):.3f} task A {acc(m, TA):.3f}") # 5) task arithmetic: base + lam * (tau_A + tau_B), formula (61.8) ftB = train(clone(base), [TB[0]], 600, 3e-3) tau = lambda m: {k: m.state_dict()[k] - sb[k] for k in sb} tA, tB = tau(ft), tau(ftB) for lam in (0.5, 1.0): m = clone(base); m.load_state_dict({k: sb[k] + lam * (tA[k] + tB[k]) for k in sb}) print(f"merge lam {lam:.1f} general {acc(m, GEN):.3f} A {acc(m, TA):.3f} B {acc(m, TB):.3f}") print(f"only B general {acc(ftB, GEN):.3f} A {acc(ftB, TA):.3f} B {acc(ftB, TB):.3f}")
# output (CPU, PyTorch 2.8; deterministic, same on every run)
base general 0.952 task A 0.269
plain FT general 0.503 task A 0.946 forgetting +0.449
low LR general 0.723 task A 0.739 forgetting +0.229
20% mix general 0.877 task A 0.927 forgetting +0.076
LoRA r=2 general 0.622 task A 0.781 forgetting +0.331
alpha 0.25 general 0.923 task A 0.357
alpha 0.50 general 0.848 task A 0.529
alpha 0.75 general 0.713 task A 0.827
merge lam 0.5 general 0.586 A 0.744 B 0.250
merge lam 1.0 general 0.204 A 0.766 B 0.423
only B general 0.471 A 0.647 B 0.970
Read the output like a regression report. Plain fine-tuning learns task A (94.6%) but loses 44.9 points of the general skill. A ten times lower learning rate forgets less but also learns less. Replay (20% general data) keeps most of the general skill (−7.6 points) and still learns the task (92.7%): the best trade-off here. LoRA with rank 2 learns less (78.1%) and forgets less than plain fine-tuning, as Biderman et al. found, but on this toy it is not as good as replay. WiSE-FT gives a smooth trade-off between the two ends. The last lines are a warning: on this small network, adding the two task vectors keeps only part of each skill. Each fine-tune moved the weights far, and the two moves interfere. Task arithmetic works best for large pretrained models whose task vectors are small, the setting where DARE measured deltas within 0.002. [Likely] Always measure a merge before you trust it.
Where this is used
| Setting | The forgetting risk | What people do |
|---|---|---|
| Domain assistants (legal, medical, internal support) | General chat, reasoning and safety drop after narrow SFT | Mix in general SFT data, low learning rate or LoRA, regression suite with safety tests |
| Continual pretraining (new language, new code, fresh data) | English or old-domain loss rises; rewarming the learning rate adds forgetting | 5% to 25% replay, rewarm then re-decay the learning rate (Ibrahim et al.) |
| Post-training of frontier models (Llama 3) | Each stage (SFT, DPO, RL) can undo the gains of the last one | Average models from several runs at each stage; general data in every mix (session 68, session 66) |
| Open-weight model hubs | A model fine-tuned for code loses chat skills, and the other way round | Merge specialist fine-tunes of the same base with TIES, DARE or SLERP (MergeKit) |
| Robust vision models (CLIP fine-tuning) | Fine-tuning on ImageNet lowers robustness to distribution shift | WiSE-FT interpolation, model soups |
| VLM instruction tuning | Training the LLM on image–text data hurts its text-only skills | Mix text-only instruction data into the VLM mix; LoRA on the LLM (session 189) |
| Video-LLMs: video QA, key-moment detection, dense captions | Fine-tuning on one video task (for example timestamp output) breaks image QA or general chat | Mix image, video and text tasks; check each task in the regression suite (session 205, session 213) |
| Video retrieval and classification with a shared encoder | Fine-tuning the encoder for one label set hurts zero-shot retrieval and fingerprinting on others | Interpolate back toward the pretrained encoder (WiSE-FT), or train only a head or adapter |
| Embedding models (session 60) | A domain-tuned embedder loses general retrieval quality | Mix general retrieval pairs; average or merge domain and general checkpoints [Likely] |
Common mistakes
- Measuring only the target task. The task score goes up and nobody checks the rest. Run a fixed general suite on the base before training and on every candidate after.
- Fine-tuning an instruct model without its chat template, or with a different one. The model “forgets” how to chat, stop and follow the system prompt. This is often a format regression, not lost knowledge (session 9).
- A pretraining-size learning rate and many epochs on a small domain set. Formula (61.3): far moves cause forgetting. Start low (around \(10^{-5}\) or below for full fine-tuning) and use 1 to 3 epochs. [Likely]
- Replay data that does not match the original training data, such as random web text for an assistant. It does not protect the skills you care about.
- Merging models that do not share a base, or that use different tokenizers or architectures. Weight averaging only makes sense within one basin; two independently trained models give a broken average (Figure 2).
- Using DARE or TIES on large deltas, such as a model after continued pretraining. DARE fails when deltas are large. Check the size of the task vectors first.
- Calling a 1-point change a regression without a confidence interval. Use paired comparisons on the same questions, report the interval, and be careful with many benchmarks at once.
- Comparing runs with different evaluation settings (decoding, prompt format, number of shots). The difference you see may come from the setup.
Interview questions
What is catastrophic forgetting, and why does it happen in fine-tuning?basic
You fine-tuned an 8B model on medical QA and MMLU dropped 5 points. What do you do?core
Why does mixing in general data help, and how is it related to EWC?core
Why does LoRA forget less than full fine-tuning?core
What is a model soup, and how is it different from an ensemble?basic
Why can you average two fine-tunes of the same model but not two models trained from scratch?advanced
What is a task vector, and what can you do with it?core
Explain TIES-Merging. Which problems of plain averaging does it fix?advanced
What does DARE do, and when does it fail?advanced
Your new model scores 68.0% vs 70.0% on 1,000 questions. Is it a regression?core
How would you add a new skill to a deployed assistant without losing the old ones?advanced
Exercises
- A base model scores 64.0 on a knowledge benchmark; after domain fine-tuning it scores 58.5. Compute the forgetting \(F\) (61.1) and the relative drop.
- In Figure 1, the mixing ratio is \(\alpha = 0.2\). Which EWC strength \(\lambda\) gives the same minimum, assuming the general loss is the quadratic of the figure and \(F\) is its Hessian diagonal?
- How many numbers does LoRA with \(r = 8\) train for one MLP up-projection of size \(4096 \times 11008\)? What share of the matrix is that?
- One parameter has trimmed task-vector values \(0.4, -0.3, -0.2, 0.05\) from four tasks. Compute the plain average and the TIES merged value.
- With DARE at \(p = 0.9\), a delta entry of 0.001 that is kept becomes what? What is its expected value after DARE?
- A benchmark has \(n = 500\) questions. The old model scores 80%, the new one 78%, and they disagree on 6% of the questions. Compute the paired 95% interval. Is the drop significant?
Answerscheck
Key takeaways
- Fine-tuning minimizes only the new loss, so the weights can leave the region that was good for old skills. Forgetting grows with how far the weights move and how sharp the general loss is in those directions (61.3).
- Replay is the main fix: mix 5% to 30% general data, close to the original training data, into every batch (61.4). For a quadratic loss it equals EWC with \(\lambda = \alpha/(1-\alpha)\) (61.5).
- A lower learning rate and fewer steps reduce forgetting but also learning. LoRA limits the change to a low-rank space (61.6): it learns less and forgets less, and the base stays intact.
- Models fine-tuned from the same base can be averaged: WiSE-FT, model soups (61.7) and task arithmetic (61.8). Models from different random starts cannot, because of a loss barrier.
- TIES (61.9) trims, elects signs and averages only agreeing values; DARE (61.10) drops and rescales small deltas. Both assume small task vectors from one base.
- A regression is real only if its confidence interval excludes 0. Compare on the same questions with a paired test (61.12), keep the setup fixed, and cover all skills you must keep.
Key formulas
References for this session
- Wortsman et al. (2022). Model soups: averaging weights of multiple fine-tuned models improves accuracy
- Ilharco et al. (2022). Editing Models with Task Arithmetic
- Wortsman et al. (2022). Model soups: averaging weights of multiple fine-tuned models improves accuracy
- Ilharco et al. (2022). Editing Models with Task Arithmetic
- Kirkpatrick et al. (2017). Overcoming catastrophic forgetting in neural networks
- Wortsman et al. (2022). Robust fine-tuning of zero-shot models
- Yadav et al. (2023). TIES-Merging: Resolving Interference When Merging Models
- Yu et al. (2024). Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (DARE)
- Biderman et al. (2024). LoRA Learns Less and Forgets Less
- Ibrahim et al. (2024). Simple and Scalable Strategies to Continually Pre-train Large Language Models
- Luo et al. (2023). An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
- Frankle et al. (2020). Linear Mode Connectivity and the Lottery Ticket Hypothesis
- Ainsworth et al. (2023). Git Re-Basin: Merging Models modulo Permutation Symmetries
- Goddard et al. (2024). Arcee's MergeKit: A Toolkit for Merging Large Language Models
- Ouyang et al. (2022). Training language models to follow instructions with human feedback
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- McCloskey & Cohen (1989). Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem