LLM & VLM courseBlog
57Track 4: Post-training and RL · session 5 of 25

LoRA and other PEFT methods

LoRA keeps the pretrained weight matrix frozen and learns only a small low-rank change to it, written as the product of two thin matrices. This session builds LoRA from the shapes up: the parameter count, the zero initialization, the α/r scale, which layers to adapt, how to merge the change back into the weights for free inference, and how one server can run many adapters at once. Then it places DoRA, rsLoRA, LoRA+, IA3, prompt tuning and prefix tuning next to LoRA and says when each one helps.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: BeginnerBefore this: Sessions 4, 21, 26 and 56

What you will learn

  • The LoRA idea: freeze W0 and learn ΔW = BA with rank r much smaller than the layer size, and why B starts at zero
  • How to count LoRA parameters for any layer and for a whole model (Llama 3 8B: 0.52% at rank 16 on all linear layers)
  • What the α/r scale does, why rank-stabilized LoRA uses α/√r, and how to choose r, α and the learning rate
  • Which layers to adapt: query and value only (the original paper) versus all linear layers (QLoRA and later work)
  • Merging an adapter into the weights for zero extra inference cost, and serving many adapters on one base model (S-LoRA, Punica)
  • DoRA (magnitude and direction), IA3, VeRA, PiSSA, prompt tuning and prefix tuning: what each changes and when it helps
  • Where LoRA falls short of full fine-tuning: it learns less, but it also forgets less

Why this matters

Full fine-tuning of an 8B model with AdamW needs about 128 GB of memory before a single activation is stored, and every fine-tuned copy is another 16 GB file. LoRA trains about half a percent of that number of weights, fits on one GPU, and saves each task as a file of tens of megabytes.

In session 56 we met the first parameter-efficient fine-tuning (PEFT) methods: adapters, BitFit, prompt tuning and prefix tuning. They all freeze the pretrained model and train a small number of new or selected numbers. This session is about the method that won: LoRA (Low-Rank Adaptation, Hu et al., 2021). Today it is the default way to fine-tune open LLMs and VLMs on a budget, and the base of QLoRA (session 58). [Certain]

LoRA solves three problems at once:

  • Training memory. AdamW keeps two extra numbers for every trainable weight (Session 4). If only 0.5% of the weights train, the optimizer states almost disappear.
  • Storage and switching. One base model plus one small file per task, instead of one full copy per task. On GPT-3 175B, the LoRA paper reduced the checkpoint from 350 GB to 35 MB. [Certain]
  • Inference cost. Unlike the adapters of Houlsby et al., the LoRA change can be added into the weights after training. The model then has exactly the same shape and speed as before. [Certain]

The idea: freeze W, learn a low-rank change

Fine-tuning changes a pretrained weight matrix \(W_0\) into \(W_0 + \Delta W\). In full fine-tuning, \(\Delta W\) has as many free numbers as \(W_0\). The LoRA paper makes a bet: the change needed for one task is simple, so it can be written with a much smaller number of parameters. In their words, the update has a low intrinsic rank. [Certain]

The rank of a matrix is the number of independent directions it can produce. A rank-\(r\) matrix of shape \(d \times k\) can always be written as a product of a \(d \times r\) matrix and an \(r \times k\) matrix. So LoRA writes the change as such a product and trains only the two thin factors:

\[ h = W_0 x + \Delta W x = W_0 x + B A x, \qquad B \in \R^{d \times r},\ \ A \in \R^{r \times k},\ \ r \ll \min(d, k) \tag{57.1} \]
Read the symbols
\(x \in \R^{k}\)
the input of the linear layer (one token’s vector)
\(h \in \R^{d}\)
the output of the layer
\(W_0\)
the pretrained weight, shape \(d \times k\). It is frozen: no gradient, no optimizer state
\(A\)
the “down” matrix: it projects the \(k\)-dimensional input to only \(r\) numbers
\(B\)
the “up” matrix: it maps those \(r\) numbers back to \(d\) output dimensions
\(r\)
the LoRA rank, typically 4 to 64; a hyper-parameter you choose

Read formula (57.1) as two paths that start from the same input and are added at the end. The wide path is the old layer. The narrow path squeezes \(x\) through a bottleneck of size \(r\) and expands it again. Only the narrow path learns.

Counting the parameters

The two factors have \(dr + rk\) numbers, against \(dk\) for the full matrix:

\[ N_{\text{LoRA}} = r\,(d + k), \qquad \frac{N_{\text{LoRA}}}{N_{\text{full}}} = \frac{r\,(d + k)}{d\,k} \tag{57.2} \]

The saving grows with the size of the layer. For a tiny \(16 \times 16\) matrix, rank 8 already costs as much as the full matrix (\(8 \times 32 = 256\)). For a real layer it is a tiny fraction.

Worked example: one attention projection of Llama 3 8B

The query projection of Llama 3 8B maps 4,096 to 4,096 numbers: \(dk = 4096^2 = 16{,}777{,}216\) weights. With \(r = 16\): \(N_{\text{LoRA}} = 16 \times (4096 + 4096) = 131{,}072\). That is \(0.78\%\) of the matrix.

The key projection is smaller, because Llama 3 uses grouped-query attention with 8 key-value heads of size 128 (session 26): it maps 4,096 to 1,024. LoRA adds \(16 \times (4096 + 1024) = 81{,}920\) numbers. The MLP down projection maps 14,336 to 4,096: \(16 \times (14336 + 4096) = 294{,}912\).

Explore any layer and any rank in Figure 1.

frozen W₀ (d × k)B (d × r), starts at zeroA (r × k), starts random
Figure 1. The shapes of LoRA for one linear layer. The grey block is the frozen weight, drawn to scale. The blue and orange strips are the trainable factors; they are drawn at least 4 pixels wide so you can see them, even when the true width would be less than one pixel.
How to read Figure 1

Move the rank. The trainable count grows linearly in \(r\), while the frozen block does not change. For the real layers, even \(r = 256\) stays a small share. Then pick the toy \(16 \times 16\) layer: at \(r = 8\) LoRA has as many numbers as the full matrix and at \(r = 16\) twice as many. Low rank only saves anything when \(r\) is much smaller than \(dk/(d+k)\).

Initialization and the α/r scale

Two details make LoRA work in practice. Both come from the original paper. [Certain]

B starts at zero

\(A\) is filled with small random numbers, and \(B\) is set to zero. Then \(BA = 0\) at step 0, so the adapted model is exactly the pretrained model when training starts. Training begins from a model that already works, not from a randomly disturbed one.

Why not set both to zero? Look at the gradients. Write \(s\) for the scale (below), \(\mathbf{g} = \partial L/\partial h\) for the gradient that arrives at the layer output, and \(\mathbf{u} = A x\) for the \(r\) numbers in the bottleneck:

\[ \frac{\partial L}{\partial B} = s\, \mathbf{g}\, \mathbf{u}^\top = s\, \mathbf{g}\, (A x)^\top, \qquad \frac{\partial L}{\partial A} = s\, B^\top \mathbf{g}\, x^\top \tag{57.3} \]

At step 0, \(B = 0\), so \(\partial L / \partial A = 0\): \(A\) does not move on the first step. But \(\partial L/\partial B\) is not zero, because \(Ax\) is not zero. After one step \(B\) is non-zero, and from then on both factors learn. If both started at zero, both gradients would be zero forever. If both were random, the model would start with a random change on top of the pretrained weights. One zero, one random is the only choice that gives a clean start and a non-zero gradient. In the PEFT library, \(A\) uses the same Kaiming-uniform initialization as nn.Linear. [Certain]

The scale α/r

The LoRA path is multiplied by a constant:

\[ h = W_0 x + \frac{\alpha}{r}\, B A x \tag{57.4} \]
Read the symbols
\(\alpha\)
“LoRA alpha”, a constant you choose (for example 16 or 32)
\(s = \alpha / r\)
the scale; with \(\alpha = 32, r = 16\) it is 2

Why divide by \(r\)? The bottleneck output \(BAx\) is a sum of \(r\) terms. With more terms, the same learning rate produces a bigger change of \(\Delta W\) per step. Dividing by \(r\) was meant to keep the effective step size roughly the same when you change \(r\), so you do not need to retune the learning rate for each rank. The paper sets \(\alpha\) equal to the first \(r\) they try and does not tune it, noting that with Adam, tuning \(\alpha\) is roughly the same as tuning the learning rate. [Certain]

The common habit today is \(\alpha = 2r\) (scale 2) or \(\alpha = r\) (scale 1). The PEFT library default is \(r = 8\), \(\alpha = 8\). [Certain] Biderman et al. (2024) found that \(\alpha = 2r\) was important at high rank, and that the best LoRA learning rate is about one order of magnitude higher than for full fine-tuning (for example \(2 \times 10^{-4}\) instead of \(2 \times 10^{-5}\)). [Certain for their setting; Likely in general]

Rank-stabilized LoRA: α/√r

Kalajdzievski (2023) showed that dividing by \(r\) divides too much. With a large rank, the update becomes too small and learning slows down, so large ranks look useless even when they are not. Rank-stabilized LoRA (rsLoRA) uses

\[ s_{\text{rs}} = \frac{\alpha}{\sqrt{r}} \tag{57.5} \]

which keeps the size of the update stable as \(r\) grows. In PEFT it is one flag: use_rslora=True. [Certain] A related idea, LoRA+ (Hayou et al., 2024), gives \(B\) a larger learning rate than \(A\), with a fixed ratio between the two; the paper reports up to about 2× faster fine-tuning and 1–2% better results. [Certain] Figure 5 below lets you see the α/r versus α/√r effect on a toy problem.

How low can the rank be?

Any matrix can be written with its singular value decomposition (SVD): \(\Delta W = \sum_i \sigma_i\, \mathbf{u}_i \mathbf{v}_i^\top\), a sum of rank-1 pieces, each with a strength \(\sigma_i \ge 0\) (the singular values, sorted from largest to smallest). The best rank-\(r\) approximation keeps the \(r\) strongest pieces (the Eckart–Young theorem), and its error is set by the ones it drops:

\[ \min_{\operatorname{rank}(M) \le r} \lVert \Delta W - M \rVert_F = \sqrt{\textstyle\sum_{i \gt r} \sigma_i^2} \tag{57.6} \]

\(\lVert \cdot \rVert_F\) is the Frobenius norm: the square root of the sum of all squared entries. So the question “is rank \(r\) enough?” is the question “do the first \(r\) singular values of the needed update carry almost all of its energy \(\sum_i \sigma_i^2\)?” If the spectrum falls fast, a small rank is enough. If it is flat, it is not.

positive entrynegative entrysingular value keptsingular value dropped
Figure 2. A made-up \(16 \times 16\) update \(\Delta W\) with fast-falling singular values (5, 3, 2, 1.2, 0.8, …, 0.01), its best rank-\(r\) approximation, and what is left over. The matrix is built from seeded random orthogonal vectors, so it is the same on every load.
How to read Figure 2

At \(r = 1\) the approximation is a single “stripe” pattern and captures 61.6% of the energy. At \(r = 3\) it captures 93.7% and already looks like the original; at \(r = 6\), 99.4%. The residual (right) fades as \(r\) grows. This is the situation where LoRA works well. A real fine-tuning update can have a flatter spectrum: Biderman et al. measured that full fine-tuning learns changes with a rank 10–100× larger than typical LoRA settings [Certain], which is one reason LoRA can fall behind on hard tasks.

What did the LoRA paper find for GPT-3 175B? With the query and value projections adapted, rank 1 was already competitive with rank 64 on WikiSQL and MultiNLI. [Certain] The authors warn that this will not hold for every task: if the new task were in a language the model never saw, a small rank would not be enough. [Certain]

Which layers to adapt

A transformer block has seven big linear layers in the Llama family (session 21): four in attention (q_proj, k_proj, v_proj, o_proj) and three in the gated MLP (gate_proj, up_proj, down_proj). You can put LoRA on any subset.

  • The original paper adapted only attention weights. With a fixed budget of 18M parameters on GPT-3 175B, spreading the budget over \(W_q\) and \(W_v\) at rank 4 was better than putting it all into \(W_q\) at rank 8. Adapting more matrix types with a smaller rank beat one type with a bigger rank. [Certain]
  • QLoRA (Dettmers et al., 2023) found that the number of LoRA adapters mattered most, and that LoRA on all linear layers of the transformer blocks was needed to match full fine-tuning; the rank mattered much less. [Certain]
  • Since then, “all linear layers” has become the common default; in PEFT you can write target_modules="all-linear". [Certain] In Llama 3 8B the MLP holds about 81% of each block’s weights (176M of 218M), and it is thought to store much of the model’s knowledge, so adapting it matters for tasks that need new knowledge or skills. [Likely]
  • The embedding matrix and the output head (lm_head) are usually left frozen. You train them (fully, not with LoRA) only when you add new tokens, for example new special tokens for a chat template (session 9). [Likely]
Worked example: the whole of Llama 3 8B

From the model’s config.json: 32 layers, hidden size 4,096, MLP size 14,336, 8 key-value heads of size 128 (so k and v project to 1,024), vocabulary 128,256, untied embeddings. Counting every weight gives 8,030,261,248 parameters. [Certain]

Per layer, with \(r = 16\) on all seven linear layers: \(q, o: 2 \times 131{,}072\); \(k, v: 2 \times 81{,}920\); \(\text{gate}, \text{up}, \text{down}: 3 \times 294{,}912\). Sum: 1,310,720 per layer, times 32 layers = 41,943,040 trainable numbers, or 0.522% of the model. In BF16 the adapter file is 2 bytes × 41.9M = 83.9 MB.

Query and value only, \(r = 16\): 6,815,744 numbers (0.085%), a 13.6 MB file.

What LoRA saves, and what it does not

Memory for training with mixed precision and AdamW was 16 bytes per trainable parameter (session 4): weights, gradients, an FP32 master copy, and two Adam states. With LoRA, the frozen weights need only their BF16 copy (2 bytes each), with no gradient and no optimizer state:

\[ M_{\text{LoRA}} \approx 2\,N + 16\,N_{\text{LoRA}} + M_{\text{act}}, \qquad M_{\text{full}} \approx 16\,N + M_{\text{act}} \tag{57.7} \]
Read the symbols
\(N\)
parameters of the base model (frozen in LoRA)
\(N_{\text{LoRA}}\)
trainable LoRA parameters
\(M_{\text{act}}\)
activation memory for the backward pass; it depends on batch size and sequence length

For Llama 3 8B with the all-linear rank-16 adapter: \(2 \times 8.03\text{B} = 16.06\) GB plus \(16 \times 41.9\text{M} = 0.67\) GB, about 16.7 GB, against 128.5 GB for full fine-tuning, both before activations. Note what does not shrink: the activations. The backward pass still has to go through every frozen layer to reach the adapters in earlier layers, so activations are about as large as in full fine-tuning; gradient checkpointing (session 39) is still useful. [Certain] The compute per step also stays close to full fine-tuning: the forward pass is the same, and the backward pass still computes input gradients through every layer, it only skips the weight gradients of frozen matrices. The LoRA paper measured a 25% training speedup on GPT-3 175B. [Certain]

Figure 3. A LoRA calculator for Llama 3 8B (32 layers, hidden 4,096, MLP 14,336, 8 key-value heads). Top: trainable parameters per module type, summed over the 32 layers. Bottom: training memory before activations, with formula (57.7), against full fine-tuning.
How to read Figure 3

The three MLP matrices dominate the adapter size, because their inputs or outputs have 14,336 dimensions. Going from “q, v only” to “all linear” multiplies the adapter by about 6, yet the memory bar barely moves: the frozen BF16 weights (16 GB) are the main cost at any reasonable rank. To shrink those, you quantize the base model: that is QLoRA, the next session.

Training LoRA: a toy run

The algorithm is ordinary training with two changes: most weights are frozen, and the layer computes two paths.

Algorithm 1: LoRA fine-tuning
  1. input: pretrained model, list of target layers, rank \(r\), \(\alpha\), learning rate \(\eta\)
  2. freeze every parameter of the model
  3. for each target linear layer \(W_0 \in \R^{d \times k}\): create \(A \in \R^{r \times k}\) (random, small) and \(B \in \R^{d \times r}\) (zeros); set \(s = \alpha/r\) (or \(\alpha/\sqrt{r}\))
  4. replace the layer’s forward by \(h = W_0 x + s\,B(Ax)\) // formula (57.4); compute \(Ax\) first, it is only \(r\) numbers
  5. create AdamW over the \(A\) and \(B\) matrices only
  6. for each batch: forward, loss (with the SFT loss mask, session 54), backward, optimizer step
  7. save only the \(A\) and \(B\) matrices and the config (\(r\), \(\alpha\), target layers): the adapter

Figure 4 opens up this loop for one tiny layer, so you can watch every line run with real numbers. The frozen layer \(W_0\) maps \(k = 8\) inputs to \(d = 6\) outputs. A “teacher” layer \(W_0 + \Delta W^\star\) produces the answers, where the needed change \(\Delta W^\star\) has rank 3 with singular values 2, 1 and 0.5. Each step draws one random input \(x\) (seeded, so every load is the same), runs the two paths, computes the loss \(L = \tfrac12 \lVert h - y \rVert^2\) and the gradients of formula (57.3), and takes a plain gradient step (no Adam, so the update is exactly \(-\eta \nabla\)). Because \(W_0\) appears in both \(h\) and \(y\), it cancels in \(g = h - y = (sBA - \Delta W^\star)x\): LoRA only has to learn the change. For random inputs with \(\E[x x^\top] = I\), the average loss is \(\tfrac12 \lVert sBA - \Delta W^\star \rVert_F^2\), which the figure calls the full loss; formula (57.6) says it cannot go below \(\tfrac12 \sum_{i \gt r} \sigma_i^2\).

positive entrynegative entrybox used by the current line; learned valueserror and gradients
  1. \(x \sim \mathcal{N}(0, I)\), \(y \leftarrow (W_0 + \Delta W^\star)\,x\) // one example
  2. \(u \leftarrow A x\) // \(r\) numbers
  3. \(h \leftarrow W_0 x + s\,B u\) // (57.4)
  4. \(g \leftarrow h - y\), \(L = \tfrac12 \lVert g \rVert^2\)
  5. \(\nabla B \leftarrow s\,g\,u^\top\), \(\nabla A \leftarrow s\,(B^\top g)\,x^\top\) // (57.3)
  6. \(B \leftarrow B - \eta \nabla B\), \(A \leftarrow A - \eta \nabla A\)
  7. after training: \(W' \leftarrow W_0 + s\,BA\) // merge (57.8)
Figure 4. One LoRA layer trained in your browser, line by line. Top: the data flow of one step; each box is a matrix or vector, blue cells positive, orange negative. Bottom left: the target change \(\Delta W^\star\), the learned change \(sBA\), and their singular values (grey slots are beyond rank \(r\), so \(sBA\) can never fill them). Bottom right: the full loss \(\tfrac12 \lVert sBA - \Delta W^\star \rVert_F^2\) and the floor of formula (57.6). Plain SGD, batch size 1, 400 steps.
How to read Figure 4

Press Step a few times. On step 1, line 3 shows \(h = W_0 x\) exactly, because \(B = 0\): the model starts as the pretrained model. Line 5 shows \(\nabla A = 0\) while \(\nabla B\) is not zero, as formula (57.3) predicts. Now set the speed to the right and press Play. Watch the bars: the largest singular value (2) is learned first, the smaller ones later, with some noise because each step sees only one example. With the default \(r = 2\) the loss stops near the floor \(\tfrac12 \times 0.5^2 = 0.125\), because the third direction is blocked; set \(r = 3\) or 4 and it goes toward zero. Try “both zero”: every gradient is zero and the loss stays at 2.625 forever. Try “both random”: the model starts with a random change, so the first loss is higher than the pretrained model’s (6.158 at \(r = 2\)). After training, press Merge: \(W'x\) equals the two-path output up to rounding. The readout counts the trainable numbers \(r(d + k) = 14r\): for this tiny \(6 \times 8\) layer LoRA saves little, which is why real layers (Figure 1) matter.

Figure 5 runs the same idea at a larger size, with Adam. The “needed update” is a fixed \(24 \times 24\) matrix of rank exactly 4, with singular values 3, 2, 1.2 and 0.6. A LoRA pair \(B, A\) of rank \(r\) is trained with Adam to reproduce it. The loss is the mean squared difference between \(s\,BA\) and the target.

your rank rranks 1, 2, 4, 16 for comparisondashed: best possible loss for rank r, formula (57.6)
Figure 5. LoRA trained with Adam (\(\beta_1 = 0.9, \beta_2 = 0.999\)) for 300 steps on a rank-4 target, with \(\alpha = 8\). The vertical axis is the loss on a log scale. Below: the target, the learned update \(sBA\) at the current step, and their singular values. Everything is seeded, so the curves are the same on every load.
How to read Figure 5

Press Play to replay the 300 steps, or drag the step slider. Watch the bars: the learned singular values grow toward the target ones (3, 2, 1.2, 0.6), at most \(r\) of them, and \(sBA\) fills in the heat map. With \(r = 1\) or 2 the loss stops at the dashed line: \((1.2^2 + 0.6^2)/576 = 0.003125\) for \(r = 2\). No training can beat formula (57.6); the rank is simply too low. From \(r = 4\) on, the loss falls toward zero. Now switch to α/√r: for small ranks almost nothing changes, but rank 16 learns visibly faster, because α/r had shrunk its update by an extra factor \(\sqrt{16} = 4\). This is the rsLoRA argument in miniature. Push the learning rate up and some curves become noisy: LoRA is sensitive to the learning rate.

The same thing in PyTorch, written from scratch so every line is visible. It wraps an nn.Linear, checks that the model is unchanged at step 0, trains on a rank-2 target, and checks that merging gives the same outputs:

import math, torch, torch.nn as nn
torch.manual_seed(0)

class LoRALinear(nn.Module):
    """y = x W0^T + (alpha/r) * x A^T B^T. W0 is frozen; only A and B train."""
    def __init__(self, base: nn.Linear, r=8, alpha=16, dropout=0.0):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad_(False)                     # freeze W0 (and its bias)
        d_out, d_in = base.weight.shape
        self.A = nn.Parameter(torch.empty(r, d_in))     # r x k
        self.B = nn.Parameter(torch.zeros(d_out, r))    # d x r, zero: Delta W = 0 at step 0
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))  # same init as nn.Linear (and PEFT)
        self.scale = alpha / r                          # formula (57.4)
        self.drop = nn.Dropout(dropout)
        self.merged = False

    def forward(self, x):
        y = self.base(x)
        if not self.merged:
            y = y + self.scale * (self.drop(x) @ self.A.T @ self.B.T)
        return y

    @torch.no_grad()
    def merge(self):                                    # formula (57.8)
        self.base.weight += self.scale * (self.B @ self.A)
        self.merged = True

    @torch.no_grad()
    def unmerge(self):
        self.base.weight -= self.scale * (self.B @ self.A)
        self.merged = False

# a frozen "pretrained" layer, and a target that differs from it by a rank-2 change
d = 64
base = nn.Linear(d, d)
U, V = torch.randn(d, 2), torch.randn(2, d)
W_target = base.weight.detach() + 0.05 * U @ V

layer = LoRALinear(base, r=4, alpha=8)
trainable = sum(p.numel() for p in layer.parameters() if p.requires_grad)
total = sum(p.numel() for p in layer.parameters())
print("trainable", trainable, "of", total)             # 512 of 4672

x0 = torch.randn(8, d)
print("same output at step 0:", torch.allclose(layer(x0), base(x0)))  # True, because B = 0

opt = torch.optim.AdamW([layer.A, layer.B], lr=1e-2, weight_decay=0.0)
for step in range(301):
    x = torch.randn(256, d)
    y = x @ W_target.T + base.bias.detach()
    loss = ((layer(x) - y) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 100 == 0:
        print(step, f"{loss.item():.2e}")                # 2.92e-01, then about 1e-05 and below

# merging gives the same outputs with zero extra cost at inference
x = torch.randn(16, d)
with torch.no_grad():
    y_unmerged = layer(x)
    layer.merge()
    y_merged = layer(x)
print("max diff after merge:", (y_unmerged - y_merged).abs().max().item())   # about 1e-06
print("singular values of the update:",
      torch.linalg.svdvals(layer.scale * layer.B @ layer.A)[:4].round(decimals=3).tolist())
# about [3.186, 2.785, 0.0, 0.0]: rank 4 was allowed, the learned update has rank 2, like the target

The last line is a nice check: the target change \(0.05\,UV\) has exactly two non-zero singular values (3.186 and 2.785), and the trained rank-4 adapter finds those two and leaves the other two directions at zero.

Merging, unmerging and serving many adapters

After training, \(W_0\) and \(BA\) have the same shape, so we can add them once:

\[ W' = W_0 + \frac{\alpha}{r}\, B A, \qquad W' x = W_0 x + \frac{\alpha}{r}\, B A x \tag{57.8} \]

The merged model is a normal model of the original shape, with zero extra latency. To switch tasks, subtract \(\tfrac{\alpha}{r} BA\) and add another adapter \(\tfrac{\alpha'}{r'} B'A'\). [Certain] In PEFT this is merge_and_unload().

Worked example: merge a rank-1 adapter by hand

Take \(W_0 = \begin{pmatrix} 1 & 0 & 2 \\ 0 & 1 & -1 \\ 1 & 1 & 0 \end{pmatrix}\), \(x = (1, 2, -1)\), \(A = (0.5,\ -1,\ 1)\), \(B = (2, 0, -1)^\top\), \(r = 1\), \(\alpha = 2\), so \(s = 2\).

Two paths: \(W_0 x = (-1, 3, 3)\). The bottleneck is one number, \(Ax = 0.5 - 2 - 1 = -2.5\). Then \(s\,B\,(Ax) = 2 \times (2, 0, -1) \times (-2.5) = (-10, 0, 5)\). Sum: \(h = (-11, 3, 8)\).

Merged: \(s\,BA = \begin{pmatrix} 2 & -4 & 4 \\ 0 & 0 & 0 \\ -1 & 2 & -2 \end{pmatrix}\), so \(W' = \begin{pmatrix} 3 & -4 & 6 \\ 0 & 1 & -1 \\ 0 & 3 & -2 \end{pmatrix}\) and \(W' x = (-11, 3, 8)\). The same answer, with one matrix product instead of three.

Merging is not always what you want. If one server must answer requests for many customers, each with its own adapter, merging would need one full copy of the model per adapter. The alternative is to keep \(W_0\) shared and not merge: every request runs the shared base product, and only the cheap LoRA path differs per request. For one layer and one token the extra cost is

\[ \frac{\text{extra FLOPs}}{\text{base FLOPs}} = \frac{2\,r\,(d + k)}{2\,d\,k} = \frac{r\,(d + k)}{d\,k} \tag{57.9} \]

the same ratio as formula (57.2): 0.78% for a \(4096 \times 4096\) layer at \(r = 16\). In practice the overhead is larger than this FLOP count, because the small LoRA products use the GPU poorly and need extra kernel launches. [Likely] This is why special systems exist. Punica (Chen et al., 2023) wrote a CUDA kernel that computes the LoRA paths of a batch where each row uses a different adapter. S-LoRA (Sheng et al., 2023) keeps all adapters in CPU memory, loads the ones in use to the GPU, manages adapter weights and KV cache in one paged memory pool, and reports serving thousands of adapters on one GPU with up to 4× the throughput of HuggingFace PEFT and of vLLM with naive LoRA support. [Certain] Today vLLM and similar servers have built-in multi-LoRA serving. [Likely]

Figure 6. Four ways to use a trained adapter, with the numbers of the worked example above: two paths during training, one merged matrix for a single-task server, an unmerge-and-swap, and one batch that mixes adapters on a shared base model.
How to read Figure 6

Step through. In steps 1–2 the output (−11, 3, 8) is the same with two paths and with one merged matrix. Step 3 shows that unmerging gives back \(W_0\) exactly in exact arithmetic; in BF16, repeated merge and unmerge adds rounding error, so keep the original weights and re-merge from them. [Likely] Step 4 is the multi-tenant case: one big shared product for the whole batch, plus a small gathered product per request.

Algorithm 2: one decoding step with many adapters (unmerged)
  1. input: batch of \(n\) tokens \(X \in \R^{n \times k}\), adapter id \(a_j\) of each row \(j\), adapters \(\{(A_a, B_a, s_a)\}\) on the GPU
  2. \(H \leftarrow X W_0^\top\) // one large product, shared by every request
  3. for each row \(j\) (in practice: one batched “gather” kernel, as in Punica’s SGMV)
  4.   \(\mathbf{u}_j \leftarrow A_{a_j} \mathbf{x}_j\) // \(r\) numbers
  5.   \(\mathbf{h}_j \leftarrow \mathbf{h}_j + s_{a_j} B_{a_j} \mathbf{u}_j\)
  6. rows with no adapter skip the loop; return \(H\)

DoRA: magnitude and direction

Every column of a weight matrix is a vector, and a vector has a length (magnitude) and a direction. Liu et al. (2024) compared how full fine-tuning and LoRA change these two parts. In full fine-tuning, changes of magnitude and direction were negatively correlated (correlation −0.62); LoRA showed a strong positive correlation (0.83), so it tends to change both together. [Certain] Their method, DoRA (Weight-Decomposed Low-Rank Adaptation), separates the two:

\[ W' = \mathbf{m} \odot \frac{W_0 + B A}{\lVert W_0 + B A \rVert_c} \tag{57.10} \]
Read the symbols
\(\lVert \cdot \rVert_c\)
the norm of each column separately; dividing by it makes every column a unit vector (pure direction)
\(\mathbf{m} \in \R^{1 \times k}\)
a trainable magnitude, one number per column, initialized to \(\lVert W_0 \rVert_c\) so that \(W' = W_0\) at the start
\(BA\)
an ordinary LoRA update; here it only steers the direction
\(\odot\)
multiply column \(j\) by \(m_j\)
LoRA: one update for both DoRA: direction and length apart w₀ BA column w₀ + BA length and angle are tied to the same B, A w₀ × m (own length) unit directionof w₀ + BA unit direction from LoRA, length from m
Figure 7. One column of a weight matrix. Left: LoRA adds a low-rank column and the new vector’s length and angle both come from that one change. Right: DoRA normalizes \(\mathbf{w}_0 + BA\) to a unit direction and multiplies it by a separately trained length \(m\).

DoRA adds only \(k\) numbers per adapted matrix (the vector \(\mathbf{m}\)), and after training it can be merged just like LoRA, so inference cost is unchanged. [Certain] The paper reports that DoRA beats LoRA when fine-tuning LLaMA, LLaVA and VL-BART on tasks such as commonsense reasoning, visual instruction tuning and image and video-text understanding. [Certain] The cost is training time and memory: the column norm must be computed every step. To save memory, the paper treats the norm as a constant in the backward pass (no gradient through it); this cut training memory by about 24% when fine-tuning LLaMA in their ablation, with no noticeable loss in accuracy. [Certain] In PEFT: LoraConfig(use_dora=True). [Certain]

The wider family: what each method changes

LoRA is not the only small change you can learn. Here is how it sits next to the methods of session 56 and a few newer LoRA variants.

Prompt tuning and prefix tuning

Prompt tuning (Lester et al., 2021) learns \(p\) “virtual token” vectors and puts them in front of the input embeddings. Nothing inside the model changes:

\[ H^{(0)} = [\,P\,;\ E(x)\,], \qquad P \in \R^{p \times d_{\text{model}}} \tag{57.11} \]

Prefix tuning (Li and Liang, 2021) goes deeper: in every layer, it adds \(p\) learned key and value vectors in front of the real keys and values, so every query can attend to them:

\[ \operatorname{Attn}\big(Q,\ [P_K^{(\ell)}; K],\ [P_V^{(\ell)}; V]\big) \tag{57.12} \]

For Llama 3 8B with \(p = 20\): prompt tuning trains \(20 \times 4096 = 81{,}920\) numbers. Prefix tuning trains \(20 \times 32 \text{ layers} \times 2 \times 1024 = 1{,}310{,}720\) (the key and value width is 1,024 because of grouped-query attention). Both are tiny. The prefix-tuning paper trained about 0.1% of the parameters of GPT-2 and BART and matched fine-tuning on table-to-text and summarization with full data, and did better with little data. [Certain] Prompt tuning on T5 became competitive with full model tuning only once models had billions of parameters; for small models it was clearly worse. [Certain]

Two practical costs: the virtual tokens use part of the context window and add to the KV cache at every request, and they cannot be merged away. Training is also known to be less stable than LoRA; prefix tuning used an extra MLP during training (a “reparameterization”) to make it stable. [Certain] Their strength is in serving: a batch can mix tasks freely, because each request just carries its own prefix.

Other ways to make the change small

MethodWhat it trainsMergeable?When it helps
LoRA\(B, A\) with \(\Delta W = \tfrac{\alpha}{r} BA\) on chosen linear layersYesThe default for SFT and preference tuning on one GPU
rsLoRASame as LoRA, scale \(\alpha/\sqrt{r}\)YesWhen you want ranks of 64 and more to actually help
LoRA+Same as LoRA, larger learning rate for \(B\) than for \(A\)YesFaster convergence at no extra cost [Certain: up to about 2× in the paper]
DoRALoRA for direction plus a magnitude vector \(\mathbf{m}\)YesSmall ranks, when LoRA lags full fine-tuning; costs more training time
PiSSALoRA whose \(B, A\) start from the top singular vectors of \(W_0\); the rest of \(W_0\) is frozenYesFaster start, because the trainable part begins with the most important directions [Likely]
IA3One scaling vector per adapted activation (keys, values, MLP), \(h = \mathbf{l} \odot (W_0 x)\)YesExtremely few parameters; few-shot tasks
VeRAOnly small scaling vectors; \(B, A\) are frozen random matrices shared by all layersYesWhen even LoRA files are too big (thousands of users) [Likely]
Adapters (Houlsby)Small bottleneck MLPs inserted in each blockNo (they are extra layers)Historical baseline; adds inference latency
BitFitOnly the biasesYes (it is the model)Encoders like BERT; modern LLMs often have no biases at all
Prompt / prefix tuningVirtual tokens, or per-layer key/value prefixesNoVery large frozen models, many tasks mixed in one batch

A good rule in practice: start with LoRA on all linear layers, \(r = 16\), \(\alpha = 32\), learning rate around \(10^{-4}\) to \(2 \times 10^{-4}\), and change only when you have measured a reason. [Likely] If LoRA clearly lags full fine-tuning, first try a higher rank (with rsLoRA or \(\alpha = 2r\)) and a learning-rate sweep, then DoRA.

LoRA learns less and forgets less

Is LoRA as good as full fine-tuning? It depends on how far the task is from what the model already knows. Biderman et al. (2024) trained Llama-2-7B on code and math, both with instruction data (about 100K examples) and with continued pretraining (up to 20B tokens). In the standard low-rank settings, LoRA learned clearly less than full fine-tuning on the target domain. But it kept more of the base model’s skills on other tasks, more than weight decay or dropout did, and its outputs stayed more diverse. [Certain] So LoRA is a good fit for teaching a model a format, a style, a task or a moderate amount of domain data, and a weaker fit for pouring in a lot of new knowledge. [Likely] The full comparison, with QLoRA, is in session 62, and forgetting itself in session 61.

The PEFT library version of everything above. This runs offline, on a tiny randomly initialized Llama that has the same layer names as Llama 3:

import torch
from transformers import LlamaConfig, LlamaForCausalLM
from peft import LoraConfig, get_peft_model
torch.manual_seed(0)

# a tiny random Llama with the same layer names as Llama 3 (no download needed)
cfg = LlamaConfig(vocab_size=1000, hidden_size=256, intermediate_size=896,
                  num_hidden_layers=4, num_attention_heads=8, num_key_value_heads=2)
model = LlamaForCausalLM(cfg).eval()

lcfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.0,
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                  "gate_proj", "up_proj", "down_proj"],  # all linear layers in the blocks
                  task_type="CAUSAL_LM")          # add use_rslora=True or use_dora=True to try the variants
model = get_peft_model(model, lcfg)
model.print_trainable_parameters()
# trainable params: 327,680 || all params: 4,249,856 || trainable%: 7.7104

ids = torch.randint(0, 1000, (2, 12))
with torch.no_grad():
    for n, p in model.named_parameters():        # pretend we trained: make B non-zero
        if "lora_B" in n:
            p.normal_(0, 0.02)
    y_adapter = model(ids).logits
    merged = model.merge_and_unload()            # W0 + (alpha/r) B A, adapters removed
    y_merged = merged(ids).logits
print("max diff:", (y_adapter - y_merged).abs().max().item())   # about 1e-06
print(type(merged).__name__)                     # LlamaForCausalLM: a plain model again

Check the count with formula (57.2): per layer \(16 \times [\,2(256 + 256) + 2(256 + 64) + 3(256 + 896)\,] = 81{,}920\), times 4 layers = 327,680. The share is large (7.7%) only because this toy model is tiny; for Llama 3 8B the same config gives 0.52%. (Tested with PEFT 0.17 and transformers 4.57.)

Where this is used

SettingHow PEFT is usedWhy
Instruction tuning and domain SFT of open LLMsLoRA or QLoRA on all linear layers, ranks 8–64, one adapter per taskFits on one GPU; adapter files of tens of MB (session 54)
Preference tuning (DPO)LoRA on the policy; the frozen base can serve as the reference model by turning the adapter off [Likely]Saves the memory of a second full model (session 68)
Multi-tenant servingOne base model, many unmerged adapters, mixed in one batch (S-LoRA, Punica, vLLM)Per-customer models at the cost of one deployment
VLM fine-tuningLoRA on the language model, often with the vision encoder frozen and the projector trained fully [Likely]Adapt to documents, charts or a new domain cheaply (session 189)
Video understanding with a Video-LLMLoRA adapters for action recognition, video question answering, or key-moment detection prompts on a shared Video-LLM [Likely]One base model, a small adapter per video task or customer (session 205)
Video retrieval and fingerprinting encodersLoRA or prompt tuning on a frozen CLIP-style encoder for a new domain (sports, surveillance) [Likely]Keeps the general embedding space, so old indexes stay roughly usable (session 208)
Image and video generationLoRA on diffusion transformers or U-Nets for a style or a subject; adapters are shared and combined by users [Likely]Small files, easy to mix and switch
On-device modelsOne quantized base model on the phone, small task adapters swapped in at run time [Likely]Storage and memory on the device are tight

Common mistakes

  • Forgetting to freeze the base. If requires_grad stays on for \(W_0\), you are doing full fine-tuning with extra parameters, and the memory saving is gone. Always print the trainable parameter count.
  • Initializing \(B\) randomly. Then the model starts with a random change and the first steps damage it. \(B = 0\) (or a method like PiSSA that starts with \(W'\) exactly equal to \(W_0\)).
  • Changing \(r\) without thinking about \(\alpha\). With a fixed \(\alpha\), doubling \(r\) halves the scale. Keep \(\alpha / r\) fixed (for example \(\alpha = 2r\)) or use rsLoRA when you sweep the rank.
  • Using the full fine-tuning learning rate. LoRA usually needs a learning rate about 10× higher (around \(10^{-4}\)); with \(2 \times 10^{-5}\) it may barely move. Sweep it.
  • Adapting only q and v and concluding that “LoRA does not work”. On hard tasks, add the MLP layers before raising the rank.
  • Merging into quantized weights carelessly. Adding \(BA\) to 4-bit weights and re-quantizing loses precision. Merge into a BF16 copy of the base, then quantize again if needed (session 58).
  • Loading an adapter on the wrong base. An adapter is a difference from one exact set of weights. On another checkpoint (even another revision of the same model) it gives wrong results without any error message.
  • Expecting less activation memory. LoRA removes optimizer states and weight gradients, not activations. Long sequences still need gradient checkpointing.

Interview questions

Explain LoRA in a few sentences.basic
LoRA freezes a pretrained weight \(W_0 \in \R^{d \times k}\) and learns its change as a low-rank product \(\Delta W = BA\) with \(B \in \R^{d \times r}\), \(A \in \R^{r \times k}\) and small \(r\). The layer computes \(W_0 x + \tfrac{\alpha}{r} BAx\). Only \(A\) and \(B\) are trained, so the number of trainable parameters and the optimizer memory drop by orders of magnitude, and after training \(BA\) can be added into \(W_0\) so inference costs nothing extra.
Why is B initialized to zero and A randomly? What happens with other choices?core
With \(B = 0\), \(\Delta W = 0\), so training starts exactly at the pretrained model. The gradient of \(B\) is \(s\,\mathbf{g}(Ax)^\top\), which is non-zero because \(A\) is random, so \(B\) moves on the first step, and then \(A\) gets gradient \(sB^\top \mathbf{g} x^\top\) too. If both were zero, both gradients would be zero forever (a saddle point). If both were random, the model would start with a random perturbation of its weights. (Zero \(A\) and random \(B\) would also work in principle; the convention is zero \(B\).)
How many trainable parameters does LoRA rank 16 add to a 4096 × 4096 projection? To all linear layers of Llama 3 8B?core
One layer: \(r(d + k) = 16 \times 8192 = 131{,}072\), 0.78% of the 16.8M weights. Llama 3 8B, all seven linear layers in all 32 blocks: per block \(2 \times 131{,}072\) (q, o) \(+ 2 \times 81{,}920\) (k, v, which output 1,024 because of GQA) \(+ 3 \times 294{,}912\) (MLP) = 1,310,720; times 32 = 41.9M, 0.52% of 8.03B, an 84 MB file in BF16.
What does α do? What is rsLoRA?core
The LoRA path is scaled by \(s = \alpha/r\). With Adam, scaling the update is close to scaling the learning rate, so \(\alpha\) is effectively a learning-rate knob; dividing by \(r\) was meant to make the step size independent of the rank. Kalajdzievski showed that \(1/r\) overcorrects: high-rank adapters then learn too slowly and look useless. rsLoRA uses \(\alpha/\sqrt{r}\), which keeps the update size stable as \(r\) grows, so higher ranks can actually improve results. A common practical alternative is to keep \(\alpha = 2r\).
Which layers would you adapt, and why?core
The original paper adapted only \(W_q\) and \(W_v\) and showed that spreading a budget over more matrix types beats a higher rank on one type. QLoRA found that adapting all linear layers in the blocks (attention and MLP) was needed to match full fine-tuning, while the rank mattered less. So the default today is all linear layers. Embeddings and the LM head stay frozen unless you add new tokens, in which case you train those rows (or the whole matrices) fully.
Where does LoRA save memory, and where does it not?core
It saves the gradients, FP32 master weights and Adam states of all frozen weights: memory becomes about \(2N + 16N_{\text{LoRA}}\) bytes instead of \(16N\) (8B model: about 16.7 GB vs 128.5 GB before activations). It does not save activation memory, because backprop still flows through every layer to reach earlier adapters, and it does not reduce the forward compute. The frozen BF16 weights become the main cost; QLoRA attacks them by storing the base in 4 bits.
You need to serve 200 customer-specific fine-tunes of one 8B model. How?advanced
Train one LoRA adapter per customer on the same base. Do not merge: keep one copy of the base weights on the GPU and keep adapters (tens of MB each) in CPU memory, loading the active ones. In each batch, run the shared base product for all rows, then the per-row LoRA path with a gather kernel (Punica’s SGMV, S-LoRA, or vLLM’s multi-LoRA support). Extra FLOPs are about \(r(d+k)/(dk)\), under 1% at rank 16, though real overhead is higher due to small kernels. For one very hot customer you can deploy a merged copy for zero overhead.
What is DoRA, and what does it cost?advanced
DoRA writes the adapted weight as \(\mathbf{m} \odot (W_0 + BA)/\lVert W_0 + BA \rVert_c\): a trainable magnitude per column times a unit direction steered by LoRA. The authors found full fine-tuning changes magnitude and direction with a negative correlation, while LoRA ties them positively; separating them made LoRA behave more like full fine-tuning and gave better results on LLaMA, LLaVA and VL-BART. It adds only \(k\) parameters per matrix and can still be merged, but training is slower and uses more memory because of the column norms; the paper detaches the norm from the gradient to save about 24% memory.
Prompt tuning, prefix tuning or LoRA: when would you choose each?core
Prompt tuning trains only input virtual tokens: the fewest parameters and free task mixing in a batch, but it is weak on models below billions of parameters and uses context length. Prefix tuning adds learned keys and values in every layer: stronger than prompt tuning, competitive with fine-tuning on generation tasks for GPT-2 and BART, but it also costs KV cache and cannot be merged. LoRA changes the weights themselves, is usually the strongest and most stable, and can be merged for zero inference cost. Default to LoRA; consider prompt or prefix tuning when one frozen model must serve many tasks in one batch and you cannot add the LoRA kernels.
Is LoRA as good as full fine-tuning?advanced
Often close on instruction-style tasks, but not always. Biderman et al. (2024) found that on code and math, with both instruction data and continued pretraining on billions of tokens, standard low-rank LoRA learned clearly less than full fine-tuning, while forgetting less of the base model’s abilities and keeping more diverse outputs. Full fine-tuning updates had a rank 10–100× higher than typical LoRA ranks. So for a lot of new knowledge, use high ranks with \(\alpha = 2r\) or rsLoRA, all layers, a tuned learning rate, or full fine-tuning; for format, style and moderate domain shift, LoRA is usually enough.
Can you merge a LoRA adapter into a 4-bit quantized model?advanced
Not directly without loss. \(BA\) is a dense high-precision matrix; adding it to quantized weights requires dequantizing, adding and re-quantizing, and the new rounding error can undo part of what the adapter learned. The usual path is to merge into a BF16 copy of the base and quantize the merged model afterwards, or to keep the adapter unmerged at inference. This is part of the QLoRA story in the next session.

Exercises

  1. A linear layer maps 8,192 inputs to 28,672 outputs. How many LoRA parameters does rank 32 add? What share of the layer is that?
  2. For which rank does LoRA on a \(d \times k\) matrix have as many parameters as the matrix itself? Compute it for \(d = k = 16\) and for \(d = 4096, k = 1024\).
  3. At the first training step, \(B = 0\). Using formula (57.3), which of \(A\) and \(B\) changes? Explain why both change from step 2.
  4. You trained with \(r = 8, \alpha = 16\). A colleague retrains with \(r = 32\) and keeps \(\alpha = 16\). By what factor does the scale change? What \(\alpha\) keeps it the same? What \(\alpha\) gives rsLoRA the same scale as the original at \(r = 32\)?
  5. In Figure 2, how much of the energy does rank 4 capture, and what is the relative error \(\lVert \Delta W - M \rVert_F / \lVert \Delta W \rVert_F\)? (Singular values 5, 3, 2, 1.2, 0.8, 0.5, 0.35, 0.25, 0.18, 0.12, 0.09, 0.06, 0.04, 0.03, 0.02, 0.01.)
  6. A base model has 8.03B parameters. Estimate the training memory before activations for full fine-tuning and for LoRA with 42M trainable parameters. Then: how much would storing the base in 4 bits (0.5 bytes per parameter) save?
Answerscheck
1. \(32 \times (8192 + 28672) = 32 \times 36{,}864 = 1{,}179{,}648\). The layer has \(8192 \times 28672 = 234{,}881{,}024\) weights, so 0.50%. 2. \(r(d + k) = dk\) gives \(r = dk/(d + k)\): 8 for \(16 \times 16\), and \(4096 \times 1024 / 5120 = 819.2\) for the second. 3. Step 1: \(\partial L/\partial A = sB^\top \mathbf{g}x^\top = 0\), so only \(B\) moves (its gradient \(s\,\mathbf{g}(Ax)^\top\) is non-zero). From step 2, \(B \ne 0\), so \(A\) gets a gradient too. (With Adam, the moment estimates of \(A\) also start from this zero gradient.) 4. Scale goes from \(16/8 = 2\) to \(16/32 = 0.5\): 4 times smaller. Keep it with \(\alpha = 64\). rsLoRA: \(\alpha/\sqrt{32} = 2\) gives \(\alpha = 2\sqrt{32} \approx 11.3\). 5. \((25 + 9 + 4 + 1.44)/40.5765 = 97.20\%\); relative error \(\sqrt{1 - 0.9720} = 0.167\). 6. Full: \(16 \times 8.03 = 128.5\) GB. LoRA: \(2 \times 8.03 + 16 \times 0.042 = 16.06 + 0.67 \approx 16.7\) GB. With a 4-bit base: \(0.5 \times 8.03 = 4.0\) GB instead of 16.06, a saving of about 12 GB, so about 4.7 GB in total (ignoring the small quantization scales).

Key takeaways

  • LoRA freezes \(W_0\) and learns \(\Delta W = BA\) of rank \(r\) (57.1): \(r(d + k)\) parameters instead of \(dk\) (57.2), 0.52% of Llama 3 8B at rank 16 on all linear layers.
  • \(B = 0\) and random \(A\) start training exactly at the pretrained model with a non-zero gradient (57.3). The scale \(\alpha/r\) (57.4) acts like a learning rate; rsLoRA’s \(\alpha/\sqrt{r}\) (57.5) makes high ranks useful.
  • Rank \(r\) can only capture the top \(r\) singular directions of the needed change (57.6). Adapting all linear layers usually matters more than a big rank.
  • LoRA cuts memory from about \(16N\) to \(2N + 16N_{\text{LoRA}}\) bytes (57.7), but not activations. Merging (57.8) gives zero inference cost; unmerged adapters let one server run many tasks at under 1% extra FLOPs (57.9).
  • DoRA (57.10) separates magnitude and direction; prompt and prefix tuning (57.11, 57.12) learn virtual tokens and cannot be merged. LoRA learns less than full fine-tuning on hard domains, but forgets less.

Key formulas

\[ h = W_0 x + \frac{\alpha}{r} B A x, \qquad N_{\text{LoRA}} = r(d + k), \qquad W' = W_0 + \frac{\alpha}{r} BA \]
\[ s_{\text{rs}} = \frac{\alpha}{\sqrt{r}}, \qquad W'_{\text{DoRA}} = \mathbf{m} \odot \frac{W_0 + BA}{\lVert W_0 + BA \rVert_c}, \qquad M_{\text{LoRA}} \approx 2N + 16 N_{\text{LoRA}} \]

References for this session

  1. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  2. Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation
  3. Li & Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation
  4. Lester et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning
  5. Hugging Face (code). PEFT library
  6. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  7. Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation
  8. Li & Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation
  9. Lester et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning
  10. Hugging Face (code). PEFT library
  11. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
  12. Kalajdzievski (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
  13. Hayou, Ghosh and Yu (2024). LoRA+: Efficient Low Rank Adaptation of Large Models
  14. Sheng et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters
  15. Chen et al. (2023). Punica: Multi-Tenant LoRA Serving
  16. Biderman et al. (2024). LoRA Learns Less and Forgets Less
  17. Liu et al. (2022). Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (IA3)
  18. Kopiczko et al. (2023). VeRA: Vector-based Random Matrix Adaptation
  19. Meng et al. (2024). PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
  20. Houlsby et al. (2019). Parameter-Efficient Transfer Learning for NLP
  21. Ben Zaken et al. (2021). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models