A LoRA adapter from scratch

You can already say that LoRA trains a small number of extra parameters instead of all of them, and that this is what made fine-tuning cost tens of dollars rather than millions. This section writes the layer. It is about fifteen lines, and writing them is the quickest way to stop treating r and alpha as dials to copy from someone else's configuration file — which matters, because the first question after "what is LoRA" is almost always what those two numbers do.

The claim it rests on#

Full fine-tuning learns an update to every weight matrix: the new weight is the old one plus some delta of the same shape. The observation behind LoRA is that this delta, for a model being adapted to one task, has much lower rank than the matrix it is added to — the change is real but it lives in a small subspace. If that is true, the delta can be written as a product of two thin matrices and only those need to be trained.

full fine-tuning    W_new = W + dW           dW is (d_out, d_in), all of it trained
LoRA                W_new = W + (a/r) B A    B is (d_out, r), A is (r, d_in)

                    r is much smaller than either dimension, typically 8 to 64

The layer#

The base weight is frozen: it is read in the forward pass and never receives a gradient. The adapter runs alongside it on the same input, and the two outputs are added. Nothing is inserted in sequence, which is the difference from the older adapter designs and the reason a merged LoRA costs no extra latency.

import numpy as np

rng = np.random.default_rng(0)
d_in, d_out, r, alpha = 768, 768, 8, 16

W = rng.normal(size=(d_out, d_in)) / np.sqrt(d_in)   # frozen, never updated
A = rng.normal(size=(r, d_in)) * 0.01                # small random
B = np.zeros((d_out, r))                             # exactly zero
scale = alpha / r


def forward(x):                                      # x is (batch, d_in)
    base = x @ W.T                                   # (batch, d_out), no gradient
    delta = (x @ A.T) @ B.T * scale                  # (batch, r) then (batch, d_out)
    return base + delta

The order of the two matmuls is not cosmetic. Going through A then B costs batch times r times (d_in + d_out) operations; forming B @ A first and then multiplying costs batch times d_in times d_out, which is the full-rank cost and throws away the saving. On a small example the difference is invisible; on a real layer it is the difference between a training step that fits and one that does not.

Why B starts at zero#

At initialisation the adapter must contribute nothing, so that training starts from exactly the base model rather than from a randomly perturbed one. Zeroing B achieves that. A is random because if both were zero the product would have no gradient direction to move in at all.

x = rng.normal(size=(4, d_in))
assert np.allclose(forward(x), x @ W.T)              # the adapter is inert at step 0

# g stands in for the gradient of the loss with respect to this layer's output
g = rng.normal(size=(4, d_out))
h = x @ A.T                                          # (batch, r)

grad_B = g.T @ h * scale                             # (d_out, r), nonzero
grad_A = (g @ B).T @ x * scale                       # (r, d_in), exactly zero: B is zero

assert np.abs(grad_B).max() > 0
assert np.abs(grad_A).max() == 0

What r does#

r caps the rank of the update, and that is the entire story. B @ A is a product of a (d_out, r) and an (r, d_in) matrix, so whatever the training does, the resulting delta can have rank at most r. If the change the task requires genuinely needs rank 20, an adapter at r=8 cannot represent it, and no amount of training will find it.

This is demonstrable without training anything. Take a target update that is exactly rank 3, and ask for the best approximation available at each r — which is the truncated singular value decomposition, by Eckart-Young.

target = rng.normal(size=(d_out, 3)) @ rng.normal(size=(3, d_in))   # rank 3 by construction
target /= np.linalg.norm(target)

U, S, Vt = np.linalg.svd(target, full_matrices=False)
for rr in [1, 2, 3, 4, 8]:
    best = (U[:, :rr] * S[:rr]) @ Vt[:rr]            # best rank-rr approximation
    print(rr, round(float(np.linalg.norm(target - best)), 4))

# 1 0.7984
# 2 0.5534
# 3 0.0
# 4 0.0
# 8 0.0

Real updates are not exactly low rank, so the curve is a gentle decline rather than a cliff, which is why r=8 or 16 is usually enough for style, format and domain vocabulary, and why raising r past 64 rarely helps. The useful instinct is that if raising r improves your evaluation substantially, the change you are asking for is large, and full fine-tuning is the honest comparison.

What alpha does#

alpha is a fixed multiplier on the adapter's output, applied as alpha over r. It is not learned and it does not change what the adapter can represent; it changes how large the update is for a given set of trained weights, which in practice behaves like a learning rate for the adapter branch.

Dividing by r is the part people misremember. Left unscaled, the product B @ A grows as r grows — more rank-one terms summed — so an r=32 adapter would arrive with a systematically larger update than an r=8 one and would need a different learning rate. The division removes most of that coupling, so r can be changed without retuning.

print(f"{'r':>3} {'||BA||':>8} {'(16/r)||BA||':>14}")
for rr in [4, 8, 16, 32]:
    Ar = rng.normal(size=(rr, d_in)) * 0.01
    Br = rng.normal(size=(d_out, rr)) * 0.05         # a trained B, roughly
    raw = np.linalg.norm(Br @ Ar)
    print(f"{rr:>3} {raw:>8.3f} {16 / rr * raw:>14.3f}")

#   r   ||BA||   (16/r)||BA||
#   4    0.774          3.097
#   8    1.084          2.169
#  16    1.534          1.534
#  32    2.165          1.083

What it actually saves#

The parameter count is arithmetic worth being able to do at a whiteboard: a full matrix is d_in times d_out, and the adapter is r times the sum of the two.

d_in = d_out = 768
r = 8

base = d_in * d_out                # 589,824
lora = r * (d_in + d_out)          # 12,288
print(round(100 * lora / base, 2)) # 2.08

The parameter saving is not the saving that matters most. Training memory is dominated by the optimiser: Adam holds two momentum terms per trainable parameter, and gradients need another copy, so a trainable parameter costs several times what a frozen one does. Frozen weights need only to be stored — often quantized — and read.

7B model, full fine-tune, fp16 weights with Adam in fp32
  weights          14 GB
  gradients        14 GB
  Adam states      56 GB          two fp32 moments per parameter
  total           ~84 GB before activations

same model, LoRA at r=16 on the attention projections (~0.2% of parameters)
  weights          14 GB          frozen, and quantizable to ~4 GB at 4-bit
  gradients      ~0.03 GB
  Adam states     ~0.1 GB
  total           ~14 GB, or ~5 GB with a quantized base

Merging#

Because the adapter is a parallel branch that is added, its contribution can be folded into the base weight once training is finished. The merged model is an ordinary model of the original shape, with no extra matrices and no extra latency at inference.

W_merged = W + scale * (B @ A)                     # (d_out, d_in), the original shape
assert np.allclose(x @ W_merged.T, forward(x))

Whether to merge is a serving decision rather than a training one. Merging gives one model per adapter and the lowest possible latency. Keeping adapters separate lets one frozen base serve many of them at once, switched per request, at the cost of the small extra matmul — which is how a provider offers thousands of fine-tunes without thousands of model copies. There is one case where merging is not clean: if the base was quantized, merging a full-precision update into it requires re-quantizing, and the rounding that follows can undo part of what was trained.