Asked about A LoRA adapter from scratch

Implement a LoRA layer, and explain what r and alpha do.

An answer that works

Keep the base weight frozen and add a parallel branch: x through A, which is (r, d_in), then through B, which is (d_out, r), scaled by alpha over r, added to the base output. A is small random, B is zero, so the adapter contributes nothing at initialisation and training starts from the base model exactly. r caps the rank of the update the adapter can represent, and so how large a behavioural change it can express. alpha is a fixed multiplier on that update, divided by r so that changing r does not require retuning the learning rate; it behaves like a learning rate for the adapter branch and changes nothing about what can be represented.

Then they push further

Why is B initialised to zero rather than randomly?

So the adapted model is identical to the base model at step zero. If both factors were random the model would start from a perturbed version of itself and the first part of training would be spent undoing that. Both cannot be zero, because the product would then have no gradient direction; zeroing one and randomising the other gives an inert start that still trains. On the first step only B receives gradient, since the gradient to A passes through B.

What is the saving, in numbers?

For a 768 by 768 projection at r=8, 12,288 trained parameters against 589,824, about 2%. The memory saving is larger than that, because Adam keeps two momentum states per trainable parameter: a 7B full fine-tune needs roughly 84 GB for weights, gradients and optimiser state, where LoRA needs around 14 GB, or about 5 GB with a 4-bit frozen base.

When would raising r not help?

When the update the task needs is already within rank r, which is the common case for format, tone and domain vocabulary. Rank caps what can be represented, so above the rank actually required, extra capacity buys nothing and costs memory. If raising r keeps improving your evaluation, that is evidence the change you want is large, and you should compare against full fine-tuning rather than keep climbing.

What gets marked up

Saying r is rank and alpha is scaling is passable. Strong answers know that alpha over r is what reaches the model and why the division is there, that B starts at zero for a reason they can state, and that the memory saving comes mostly from optimiser state rather than from the parameter count.

Reading an answer is not being able to give one. The section that teaches this is A LoRA adapter from scratch, 4 hours of Fine-tuning.

Learn this properly

Also asked about Fine-tuning

When would you fine-tune rather than use RAG?What is LoRA?What is the cost of fine-tuning beyond the training run?What is distillation?How much data do you need to fine-tune?Where do the examples come from?What is catastrophic forgetting?How do you know the fine-tune actually beat the prompt?
All 13 questions on Fine-tuning