Asked about A LoRA adapter from scratch
Why can a LoRA adapter be merged into the base weights but an older adapter layer cannot?
An answer that works
Because LoRA is a parallel branch that is added to the output, and addition of a low-rank product into the weight matrix is exact: W plus alpha over r times B times A has the same shape as W and computes the same function. Older adapters inserted a small bottleneck layer with its own nonlinearity in sequence between the existing layers, and a nonlinearity cannot be folded into a neighbouring matrix. That is why they added inference latency and LoRA does not.
Reading an answer is not being able to give one. The section that teaches this is A LoRA adapter from scratch, 4 hours of Fine-tuning.
Learn this properlyAlso asked about Fine-tuning
When would you fine-tune rather than use RAG?What is LoRA?What is the cost of fine-tuning beyond the training run?What is distillation?How much data do you need to fine-tune?Where do the examples come from?What is catastrophic forgetting?How do you know the fine-tune actually beat the prompt?
All 13 questions on Fine-tuning