Asked about A LoRA adapter from scratch
Which layers would you put adapters on?
An answer that works
The attention projections first — query and value are the usual default, and they were what the original work found sufficient. Adding key and output projections, and then the feed-forward matrices, covers more of the model at proportionally more cost; the feed-forward layers hold most of the parameters, so putting adapters there is the largest step up. Start narrow and widen only if evaluation says the capacity was the limit, because a wider adapter set is the same kind of capacity increase as a larger r.
Reading an answer is not being able to give one. The section that teaches this is A LoRA adapter from scratch, 4 hours of Fine-tuning.
Learn this properlyAlso asked about Fine-tuning
When would you fine-tune rather than use RAG?What is LoRA?What is the cost of fine-tuning beyond the training run?What is distillation?How much data do you need to fine-tune?Where do the examples come from?What is catastrophic forgetting?How do you know the fine-tune actually beat the prompt?
All 13 questions on Fine-tuning