LoRA or a full fine-tune
You have decided to fine-tune. Do you update every weight, or train small adapter matrices and leave the base model frozen?
The short answer
Start with LoRA. If it plateaus below what the eval needs, raise the rank before you reach for a full tune; most reports of LoRA underperforming are rank 8 doing a rank 64 job.
LoRA and other adapters
Good at
- Fitting on hardware you have — with 4-bit quantized base weights, a 7B tune runs on one 24GB card.
- Cheap experiments: an adapter is tens of megabytes, so you can keep twenty of them.
- Serving many variants, since adapters can share one base model in memory.
- Reversibility — the base model is untouched, so a bad run costs you nothing permanent.
- Resisting catastrophic forgetting, because most of the model cannot move.
Costs you
- A limited budget of change: rank r caps how much behaviour you can add.
- Two more hyperparameters to get wrong — rank and alpha — plus which modules to adapt.
- Usually slightly behind a full tune on the hardest tasks, and reliably behind for large domain shifts.
- Adapter serving adds a small inference overhead unless you merge, and merging loses the swapping.
- Quantized base weights (QLoRA) add their own small quality loss on top.
Pick this when: Almost always first. Certainly when you are adapting behaviour or style, running on one or two GPUs, or expect to train more than one variant.
Full fine-tune
Good at
- Large shifts: a new language, a new modality, a domain the base model barely saw.
- Squeezing out the last few points when a LoRA run has plateaued and the eval still matters.
- Continued pretraining on a big unlabelled corpus, where every weight should move.
- Producing one self-contained model with nothing to compose at serving time.
Costs you
- Memory: weights, gradients and optimiser state together run roughly 16 bytes per parameter in mixed precision with Adam.
- Multi-GPU sharding for anything above a few billion parameters, and the engineering that comes with it.
- A full checkpoint per experiment, so storage and comparison get expensive.
- Catastrophic forgetting is a live risk — general ability can fall while your task metric rises.
- Slow feedback, which quietly reduces how many ideas you try.
Pick this when: LoRA has plateaued below what you need, or you are adapting to a domain far enough from the base distribution that a low-rank update cannot express it.