Asked about Running a fine-tune properly

You have five fine-tunes of one base model to serve. Merged or separate adapters?

An answer that works

Separate. Merging gives five full copies of the base weights, which for an 8B model at fp16 is about fifteen gigabytes each, against one shared base plus five adapters of a few tens of megabytes. Unmerged costs one extra low-rank matmul in the adapted layers, roughly one per cent of their work at r=16, and lets requests using different adapters share a batch if the server supports it. Merge only when you serve one adapter at high steady volume and want the last of the latency — and not at all if the base is quantized, because merging then requires re-quantizing and the rounding can undo part of what was trained.

Reading an answer is not being able to give one. The section that teaches this is Running a fine-tune properly, 6 hours of Fine-tuning.

Learn this properly

Also asked about Fine-tuning

When would you fine-tune rather than use RAG?What is LoRA?What is the cost of fine-tuning beyond the training run?What is distillation?How much data do you need to fine-tune?Where do the examples come from?What is catastrophic forgetting?How do you know the fine-tune actually beat the prompt?
All 13 questions on Fine-tuning