Asked about Prompt, retrieve or fine-tune
What is distillation?
An answer that works
Training a smaller model to imitate a larger one's outputs. For LLM products it is frequently the largest cost reduction available, because a distilled small model can match the large one on one narrow task at a fraction of the price.
Reading an answer is not being able to give one. The section that teaches this is Prompt, retrieve or fine-tune, 8 hours of Fine-tuning.
Learn this properlyAlso asked about Fine-tuning
When would you fine-tune rather than use RAG?What is LoRA?What is the cost of fine-tuning beyond the training run?How much data do you need to fine-tune?Where do the examples come from?What is catastrophic forgetting?How do you know the fine-tune actually beat the prompt?You have five fine-tunes of one base model to serve. Merged or separate adapters?
All 13 questions on Fine-tuning