RAG or fine-tuning
The model does not know something it needs to know. Do you retrieve the knowledge at request time, or train it into the weights?
The short answer
RAG for what the model does not know, fine-tuning for what it will not do. Most systems that need both start with RAG, because it is reversible and a fine-tune is not.
Retrieval (RAG)
Good at
- Facts that change: update the index, not the model.
- Attribution — you can show the passage the answer came from.
- Corpora too large to train on, and corpora you do not own.
- Per-user and per-tenant knowledge, where one model must serve everyone.
- Getting to a working system in days rather than weeks.
Costs you
- Every request pays for retrieval latency and the tokens of the retrieved passages.
- Answer quality is capped by retrieval quality, and retrieval fails quietly.
- A whole second system to build and monitor: parsing, chunking, embedding, index freshness.
- Does not teach format, tone or a task the model cannot already do.
- Long contexts of retrieved text degrade — relevant passages in the middle get missed.
Pick this when: The gap is knowledge the model has never seen, it changes, or the user needs to know where the answer came from.
Fine-tuning
Good at
- Format and style the model will not hold reliably from instructions.
- A narrow repeated task, where a small tuned model beats a large prompted one.
- Cutting per-request cost: the behaviour moves from prompt tokens into weights.
- Latency, because the instructions and examples are no longer sent every call.
- Behaviour that is hard to describe but easy to demonstrate with a few hundred examples.
Costs you
- You need training data — usually several hundred to a few thousand good examples.
- Every knowledge update means another training run and another evaluation.
- No attribution: the model cannot cite what it absorbed.
- It happily learns the wrong thing, and only an eval set will tell you.
- Serving a custom model is an operational commitment that a shared API is not.
Pick this when: The gap is behaviour rather than facts, you have examples of the behaviour, and the task is stable enough to be worth a training run.
Where the field disagrees
How much fine-tuning can teach new facts. One camp holds that facts belong only in context and tuning on them mostly teaches confident hallucination; another reports real gains from continued pretraining on a domain corpus before instruction tuning. Both have results. The difference is usually corpus size: a few thousand documents is a retrieval problem, a few billion tokens is a training one.