Prompt, retrieve or fine-tune

Fine-tuning is the first thing people reach for when a model is not doing what they want, and it is usually the wrong first move. This section is about choosing between three different tools: changing the prompt, supplying facts at request time, and training the model itself. Getting the order wrong is expensive, because you end up building a dataset and a training pipeline for a problem a better prompt would have solved, and you own both of them afterwards.

Start from the symptom, not from the technique#

The decision is easier to make backwards. Do not ask which technique to use; ask what the model did wrong, and what a correct answer would have needed that the model did not have. Nearly every failure falls into one of a handful of shapes, and each shape has one tool that addresses it and two that do not. This table is the first thing an applied interviewer probes, and it is worth memorising, because the penalty for reading a row wrong is measured in weeks.

the complaint                              what was missing        reach for
"it invented a customer who does not       a fact, at request      retrieval
 exist"                                     time
"it does not know our Q3 pricing"          a fact, at request      retrieval
                                            time
"it answers in prose when we need JSON"    a constraint on the     the prompt, then
                                            output shape            constrained decoding
"it calls it a policy, we call it a        a vocabulary it has     fine-tuning
 cover note"                                never seen
"it is right, but 900 ms too slow and      nothing - the same      fine-tune or distil
 four times the budget"                     behaviour, cheaper      a smaller model
"it follows six instructions and drops     attention on a long     restructure the
 the seventh"                               instruction             prompt, then measure

Two rows repay a second look. The JSON row is the one people fine-tune for most often and need to least often: asking for the format usually works, giving two examples nearly always works, and constrained decoding, where the serving stack will not let the model emit a token that breaks the schema, works by construction. Fine-tuning for format earns its place in one case, which is high enough volume that the instruction tokens themselves are a line item you can see.

The latency row is the opposite: it is where fine-tuning most reliably wins, and it is the row people reach for last. The goal there is not better behaviour but the same behaviour from a model an order of magnitude cheaper, and that is exactly the thing a few thousand examples of the large model's own output can buy. If you are going to fine-tune once in a year, this is probably the reason.

Three different problems#

Prompting      changes behaviour now, costs nothing to try
Retrieval      supplies facts the model does not have
Fine-tuning    changes form: format, style, domain vocabulary

Fine-tuning does not reliably install new knowledge. It shifts how a model responds rather than what it knows, and a fact taught this way is not dependable and cannot be updated without training again. If the model does not know something, that is a retrieval problem.

It is worth being precise about where in the system each lever sits, because that is what decides how fast you can undo it. Prompting changes the input, and the same input every time. Retrieval changes the input too, but chooses its contents per request, which is why a fact can be corrected by editing a document rather than by touching the model. Fine-tuning changes the function that maps input to output. Nothing else in the pipeline moves, and nothing in the request can override what it learned.

That difference shows up at three in the morning. A prompt change ships in a minute and reverts in a minute. A retrieval change ships in an hour, and a bad document can be deleted. A fine-tune ships in a day at best, and reverts by pointing the serving stack back at the previous adapter, which means you have to have kept it, versioned it, and know which one was live. The ability to roll back is not a property of the technique; it is a property of the pipeline you built around it, and it is the part people skip.

The order#

The order below is not a ranking by power. It is a ranking by how cheaply you can discover you were wrong. Each step is a test of a hypothesis about what is missing, and the first three steps can each be abandoned in an afternoon. The fourth cannot, which is the only reason it is fourth.

1. 1.Improve the prompt. Free, immediate, and frequently sufficient.
2. 2.Add examples. Still free to change, costs tokens per call.
3. 3.Add retrieval, if the gap is knowledge.
4. 4.Fine-tune, if you still need consistent form, a narrower model, or lower latency.

Skipping steps is a bet, so it is worth writing down what each bet costs when it loses. These are rough figures rather than measurements, and the ratio between the rows is the part that matters, not the absolute numbers.

you tried it first   and you were wrong because   the loss was
prompting            it genuinely was not enough  an afternoon, and you now
                                                  have the baseline you will
                                                  need anyway
retrieval            the gap was form, not fact   a week of index plumbing,
                                                  and the retrieval is still
                                                  useful for something else
fine-tuning          a prompt would have done it  two to six weeks, plus a
                                                  dataset to maintain, a
                                                  training job to run again on
                                                  every base model change, an
                                                  eval set to keep honest, and
                                                  a rollback you must build

The last row is the one people underestimate, because the training run is the cheap part. Renting a GPU for a few hours to train an adapter costs less than a working day of your time. The expensive part is that the dataset is now a production artefact: it has to be versioned, reviewed when the task changes, and rebuilt when the base model does, and somebody has to own it after you move teams.

What it is genuinely good at#

• Consistent output format, without spending tokens on instructions every call.
• A domain style or vocabulary that is hard to specify in words.
• Making a small model good enough at one narrow task, which cuts cost and latency substantially.
• Behaviour you can demonstrate with examples but not describe.

Form generalises, facts do not: a demonstration#

The claim that fine-tuning teaches form and not facts is usually asserted and rarely shown, which is why people keep trying it anyway. It can be shown on a model small enough to train on a laptop in twenty seconds, and once you have watched it happen the instinct sticks.

The setup is a hundred entities, each with a two-letter code, and a fixed answer template. Two models are trained on the same eighty entities and tested on the twenty they never saw. The first only ever meets the code in the answer, so the only place it can store it is in the weights. The second is given the code in the prompt, and the code varies from example to example so that memorising it is useless: the only thing it can learn is to copy. That second model is retrieval, stripped down to its mechanism.

import random, torch, torch.nn as nn

letters = "abcdefghijklmnopqrstuvwxyz"
rng = random.Random(0)
CODE = {i: rng.choice(letters) + rng.choice(letters) for i in range(100)}
TRAIN, HELD = list(range(80)), list(range(80, 100))

ALPHABET = sorted(set("?>=; ansfe0123456789" + letters))
STOI = {c: i for i, c in enumerate(ALPHABET)}
ITOS = {i: c for c, i in STOI.items()}

# A: the fact only ever appears in the answer, so it must live in the weights.
weights_rows = ["?e%02d>ans=%s;" % (i, CODE[i]) for i in TRAIN] * 8
# B: the fact is in the prompt and differs every time, so it cannot be memorised.
r2 = random.Random(7)
context_rows = ["?e%02d f=%s>ans=%s;" % (i, c, c)
                for i in TRAIN
                for c in [r2.choice(letters) + r2.choice(letters) for _ in range(8)]]
print(weights_rows[0], "  ", context_rows[0])
print("rows:", len(weights_rows), len(context_rows))

# ?e00>ans=my;    ?e00 f=ke>ans=ke;
# rows: 640 640

class Tiny(nn.Module):
    def __init__(self, L, V):
        super().__init__()
        self.emb, self.pos = nn.Embedding(V, 64), nn.Embedding(L, 64)
        layer = nn.TransformerEncoderLayer(64, 4, 128, dropout=0.0,
                                           batch_first=True, norm_first=True)
        self.tr = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
        self.out = nn.Linear(64, V)

    def forward(self, x):
        n = x.shape[1]
        h = self.emb(x) + self.pos(torch.arange(n))
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1)
        return self.out(self.tr(h, mask=mask))

def fit(rows, epochs=400):
    V, L = len(STOI), len(rows[0])
    X = torch.tensor([[STOI[c] for c in r] for r in rows])
    torch.manual_seed(0)
    m = Tiny(L, V)
    opt = torch.optim.AdamW(m.parameters(), lr=3e-3)
    for _ in range(epochs):
        logits = m(X[:, :-1])
        loss = nn.functional.cross_entropy(logits.reshape(-1, V), X[:, 1:].reshape(-1))
        opt.zero_grad(); loss.backward(); opt.step()
    return m, L, loss.item()

def answer(m, L, prompt):
    ids = [STOI[c] for c in prompt]
    with torch.no_grad():
        while len(ids) < L:
            ids.append(m(torch.tensor([ids]))[0, -1].argmax().item())
    return "".join(ITOS[i] for i in ids)[len(prompt):]

def score(m, L, ids, with_fact):
    form = fact = 0
    for i in ids:
        p = "?e%02d f=%s>" % (i, CODE[i]) if with_fact else "?e%02d>" % i
        t = answer(m, L, p)
        if len(t) == 7 and t[:4] == "ans=" and t[6] == ";":
            form += 1
            fact += t[4:6] == CODE[i]
    return form, fact

weights_model, Lw, loss_w = fit(weights_rows)
context_model, Lc, loss_c = fit(context_rows)
print("final train loss  weights %.3f  context %.3f" % (loss_w, loss_c))

for name, m, L, wf in (("facts in weights ", weights_model, Lw, False),
                       ("fact in context  ", context_model, Lc, True)):
    f1, k1 = score(m, L, TRAIN, wf)
    f2, k2 = score(m, L, HELD, wf)
    print("%s trained 80: format %2d/80 fact %2d/80 | unseen 20: format %2d/20 fact %2d/20"
          % (name, f1, k1, f2, k2))

print("unseen entity e93, true code", CODE[93])
print("  facts in weights ->", answer(weights_model, Lw, "?e93>"))
print("  fact in context  ->", answer(context_model, Lc, "?e93 f=%s>" % CODE[93]))

# final train loss  weights 0.399  context 0.410
# facts in weights  trained 80: format 80/80 fact 80/80 | unseen 20: format 20/20 fact  0/20
# fact in context   trained 80: format 80/80 fact 77/80 | unseen 20: format 20/20 fact 20/20
# unseen entity e93, true code mc
#   facts in weights -> ans=qi;
#   fact in context  -> ans=mc;

Read the unseen column and nothing else. The fine-tuned model produced a perfectly formatted answer for all twenty entities it had never met, and the content of every one of them was wrong. It did not hedge, it did not decline, and its output is indistinguishable in shape from the eighty it had memorised. That is the failure mode in one line: a fine-tune generalises its form to inputs it has never seen, and invents the part that was supposed to be a fact.

The second model got all twenty right, having seen none of them, because the answer was in its input. It never learned a single code; it learned where to look. That is the whole argument for retrieval over fine-tuning when the gap is knowledge, and it also explains the update story: changing a fact means changing a document, not retraining.

Be honest about what this does and does not show. It is a hundred two-letter facts and a twelve-character template, not an 8B model on your documents, and a large model with enough repetition can memorise a great deal. What transfers is the shape of the failure rather than the exact percentages: form is learned from the aggregate of every example and applies to everything, while a fact is learned only from the examples that contain it and applies to nothing else. Nothing about scale changes that asymmetry; it only changes how many repetitions a fact needs before it sticks, and how confidently the model is wrong about the ones that did not.

LoRA#

Full fine-tuning updates every parameter, which needs serious hardware. LoRA freezes the original weights and trains a small number of added ones, reaching most of the benefit at a fraction of the cost. It is why fine-tuning is available to teams without a cluster.

Full fine-tune    every parameter     expensive, a new full model
LoRA              small added set     cheap, a small adapter file

Adapters can be swapped per task against one base model.

A fraction of the cost is not a number, and the number is arithmetic you can do from a model's configuration file in a couple of minutes. It is worth doing once for a model you actually use, because it settles two questions that otherwise get answered by folklore: whether the rank is the expensive choice, and whether you can train this on the hardware you have. The section on building a LoRA layer from scratch derives where these shapes come from; here they are only counted.

# Llama-3-8B shapes. Every number below is arithmetic on these five integers.
L, D, D_KV, FF, VOCAB = 32, 4096, 1024, 14336, 128256

LINEARS = {"q": (D, D), "k": (D_KV, D), "v": (D_KV, D), "o": (D, D),
           "gate": (FF, D), "up": (FF, D), "down": (D, FF)}
TARGETS = {"q,v": ["q", "v"],
           "attention": ["q", "k", "v", "o"],
           "every linear": list(LINEARS)}

base = VOCAB * D * 2 + L * sum(o * i for o, i in LINEARS.values())
print("base parameters: %.2f B" % (base / 1e9))

GB = 1024 ** 3
print()
print("target        r    trainable   % of base  adapter fp16  Adam+grad")
for name, keys in TARGETS.items():
    for r in (8, 16, 64):
        p = L * sum(r * (o + i) for o, i in (LINEARS[k] for k in keys))
        print("%-12s %3d %12s  %8.3f%%  %9.1f MB  %8.2f GB"
              % (name, r, format(p, ","), 100 * p / base, p * 2 / 1024 ** 2, p * 12 / GB))

# base parameters: 8.03 B
#
# target        r    trainable   % of base  adapter fp16  Adam+grad
# q,v            8    3,407,872     0.042%        6.5 MB      0.04 GB
# q,v           16    6,815,744     0.085%       13.0 MB      0.08 GB
# q,v           64   27,262,976     0.340%       52.0 MB      0.30 GB
# attention      8    6,815,744     0.085%       13.0 MB      0.08 GB
# attention     16   13,631,488     0.170%       26.0 MB      0.15 GB
# attention     64   54,525,952     0.679%      104.0 MB      0.61 GB
# every linear   8   20,971,520     0.261%       40.0 MB      0.23 GB
# every linear  16   41,943,040     0.522%       80.0 MB      0.47 GB
# every linear  64  167,772,160     2.089%      320.0 MB      1.88 GB

Two things fall out of that table. The rank is nearly free: going from r=8 to r=64 on the attention projections is an eightfold increase in trainable parameters and still leaves you under 0.7 per cent of the model and a 104 MB file. The choice that costs is which matrices you attach to, because the feed-forward layers hold most of the parameters, so widening from the attention projections to every linear multiplies the adapter by three at the same rank.

The memory that decides whether you can train at all is dominated by the frozen base, not by the adapter, and that is the argument for quantizing it.

print("full fine-tune, fp16 weights and grads, fp32 Adam moments:")
print("  weights %5.1f GB   grads %5.1f GB   Adam %5.1f GB   total %5.1f GB"
      % (base * 2 / GB, base * 2 / GB, base * 8 / GB, base * 12 / GB))
p16 = L * sum(16 * (o + i) for o, i in (LINEARS[k] for k in TARGETS["attention"]))
print("LoRA r=16 on attention, frozen fp16 base:")
print("  weights %5.1f GB   grads %5.2f GB   Adam %5.2f GB   total %5.1f GB"
      % (base * 2 / GB, p16 * 2 / GB, p16 * 8 / GB, (base * 2 + p16 * 12) / GB))
print("  same, frozen base quantized to 4 bits:  total %5.1f GB"
      % ((base * 0.5 + p16 * 12) / GB))

print()
print("extra matmul per token when the adapter is not merged:")
for r in (8, 16, 64):
    extra = sum(r * (o + i) for o, i in (LINEARS[k] for k in TARGETS["attention"]))
    full = sum(o * i for o, i in (LINEARS[k] for k in TARGETS["attention"]))
    print("  r=%2d  %.2f%% of the targeted layers' work" % (r, 100 * extra / full))

# full fine-tune, fp16 weights and grads, fp32 Adam moments:
#   weights  15.0 GB   grads  15.0 GB   Adam  59.8 GB   total  89.7 GB
# LoRA r=16 on attention, frozen fp16 base:
#   weights  15.0 GB   grads  0.03 GB   Adam  0.10 GB   total  15.1 GB
#   same, frozen base quantized to 4 bits:  total   3.9 GB
#
# extra matmul per token when the adapter is not merged:
#   r= 8  0.51% of the targeted layers' work
#   r=16  1.02% of the targeted layers' work
#   r=64  4.06% of the targeted layers' work

Ninety gigabytes is several data-centre cards. Fifteen is one. Under four is a consumer card with room to spare, and that last row is what QLoRA named: keep the frozen base at four bits and the trainable adapter at full precision, so the only thing you lose precision on is the part that is not learning. The last block answers the other question people ask, which is what an unmerged adapter costs at inference: about one per cent of the work in the layers it touches at r=16, which is inside the noise of anything else in a serving stack.

Vocabulary without the mathematics#

• Pretraining: learning language from an enormous corpus. Almost nobody does this.
• SFT, supervised fine-tuning: training on input and output pairs you provide.
• RLHF: shaping behaviour using human preferences between responses.
• DPO: a simpler method reaching similar ends without a separate reward model.
• Distillation: training a small model to imitate a large one. Often the best cost lever in an LLM product.

Three of those five are one topic away. Supervised fine-tuning, DPO and RLHF are taught properly under alignment, with the objectives written out and runnable, so if an interview is going in that direction read that rather than a longer list here. The distinction to hold on to is what each learns from. Supervised fine-tuning learns from examples of the answer you wanted. Preference methods learn from judgements about which of two answers was better, which is the only option when you can tell good from bad but cannot write down good yourself.

Distillation is the one on that list an applied engineer is most likely to do, and it is barely mentioned in interviews. It is supervised fine-tuning where the labels come from a larger model rather than from people, which makes the dataset cheap: run the expensive model over the traffic you already have, keep the outputs that pass your evaluation, and train the small model on those. The cases where this works are narrow tasks with a checkable output, and the honest limit is that the student inherits the teacher's mistakes as well as its behaviour, so the filtering step is not optional.