Running a fine-tune properly

You have decided a fine-tune is the right answer. Almost all the work that remains is the dataset, because a model learns exactly what the examples show, including the inconsistencies nobody noticed. This section covers preparing that data, how much of it is enough for different goals, the general abilities a narrow fine-tune can quietly damage, and the comparison that decides whether the whole thing was worth doing: against the best prompt you can write, not a weak one.

The data is the work#

A fine-tune learns exactly what the examples show, including their mistakes and inconsistencies. A few hundred clean, consistent, representative examples beat thousands of noisy ones.

• Consistent format across every example.
• Cover the real distribution of inputs, including hard and edge cases.
• Remove duplicates and contradictory examples.
• Hold out a test set before you start, and never train on it.

Where the examples come from#

Nobody sits down and writes two thousand examples, and the teams that try stop at four hundred and ship something worse than their prompt. Real datasets are assembled from sources that already exist, and each source has a characteristic defect you should go looking for rather than wait to discover in the evaluation.

source                        cost      the defect it brings

production logs, corrected    medium    only covers what users already
by whoever owns the task                do, so the tail is missing and
                                        the model will never see it

a large model's output,       low       the student inherits the teacher's
filtered by your own eval               errors; filtering is not optional

hand-written by an expert     high      too clean, too consistent, and
                                        unconsciously written to the
                                        format rather than to the task

existing structured data      very low  the mapping is not the one users
reshaped into examples                  ask for; you are teaching a
                                        translation nobody requested

synthetic variation of real   low       duplicates in disguise; a hundred
examples                                paraphrases of forty cases is
                                        forty examples with extra noise

The first row is the one that quietly decides the project. A dataset built from logs matches the traffic the model already handles, which is the traffic where the prompt was already good enough. The examples you actually need are the ones the current system got wrong, and those are exactly the ones that do not appear in a log of successful calls. Sample the failures deliberately, and be prepared for the fact that they are harder to label because they are the hard cases.

Hold out the test set before you build anything, and hold it out by something meaningful rather than at random. If your examples come from a hundred customers, split by customer, or a near-duplicate ticket from the same customer will sit on both sides and your evaluation will report a number you cannot ship against.

The quality bar, as a script you can run#

Consistent format and no contradictions are easy to agree with and hard to eyeball at two thousand rows. The checks below are twenty lines and catch the defects that cost the most, and they should run before every training job rather than once at the start. The dataset here is twelve rows with one of each defect planted, which is roughly the hit rate of a real first draft.

import json, collections

# Twelve rows of a support-ticket classifier set, with the defects a real one has.
DATA = [
    {"in": "card declined at checkout",        "out": '{"label": "billing"}'},
    {"in": "card declined at checkout",        "out": '{"label": "payments"}'},
    {"in": "cannot log in after password reset", "out": '{"label": "auth"}'},
    {"in": "cannot log in after password reset", "out": '{"label": "auth"}'},
    {"in": "invoice shows the wrong VAT rate", "out": '{"label":"billing"}'},
    {"in": "app crashes on the settings page", "out": "bug"},
    {"in": "how do I export my data",          "out": '{"label": "howto"}'},
    {"in": "refund not received after 10 days", "out": '{"label": "billing"}'},
    {"in": "SSO metadata upload fails",        "out": '{"label": "auth"}'},
    {"in": "where is the dark mode toggle",    "out": '{"label": "howto"}'},
    {"in": "i was charged twice this month",   "out": '{"label": "billing"}'},
    {"in": ("the integration has been failing since tuesday and the logs show "
            "a 502 from your side every fifteen minutes, we have tried rotating "
            "the key and it made no difference, please advise urgently"),
     "out": '{"label": "bug"}'},
]
TEST = [
    {"in": "how do I export my data", "out": '{"label": "howto"}'},
    {"in": "payment failed twice",    "out": '{"label": "billing"}'},
]

def lint(rows, test):
    seen = collections.defaultdict(set)
    for r in rows:
        seen[r["in"]].add(r["out"])

    dupes = sum(1 for r, c in collections.Counter(
        (r["in"], r["out"]) for r in rows).items() if c > 1)
    contradictions = [k for k, v in seen.items() if len(v) > 1]

    bad_json, labels = [], collections.Counter()
    for r in rows:
        try:
            labels[json.loads(r["out"])["label"]] += 1
        except Exception:
            bad_json.append(r["in"])

    spacing = collections.Counter(
        "spaced" if '": "' in r["out"] else "compact"
        for r in rows if r["out"].startswith("{"))

    lens = sorted(len(r["in"]) for r in rows)
    p50 = lens[len(lens) // 2]
    long = [r["in"][:30] + "..." for r in rows if len(r["in"]) > 4 * p50]

    leaked = [r["in"] for r in test if r["in"] in {x["in"] for x in rows}]

    print("rows", len(rows), " unique inputs", len(seen))
    print("exact duplicate rows        ", dupes)
    print("same input, different output", contradictions)
    print("output not parseable        ", bad_json)
    print("label counts                ", dict(labels))
    print("output spacing conventions  ", dict(spacing))
    print("input length p50            ", p50, " outliers:", long)
    print("test inputs seen in training", leaked)

lint(DATA, TEST)

# rows 12  unique inputs 10
# exact duplicate rows         1
# same input, different output ['card declined at checkout']
# output not parseable         ['app crashes on the settings page']
# label counts                 {'billing': 4, 'payments': 1, 'auth': 3, 'howto': 2, 'bug': 1}
# output spacing conventions   {'spaced': 10, 'compact': 1}
# input length p50             32  outliers: ['the integration has been faili...']
# test inputs seen in training ['how do I export my data']

Read the findings in order of how much damage they do. The leak is worst, because it does not degrade the model, it degrades your ability to know anything about the model. The contradiction is next: the same ticket labelled billing once and payments once teaches the model that the boundary is arbitrary, and it will be arbitrary at inference. The label counts expose the same problem at the taxonomy level, since payments appearing once against billing four times is almost always a synonym nobody standardised rather than a real class.

The two formatting findings look trivial and are not. A model trained on a mixture of compact and spaced JSON will produce a mixture, which means the parser you were going to delete has to stay. One unparseable row in twelve is eight per cent of your training signal teaching the model that plain text is sometimes acceptable, and that is enough to produce it occasionally in production, at the point where nothing is watching.

The length outlier is the one to decide about rather than fix. A single example four times the median length is either a case you genuinely need to handle, in which case you need twenty more like it, or it is noise, in which case it is teaching the model a length distribution that does not exist. What you must not do is leave one of them in and hope.

How much#

Format or style adherence     a few hundred examples
A narrow classification task  hundreds to low thousands
Broad new behaviour           many thousands, and reconsider

That guidance is worth believing more than a rule of thumb usually is, because the shape behind it can be measured. Below, the same tiny transformer is trained from scratch on the same task with training sets from 25 to 800 examples, and both the training accuracy and the held-out accuracy are reported. The task is to sort four digits, written as a twelve-character line, and nothing about it matters except that it is a rule the model has to infer rather than a list it can memorise.

import random, copy, torch, torch.nn as nn

ALPHABET = sorted(set("mrs:>;0123456789"))
STOI = {c: i for i, c in enumerate(ALPHABET)}
ITOS = {i: c for c, i in STOI.items()}
L = 12

def max_row(r):
    d = "".join(str(r.randrange(10)) for _ in range(4))
    return "m:%s>000%s;" % (d, max(d))

def rev_row(r):
    d = "".join(str(r.randrange(10)) for _ in range(4))
    return "r:%s>%s;" % (d, d[::-1])

def sort_row(r):
    d = "".join(str(r.randrange(10)) for _ in range(4))
    return "s:%s>%s;" % (d, "".join(sorted(d)))

MAKERS = {"max": max_row, "rev": rev_row, "sort": sort_row}

def sample(kinds, n, seed):
    r = random.Random(seed)
    return [MAKERS[kinds[i % len(kinds)]](r) for i in range(n)]

class Tiny(nn.Module):
    def __init__(self):
        super().__init__()
        V = len(STOI)
        self.emb, self.pos = nn.Embedding(V, 64), nn.Embedding(L, 64)
        layer = nn.TransformerEncoderLayer(64, 4, 256, dropout=0.0,
                                           batch_first=True, norm_first=True)
        self.tr = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
        self.out = nn.Linear(64, V)

    def forward(self, x):
        n = x.shape[1]
        h = self.emb(x) + self.pos(torch.arange(n))
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1)
        return self.out(self.tr(h, mask=mask))

def fit(model, rows, steps, lr=3e-3, bs=256, seed=0):
    X = torch.tensor([[STOI[c] for c in r] for r in rows])
    g = torch.Generator().manual_seed(seed)
    opt = torch.optim.AdamW(model.parameters(), lr=lr)
    for _ in range(steps):
        b = X[torch.randint(0, len(X), (min(bs, len(X)),), generator=g)]
        logits = model(b[:, :-1])
        loss = nn.functional.cross_entropy(logits.reshape(-1, len(STOI)),
                                           b[:, 1:].reshape(-1))
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        logits = model(X[:, :-1])
        return float(nn.functional.cross_entropy(
            logits.reshape(-1, len(STOI)), X[:, 1:].reshape(-1)))

def exact(model, rows):
    ok = 0
    with torch.no_grad():
        for r in rows:
            ids = [STOI[ch] for ch in r[:r.index(">") + 1]]
            while len(ids) < L:
                ids.append(model(torch.tensor([ids]))[0, -1].argmax().item())
            ok += "".join(ITOS[i] for i in ids) == r
    return ok / len(rows)

test = {k: sample([k], 200, 100 + i) for i, k in enumerate(MAKERS)}
print(test["max"][0], test["rev"][0], test["sort"][0])

print()
print("training set size against held-out accuracy, sort task")
for n in (25, 50, 100, 200, 400, 800):
    torch.manual_seed(0)
    m = Tiny()
    rows = sample(["sort"], n, 1)
    tl = fit(m, rows, 800)
    print("n=%4d  train loss %.4f  train exact %.2f  held-out exact %.2f"
          % (n, tl, exact(m, rows[:100]), exact(m, test["sort"])))

# m:2772>0007; r:9385>5839; s:2952>2259;
#
# training set size against held-out accuracy, sort task
# n=  25  train loss 0.3076  train exact 1.00  held-out exact 0.06
# n=  50  train loss 0.3698  train exact 1.00  held-out exact 0.19
# n= 100  train loss 0.4286  train exact 1.00  held-out exact 0.40
# n= 200  train loss 0.4954  train exact 1.00  held-out exact 0.57
# n= 400  train loss 0.5599  train exact 1.00  held-out exact 0.77
# n= 800  train loss 0.6400  train exact 1.00  held-out exact 0.94

The column to stare at is the third one. At every training set size, from 25 examples to 800, the model gets every training example exactly right. If that were the number you reported, the 25-example run and the 800-example run would look identical, and the first one has learned nothing: six per cent on held-out data against ninety-four. This is what overfitting on a small dataset looks like from the inside, and it is why a fine-tune that looks perfect on the data you built it from tells you nothing at all.

Notice also that the training loss goes up as the dataset grows, from 0.31 to 0.64, while the thing you care about goes up with it. Loss is measured against a training set that is getting harder to fit, so it is not comparable between the rows and should never be the number you optimise. The held-out exact-match is the only column that means anything, and the shape it traces is the ordinary one: fast improvement, then a knee, then a long expensive tail. The practical reading is to plot this curve for your own task at three sizes before committing to a labelling budget, because the difference between the knee being at 200 examples and at 5,000 is the difference between an afternoon and a quarter.

Forgetting#

Fine-tuning on a narrow task can degrade the model's general abilities. LoRA reduces this because the original weights are untouched, and mixing some general examples into the training data helps further. Always check that general capability survived, not only the target task.

The word catastrophic is not rhetorical, and the effect is easy to produce. Take the same architecture, train it on two tasks until it is good at both, then fine-tune it on a third task alone and re-measure the first two. The fine-tuning set has four hundred examples and runs for six hundred steps, which by the standards of a real fine-tune is a short, gentle run.

print()
print("catastrophic forgetting")
torch.manual_seed(0)
base = Tiny()
fit(base, sample(["max", "rev"], 2000, 2), 2500)
print("base                     max %.2f  rev %.2f  sort %.2f"
      % (exact(base, test["max"]), exact(base, test["rev"]), exact(base, test["sort"])))

narrow = sample(["sort"], 400, 3)
for name, rows, lr in (
        ("sort only,       lr 1e-3", narrow, 1e-3),
        ("sort only,       lr 3e-4", narrow, 3e-4),
        ("+ 20% replay,    lr 1e-3", narrow + sample(["max", "rev"], 100, 4), 1e-3),
        ("+ 50% replay,    lr 1e-3", narrow + sample(["max", "rev"], 400, 4), 1e-3)):
    m = copy.deepcopy(base)
    torch.manual_seed(0)
    fit(m, rows, 600, lr=lr)
    print("%s  max %.2f  rev %.2f  sort %.2f"
          % (name, exact(m, test["max"]), exact(m, test["rev"]), exact(m, test["sort"])))

# catastrophic forgetting
# base                     max 1.00  rev 0.99  sort 0.01
# sort only,       lr 1e-3  max 0.03  rev 0.05  sort 0.39
# sort only,       lr 3e-4  max 0.24  rev 0.14  sort 0.20
# + 20% replay,    lr 1e-3  max 0.94  rev 0.44  sort 0.28
# + 50% replay,    lr 1e-3  max 0.96  rev 0.78  sort 0.32

The base model does both old tasks essentially perfectly. Six hundred steps on a fourth of a thousand examples of something else takes one of them from 1.00 to 0.03 and the other from 0.99 to 0.05. Nothing in the fine-tuning loss curve indicates this, because the fine-tuning loss is computed on the fine-tuning data, where everything is going well. If you do not measure the old capability, you do not find out.

The three mitigation rows are the useful part, and each one costs something. Dropping the learning rate to a third preserved more of the old tasks, 0.24 and 0.14 against 0.03 and 0.05, and learned the new one only half as well, 0.20 against 0.39 — which is the trade in a single line: forgetting and learning are the same process, and you cannot slow one without slowing the other. Replay is the better lever: mixing in old examples at a fifth of the new set restored the first task almost completely, and at an equal share restored most of the second, while costing under a tenth of the new task's score.

Two things transfer from this toy to a real fine-tune. The first is that replay is the cheapest insurance available and almost nobody budgets for it: reserve a fifth to a half of your training set for examples of the general behaviour you want to keep, drawn from the same kind of instruction data the base model was tuned on. The second is that the old-capability check must be in the evaluation from the first run, not added after somebody notices in production. A short fixed set of general instructions, scored the same way every time, is enough.

LoRA genuinely helps here, and it is worth being exact about why. The frozen base weights cannot move, so the only thing that can drift is the low-rank update, and at r=8 or 16 that update has nowhere near the capacity to overwrite general behaviour. It is not immunity — the adapter is applied on every input, including the ones you did not fine-tune for — but it puts a ceiling on the damage that full fine-tuning does not have, and it gives you the one-line rollback of serving without the adapter.

The evaluation that tells you whether it worked#

Three numbers decide whether to ship a fine-tune, and only one of them is about the new task. The target-task score against the best prompt you can write. The general-capability score, to catch what the previous section measured. And the cost per request, because the usual reason to do this at all is to move work onto a smaller model, and a fine-tune that matches the big model at the same price has bought nothing.

The comparison itself is where results get overstated, and the fix is to score both systems on the same examples and keep the pairing. Suppose a 200-example held-out set comes back like the counts below: the two systems agree on 158 of them and disagree on 42.

import math, numpy as np

# One row per held-out example, scored by the same grader.
# These four counts are what a paired eval of 200 examples gives you.
both_right, both_wrong = 128, 30
prompt_only, tuned_only = 14, 28          # b and c: the cases that disagree
n = both_right + both_wrong + prompt_only + tuned_only

acc_prompt = (both_right + prompt_only) / n
acc_tuned = (both_right + tuned_only) / n
print("n = %d   best prompt %.3f   fine-tuned %.3f   difference %+.3f"
      % (n, acc_prompt, acc_tuned, acc_tuned - acc_prompt))

# McNemar, exact: under the null the disagreements split 50/50.
b, c = prompt_only, tuned_only
k, m = min(b, c), b + c
p = 2 * sum(math.comb(m, i) for i in range(k + 1)) / 2 ** m
print("disagreements %d (%d vs %d)   exact McNemar p = %.4f" % (m, b, c, min(p, 1.0)))

# The unpaired version, for comparison: throw the pairing away and bootstrap.
rng = np.random.default_rng(0)
a = np.array([1] * (both_right + prompt_only) + [0] * (both_wrong + tuned_only))
t = np.array([1] * (both_right + tuned_only) + [0] * (both_wrong + prompt_only))
diffs = [t[rng.integers(0, n, n)].mean() - a[rng.integers(0, n, n)].mean()
         for _ in range(10000)]
lo, hi = np.percentile(diffs, [2.5, 97.5])
print("unpaired bootstrap 95%% interval on the difference: %+.3f to %+.3f" % (lo, hi))

# n = 200   best prompt 0.710   fine-tuned 0.780   difference +0.070
# disagreements 42 (14 vs 28)   exact McNemar p = 0.0436
# unpaired bootstrap 95% interval on the difference: -0.015 to +0.155

Same data, two conclusions. The paired test asks only about the 42 examples where the systems differed and finds that 28 to 14 is unlikely under a coin flip, p = 0.044. The unpaired bootstrap throws away which example was which, and its interval on the difference runs from minus 1.5 points to plus 15.5 — it cannot rule out that the fine-tune is worse. The pairing is not a statistical nicety; it is most of your power, because the 158 examples both systems agreed on carry no information about the difference and the unpaired version drowns the signal in their variance.

The practical rules that follow are short. Score both systems on the identical held-out set, with the identical grader, in the same run. Report the disagreement counts, not just the two accuracies, because 78 against 71 sounds the same whether the disagreements are 28 to 14 or 70 to 63 and only one of those is a result. And read the fourteen examples the prompt got right and the fine-tune got wrong before shipping, because a fine-tune that is better on average and worse on a category you care about is a decision, not a number.

Serving it, and the adapter-swapping question#

A fine-tune that only exists as a checkpoint on the machine that trained it is not finished. The serving decision is a single fork: merge the adapter into the base weights and serve one ordinary model, or keep the adapter separate and apply it on top of a shared frozen base at request time. The section on building a LoRA layer from scratch shows why merging is exact and costs no latency; what follows is when to choose which.

                      merged                   adapter kept separate

memory for N tunes    N full copies of the     one base, plus N adapters
                      base weights             at tens of MB each

latency               identical to the base    one extra low-rank matmul
                                               per adapted layer, about
                                               1% of those layers at r=16

batching              one model per batch      requests using different
                                               adapters can share a batch
                                               if the server supports it

switching             load a new model         swap a tensor, milliseconds

rollback              redeploy                 stop passing the adapter

quantized base        needs re-quantizing      no interaction; the base
                      after the merge, which   stays exactly as it is
                      can undo the training

Merge when you serve one adapter at high, steady volume and the last milliseconds matter. Keep them separate in every other case, and particularly whenever there is more than one fine-tune, because the memory column is not a small difference: fifty merged copies of an 8B model is fifteen gigabytes each, and fifty adapters at r=16 is about twenty-six megabytes each on top of one shared base.

Three operational details decide whether this is pleasant or not. Version the adapter with the base model it was trained against, in the filename, because an adapter applied to a different base of the same shape will load without error and produce quiet nonsense. Keep the previous adapter served and reachable behind a flag, so a rollback is a configuration change rather than a training run. And pin the exact prompt template used during training alongside the adapter: a fine-tune learns the template as much as the task, and serving it with a prompt that differs by a system message or a separator token is the most common way a fine-tune that evaluated well behaves badly in production.