Fine-tuning

Fine-tuning is the first thing people reach for when a model is not doing what they want, and it is usually the wrong first move. This section is about choosing between three different tools: changing the prompt, supplying facts at request time, and training the model itself. Getting the order wrong is expensive, because you end up building a dataset and a training pipeline for a problem a better prompt would have solved, and you own both of them afterwards.

Start from the symptom, not from the technique#

The decision is easier to make backwards. Do not ask which technique to use; ask what the model did wrong, and what a correct answer would have needed that the model did not have. Nearly every failure falls into one of a handful of shapes, and each shape has one tool that addresses it and two that do not. This table is the first thing an applied interviewer probes, and it is worth memorising, because the penalty for reading a row wrong is measured in weeks.

the complaint                              what was missing        reach for
"it invented a customer who does not       a fact, at request      retrieval
 exist"                                     time
"it does not know our Q3 pricing"          a fact, at request      retrieval
                                            time
"it answers in prose when we need JSON"    a constraint on the     the prompt, then
                                            output shape            constrained decoding
"it calls it a policy, we call it a        a vocabulary it has     fine-tuning
 cover note"                                never seen
"it is right, but 900 ms too slow and      nothing - the same      fine-tune or distil
 four times the budget"                     behaviour, cheaper      a smaller model
"it follows six instructions and drops     attention on a long     restructure the
 the seventh"                               instruction             prompt, then measure

Two rows repay a second look. The JSON row is the one people fine-tune for most often and need to least often: asking for the format usually works, giving two examples nearly always works, and constrained decoding, where the serving stack will not let the model emit a token that breaks the schema, works by construction. Fine-tuning for format earns its place in one case, which is high enough volume that the instruction tokens themselves are a line item you can see.

The latency row is the opposite: it is where fine-tuning most reliably wins, and it is the row people reach for last. The goal there is not better behaviour but the same behaviour from a model an order of magnitude cheaper, and that is exactly the thing a few thousand examples of the large model's own output can buy. If you are going to fine-tune once in a year, this is probably the reason.

Three different problems#

Prompting      changes behaviour now, costs nothing to try
Retrieval      supplies facts the model does not have
Fine-tuning    changes form: format, style, domain vocabulary

Fine-tuning does not reliably install new knowledge. It shifts how a model responds rather than what it knows, and a fact taught this way is not dependable and cannot be updated without training again. If the model does not know something, that is a retrieval problem.

It is worth being precise about where in the system each lever sits, because that is what decides how fast you can undo it. Prompting changes the input, and the same input every time. Retrieval changes the input too, but chooses its contents per request, which is why a fact can be corrected by editing a document rather than by touching the model. Fine-tuning changes the function that maps input to output. Nothing else in the pipeline moves, and nothing in the request can override what it learned.

That difference shows up at three in the morning. A prompt change ships in a minute and reverts in a minute. A retrieval change ships in an hour, and a bad document can be deleted. A fine-tune ships in a day at best, and reverts by pointing the serving stack back at the previous adapter, which means you have to have kept it, versioned it, and know which one was live. The ability to roll back is not a property of the technique; it is a property of the pipeline you built around it, and it is the part people skip.

The order#

The order below is not a ranking by power. It is a ranking by how cheaply you can discover you were wrong. Each step is a test of a hypothesis about what is missing, and the first three steps can each be abandoned in an afternoon. The fourth cannot, which is the only reason it is fourth.

1. 1.Improve the prompt. Free, immediate, and frequently sufficient.
2. 2.Add examples. Still free to change, costs tokens per call.
3. 3.Add retrieval, if the gap is knowledge.
4. 4.Fine-tune, if you still need consistent form, a narrower model, or lower latency.

Skipping steps is a bet, so it is worth writing down what each bet costs when it loses. These are rough figures rather than measurements, and the ratio between the rows is the part that matters, not the absolute numbers.

you tried it first   and you were wrong because   the loss was
prompting            it genuinely was not enough  an afternoon, and you now
                                                  have the baseline you will
                                                  need anyway
retrieval            the gap was form, not fact   a week of index plumbing,
                                                  and the retrieval is still
                                                  useful for something else
fine-tuning          a prompt would have done it  two to six weeks, plus a
                                                  dataset to maintain, a
                                                  training job to run again on
                                                  every base model change, an
                                                  eval set to keep honest, and
                                                  a rollback you must build

The last row is the one people underestimate, because the training run is the cheap part. Renting a GPU for a few hours to train an adapter costs less than a working day of your time. The expensive part is that the dataset is now a production artefact: it has to be versioned, reviewed when the task changes, and rebuilt when the base model does, and somebody has to own it after you move teams.

What it is genuinely good at#

• Consistent output format, without spending tokens on instructions every call.
• A domain style or vocabulary that is hard to specify in words.
• Making a small model good enough at one narrow task, which cuts cost and latency substantially.
• Behaviour you can demonstrate with examples but not describe.

Form generalises, facts do not: a demonstration#

The claim that fine-tuning teaches form and not facts is usually asserted and rarely shown, which is why people keep trying it anyway. It can be shown on a model small enough to train on a laptop in twenty seconds, and once you have watched it happen the instinct sticks.

The setup is a hundred entities, each with a two-letter code, and a fixed answer template. Two models are trained on the same eighty entities and tested on the twenty they never saw. The first only ever meets the code in the answer, so the only place it can store it is in the weights. The second is given the code in the prompt, and the code varies from example to example so that memorising it is useless: the only thing it can learn is to copy. That second model is retrieval, stripped down to its mechanism.

import random, torch, torch.nn as nn

letters = "abcdefghijklmnopqrstuvwxyz"
rng = random.Random(0)
CODE = {i: rng.choice(letters) + rng.choice(letters) for i in range(100)}
TRAIN, HELD = list(range(80)), list(range(80, 100))

ALPHABET = sorted(set("?>=; ansfe0123456789" + letters))
STOI = {c: i for i, c in enumerate(ALPHABET)}
ITOS = {i: c for c, i in STOI.items()}

# A: the fact only ever appears in the answer, so it must live in the weights.
weights_rows = ["?e%02d>ans=%s;" % (i, CODE[i]) for i in TRAIN] * 8
# B: the fact is in the prompt and differs every time, so it cannot be memorised.
r2 = random.Random(7)
context_rows = ["?e%02d f=%s>ans=%s;" % (i, c, c)
                for i in TRAIN
                for c in [r2.choice(letters) + r2.choice(letters) for _ in range(8)]]
print(weights_rows[0], "  ", context_rows[0])
print("rows:", len(weights_rows), len(context_rows))

# ?e00>ans=my;    ?e00 f=ke>ans=ke;
# rows: 640 640

class Tiny(nn.Module):
    def __init__(self, L, V):
        super().__init__()
        self.emb, self.pos = nn.Embedding(V, 64), nn.Embedding(L, 64)
        layer = nn.TransformerEncoderLayer(64, 4, 128, dropout=0.0,
                                           batch_first=True, norm_first=True)
        self.tr = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
        self.out = nn.Linear(64, V)

    def forward(self, x):
        n = x.shape[1]
        h = self.emb(x) + self.pos(torch.arange(n))
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1)
        return self.out(self.tr(h, mask=mask))

def fit(rows, epochs=400):
    V, L = len(STOI), len(rows[0])
    X = torch.tensor([[STOI[c] for c in r] for r in rows])
    torch.manual_seed(0)
    m = Tiny(L, V)
    opt = torch.optim.AdamW(m.parameters(), lr=3e-3)
    for _ in range(epochs):
        logits = m(X[:, :-1])
        loss = nn.functional.cross_entropy(logits.reshape(-1, V), X[:, 1:].reshape(-1))
        opt.zero_grad(); loss.backward(); opt.step()
    return m, L, loss.item()

def answer(m, L, prompt):
    ids = [STOI[c] for c in prompt]
    with torch.no_grad():
        while len(ids) < L:
            ids.append(m(torch.tensor([ids]))[0, -1].argmax().item())
    return "".join(ITOS[i] for i in ids)[len(prompt):]

def score(m, L, ids, with_fact):
    form = fact = 0
    for i in ids:
        p = "?e%02d f=%s>" % (i, CODE[i]) if with_fact else "?e%02d>" % i
        t = answer(m, L, p)
        if len(t) == 7 and t[:4] == "ans=" and t[6] == ";":
            form += 1
            fact += t[4:6] == CODE[i]
    return form, fact

weights_model, Lw, loss_w = fit(weights_rows)
context_model, Lc, loss_c = fit(context_rows)
print("final train loss  weights %.3f  context %.3f" % (loss_w, loss_c))

for name, m, L, wf in (("facts in weights ", weights_model, Lw, False),
                       ("fact in context  ", context_model, Lc, True)):
    f1, k1 = score(m, L, TRAIN, wf)
    f2, k2 = score(m, L, HELD, wf)
    print("%s trained 80: format %2d/80 fact %2d/80 | unseen 20: format %2d/20 fact %2d/20"
          % (name, f1, k1, f2, k2))

print("unseen entity e93, true code", CODE[93])
print("  facts in weights ->", answer(weights_model, Lw, "?e93>"))
print("  fact in context  ->", answer(context_model, Lc, "?e93 f=%s>" % CODE[93]))

# final train loss  weights 0.399  context 0.410
# facts in weights  trained 80: format 80/80 fact 80/80 | unseen 20: format 20/20 fact  0/20
# fact in context   trained 80: format 80/80 fact 77/80 | unseen 20: format 20/20 fact 20/20
# unseen entity e93, true code mc
#   facts in weights -> ans=qi;
#   fact in context  -> ans=mc;

Read the unseen column and nothing else. The fine-tuned model produced a perfectly formatted answer for all twenty entities it had never met, and the content of every one of them was wrong. It did not hedge, it did not decline, and its output is indistinguishable in shape from the eighty it had memorised. That is the failure mode in one line: a fine-tune generalises its form to inputs it has never seen, and invents the part that was supposed to be a fact.

The second model got all twenty right, having seen none of them, because the answer was in its input. It never learned a single code; it learned where to look. That is the whole argument for retrieval over fine-tuning when the gap is knowledge, and it also explains the update story: changing a fact means changing a document, not retraining.

Be honest about what this does and does not show. It is a hundred two-letter facts and a twelve-character template, not an 8B model on your documents, and a large model with enough repetition can memorise a great deal. What transfers is the shape of the failure rather than the exact percentages: form is learned from the aggregate of every example and applies to everything, while a fact is learned only from the examples that contain it and applies to nothing else. Nothing about scale changes that asymmetry; it only changes how many repetitions a fact needs before it sticks, and how confidently the model is wrong about the ones that did not.

LoRA#

Full fine-tuning updates every parameter, which needs serious hardware. LoRA freezes the original weights and trains a small number of added ones, reaching most of the benefit at a fraction of the cost. It is why fine-tuning is available to teams without a cluster.

Full fine-tune    every parameter     expensive, a new full model
LoRA              small added set     cheap, a small adapter file

Adapters can be swapped per task against one base model.

A fraction of the cost is not a number, and the number is arithmetic you can do from a model's configuration file in a couple of minutes. It is worth doing once for a model you actually use, because it settles two questions that otherwise get answered by folklore: whether the rank is the expensive choice, and whether you can train this on the hardware you have. The section on building a LoRA layer from scratch derives where these shapes come from; here they are only counted.

# Llama-3-8B shapes. Every number below is arithmetic on these five integers.
L, D, D_KV, FF, VOCAB = 32, 4096, 1024, 14336, 128256

LINEARS = {"q": (D, D), "k": (D_KV, D), "v": (D_KV, D), "o": (D, D),
           "gate": (FF, D), "up": (FF, D), "down": (D, FF)}
TARGETS = {"q,v": ["q", "v"],
           "attention": ["q", "k", "v", "o"],
           "every linear": list(LINEARS)}

base = VOCAB * D * 2 + L * sum(o * i for o, i in LINEARS.values())
print("base parameters: %.2f B" % (base / 1e9))

GB = 1024 ** 3
print()
print("target        r    trainable   % of base  adapter fp16  Adam+grad")
for name, keys in TARGETS.items():
    for r in (8, 16, 64):
        p = L * sum(r * (o + i) for o, i in (LINEARS[k] for k in keys))
        print("%-12s %3d %12s  %8.3f%%  %9.1f MB  %8.2f GB"
              % (name, r, format(p, ","), 100 * p / base, p * 2 / 1024 ** 2, p * 12 / GB))

# base parameters: 8.03 B
#
# target        r    trainable   % of base  adapter fp16  Adam+grad
# q,v            8    3,407,872     0.042%        6.5 MB      0.04 GB
# q,v           16    6,815,744     0.085%       13.0 MB      0.08 GB
# q,v           64   27,262,976     0.340%       52.0 MB      0.30 GB
# attention      8    6,815,744     0.085%       13.0 MB      0.08 GB
# attention     16   13,631,488     0.170%       26.0 MB      0.15 GB
# attention     64   54,525,952     0.679%      104.0 MB      0.61 GB
# every linear   8   20,971,520     0.261%       40.0 MB      0.23 GB
# every linear  16   41,943,040     0.522%       80.0 MB      0.47 GB
# every linear  64  167,772,160     2.089%      320.0 MB      1.88 GB

Two things fall out of that table. The rank is nearly free: going from r=8 to r=64 on the attention projections is an eightfold increase in trainable parameters and still leaves you under 0.7 per cent of the model and a 104 MB file. The choice that costs is which matrices you attach to, because the feed-forward layers hold most of the parameters, so widening from the attention projections to every linear multiplies the adapter by three at the same rank.

The memory that decides whether you can train at all is dominated by the frozen base, not by the adapter, and that is the argument for quantizing it.

print("full fine-tune, fp16 weights and grads, fp32 Adam moments:")
print("  weights %5.1f GB   grads %5.1f GB   Adam %5.1f GB   total %5.1f GB"
      % (base * 2 / GB, base * 2 / GB, base * 8 / GB, base * 12 / GB))
p16 = L * sum(16 * (o + i) for o, i in (LINEARS[k] for k in TARGETS["attention"]))
print("LoRA r=16 on attention, frozen fp16 base:")
print("  weights %5.1f GB   grads %5.2f GB   Adam %5.2f GB   total %5.1f GB"
      % (base * 2 / GB, p16 * 2 / GB, p16 * 8 / GB, (base * 2 + p16 * 12) / GB))
print("  same, frozen base quantized to 4 bits:  total %5.1f GB"
      % ((base * 0.5 + p16 * 12) / GB))

print()
print("extra matmul per token when the adapter is not merged:")
for r in (8, 16, 64):
    extra = sum(r * (o + i) for o, i in (LINEARS[k] for k in TARGETS["attention"]))
    full = sum(o * i for o, i in (LINEARS[k] for k in TARGETS["attention"]))
    print("  r=%2d  %.2f%% of the targeted layers' work" % (r, 100 * extra / full))

# full fine-tune, fp16 weights and grads, fp32 Adam moments:
#   weights  15.0 GB   grads  15.0 GB   Adam  59.8 GB   total  89.7 GB
# LoRA r=16 on attention, frozen fp16 base:
#   weights  15.0 GB   grads  0.03 GB   Adam  0.10 GB   total  15.1 GB
#   same, frozen base quantized to 4 bits:  total   3.9 GB
#
# extra matmul per token when the adapter is not merged:
#   r= 8  0.51% of the targeted layers' work
#   r=16  1.02% of the targeted layers' work
#   r=64  4.06% of the targeted layers' work

Ninety gigabytes is several data-centre cards. Fifteen is one. Under four is a consumer card with room to spare, and that last row is what QLoRA named: keep the frozen base at four bits and the trainable adapter at full precision, so the only thing you lose precision on is the part that is not learning. The last block answers the other question people ask, which is what an unmerged adapter costs at inference: about one per cent of the work in the layers it touches at r=16, which is inside the noise of anything else in a serving stack.

Vocabulary without the mathematics#

• Pretraining: learning language from an enormous corpus. Almost nobody does this.
• SFT, supervised fine-tuning: training on input and output pairs you provide.
• RLHF: shaping behaviour using human preferences between responses.
• DPO: a simpler method reaching similar ends without a separate reward model.
• Distillation: training a small model to imitate a large one. Often the best cost lever in an LLM product.

Three of those five are one topic away. Supervised fine-tuning, DPO and RLHF are taught properly under alignment, with the objectives written out and runnable, so if an interview is going in that direction read that rather than a longer list here. The distinction to hold on to is what each learns from. Supervised fine-tuning learns from examples of the answer you wanted. Preference methods learn from judgements about which of two answers was better, which is the only option when you can tell good from bad but cannot write down good yourself.

Distillation is the one on that list an applied engineer is most likely to do, and it is barely mentioned in interviews. It is supervised fine-tuning where the labels come from a larger model rather than from people, which makes the dataset cheap: run the expensive model over the traffic you already have, keep the outputs that pass your evaluation, and train the small model on those. The cases where this works are narrow tasks with a checkable output, and the honest limit is that the student inherits the teacher's mistakes as well as its behaviour, so the filtering step is not optional.

You have decided a fine-tune is the right answer. Almost all the work that remains is the dataset, because a model learns exactly what the examples show, including the inconsistencies nobody noticed. This section covers preparing that data, how much of it is enough for different goals, the general abilities a narrow fine-tune can quietly damage, and the comparison that decides whether the whole thing was worth doing: against the best prompt you can write, not a weak one.

The data is the work#

A fine-tune learns exactly what the examples show, including their mistakes and inconsistencies. A few hundred clean, consistent, representative examples beat thousands of noisy ones.

• Consistent format across every example.
• Cover the real distribution of inputs, including hard and edge cases.
• Remove duplicates and contradictory examples.
• Hold out a test set before you start, and never train on it.

Where the examples come from#

Nobody sits down and writes two thousand examples, and the teams that try stop at four hundred and ship something worse than their prompt. Real datasets are assembled from sources that already exist, and each source has a characteristic defect you should go looking for rather than wait to discover in the evaluation.

source                        cost      the defect it brings

production logs, corrected    medium    only covers what users already
by whoever owns the task                do, so the tail is missing and
                                        the model will never see it

a large model's output,       low       the student inherits the teacher's
filtered by your own eval               errors; filtering is not optional

hand-written by an expert     high      too clean, too consistent, and
                                        unconsciously written to the
                                        format rather than to the task

existing structured data      very low  the mapping is not the one users
reshaped into examples                  ask for; you are teaching a
                                        translation nobody requested

synthetic variation of real   low       duplicates in disguise; a hundred
examples                                paraphrases of forty cases is
                                        forty examples with extra noise

The first row is the one that quietly decides the project. A dataset built from logs matches the traffic the model already handles, which is the traffic where the prompt was already good enough. The examples you actually need are the ones the current system got wrong, and those are exactly the ones that do not appear in a log of successful calls. Sample the failures deliberately, and be prepared for the fact that they are harder to label because they are the hard cases.

Hold out the test set before you build anything, and hold it out by something meaningful rather than at random. If your examples come from a hundred customers, split by customer, or a near-duplicate ticket from the same customer will sit on both sides and your evaluation will report a number you cannot ship against.

The quality bar, as a script you can run#

Consistent format and no contradictions are easy to agree with and hard to eyeball at two thousand rows. The checks below are twenty lines and catch the defects that cost the most, and they should run before every training job rather than once at the start. The dataset here is twelve rows with one of each defect planted, which is roughly the hit rate of a real first draft.

import json, collections

# Twelve rows of a support-ticket classifier set, with the defects a real one has.
DATA = [
    {"in": "card declined at checkout",        "out": '{"label": "billing"}'},
    {"in": "card declined at checkout",        "out": '{"label": "payments"}'},
    {"in": "cannot log in after password reset", "out": '{"label": "auth"}'},
    {"in": "cannot log in after password reset", "out": '{"label": "auth"}'},
    {"in": "invoice shows the wrong VAT rate", "out": '{"label":"billing"}'},
    {"in": "app crashes on the settings page", "out": "bug"},
    {"in": "how do I export my data",          "out": '{"label": "howto"}'},
    {"in": "refund not received after 10 days", "out": '{"label": "billing"}'},
    {"in": "SSO metadata upload fails",        "out": '{"label": "auth"}'},
    {"in": "where is the dark mode toggle",    "out": '{"label": "howto"}'},
    {"in": "i was charged twice this month",   "out": '{"label": "billing"}'},
    {"in": ("the integration has been failing since tuesday and the logs show "
            "a 502 from your side every fifteen minutes, we have tried rotating "
            "the key and it made no difference, please advise urgently"),
     "out": '{"label": "bug"}'},
]
TEST = [
    {"in": "how do I export my data", "out": '{"label": "howto"}'},
    {"in": "payment failed twice",    "out": '{"label": "billing"}'},
]

def lint(rows, test):
    seen = collections.defaultdict(set)
    for r in rows:
        seen[r["in"]].add(r["out"])

    dupes = sum(1 for r, c in collections.Counter(
        (r["in"], r["out"]) for r in rows).items() if c > 1)
    contradictions = [k for k, v in seen.items() if len(v) > 1]

    bad_json, labels = [], collections.Counter()
    for r in rows:
        try:
            labels[json.loads(r["out"])["label"]] += 1
        except Exception:
            bad_json.append(r["in"])

    spacing = collections.Counter(
        "spaced" if '": "' in r["out"] else "compact"
        for r in rows if r["out"].startswith("{"))

    lens = sorted(len(r["in"]) for r in rows)
    p50 = lens[len(lens) // 2]
    long = [r["in"][:30] + "..." for r in rows if len(r["in"]) > 4 * p50]

    leaked = [r["in"] for r in test if r["in"] in {x["in"] for x in rows}]

    print("rows", len(rows), " unique inputs", len(seen))
    print("exact duplicate rows        ", dupes)
    print("same input, different output", contradictions)
    print("output not parseable        ", bad_json)
    print("label counts                ", dict(labels))
    print("output spacing conventions  ", dict(spacing))
    print("input length p50            ", p50, " outliers:", long)
    print("test inputs seen in training", leaked)

lint(DATA, TEST)

# rows 12  unique inputs 10
# exact duplicate rows         1
# same input, different output ['card declined at checkout']
# output not parseable         ['app crashes on the settings page']
# label counts                 {'billing': 4, 'payments': 1, 'auth': 3, 'howto': 2, 'bug': 1}
# output spacing conventions   {'spaced': 10, 'compact': 1}
# input length p50             32  outliers: ['the integration has been faili...']
# test inputs seen in training ['how do I export my data']

Read the findings in order of how much damage they do. The leak is worst, because it does not degrade the model, it degrades your ability to know anything about the model. The contradiction is next: the same ticket labelled billing once and payments once teaches the model that the boundary is arbitrary, and it will be arbitrary at inference. The label counts expose the same problem at the taxonomy level, since payments appearing once against billing four times is almost always a synonym nobody standardised rather than a real class.

The two formatting findings look trivial and are not. A model trained on a mixture of compact and spaced JSON will produce a mixture, which means the parser you were going to delete has to stay. One unparseable row in twelve is eight per cent of your training signal teaching the model that plain text is sometimes acceptable, and that is enough to produce it occasionally in production, at the point where nothing is watching.

The length outlier is the one to decide about rather than fix. A single example four times the median length is either a case you genuinely need to handle, in which case you need twenty more like it, or it is noise, in which case it is teaching the model a length distribution that does not exist. What you must not do is leave one of them in and hope.

How much#

Format or style adherence     a few hundred examples
A narrow classification task  hundreds to low thousands
Broad new behaviour           many thousands, and reconsider

That guidance is worth believing more than a rule of thumb usually is, because the shape behind it can be measured. Below, the same tiny transformer is trained from scratch on the same task with training sets from 25 to 800 examples, and both the training accuracy and the held-out accuracy are reported. The task is to sort four digits, written as a twelve-character line, and nothing about it matters except that it is a rule the model has to infer rather than a list it can memorise.

import random, copy, torch, torch.nn as nn

ALPHABET = sorted(set("mrs:>;0123456789"))
STOI = {c: i for i, c in enumerate(ALPHABET)}
ITOS = {i: c for c, i in STOI.items()}
L = 12

def max_row(r):
    d = "".join(str(r.randrange(10)) for _ in range(4))
    return "m:%s>000%s;" % (d, max(d))

def rev_row(r):
    d = "".join(str(r.randrange(10)) for _ in range(4))
    return "r:%s>%s;" % (d, d[::-1])

def sort_row(r):
    d = "".join(str(r.randrange(10)) for _ in range(4))
    return "s:%s>%s;" % (d, "".join(sorted(d)))

MAKERS = {"max": max_row, "rev": rev_row, "sort": sort_row}

def sample(kinds, n, seed):
    r = random.Random(seed)
    return [MAKERS[kinds[i % len(kinds)]](r) for i in range(n)]

class Tiny(nn.Module):
    def __init__(self):
        super().__init__()
        V = len(STOI)
        self.emb, self.pos = nn.Embedding(V, 64), nn.Embedding(L, 64)
        layer = nn.TransformerEncoderLayer(64, 4, 256, dropout=0.0,
                                           batch_first=True, norm_first=True)
        self.tr = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
        self.out = nn.Linear(64, V)

    def forward(self, x):
        n = x.shape[1]
        h = self.emb(x) + self.pos(torch.arange(n))
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1)
        return self.out(self.tr(h, mask=mask))

def fit(model, rows, steps, lr=3e-3, bs=256, seed=0):
    X = torch.tensor([[STOI[c] for c in r] for r in rows])
    g = torch.Generator().manual_seed(seed)
    opt = torch.optim.AdamW(model.parameters(), lr=lr)
    for _ in range(steps):
        b = X[torch.randint(0, len(X), (min(bs, len(X)),), generator=g)]
        logits = model(b[:, :-1])
        loss = nn.functional.cross_entropy(logits.reshape(-1, len(STOI)),
                                           b[:, 1:].reshape(-1))
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        logits = model(X[:, :-1])
        return float(nn.functional.cross_entropy(
            logits.reshape(-1, len(STOI)), X[:, 1:].reshape(-1)))

def exact(model, rows):
    ok = 0
    with torch.no_grad():
        for r in rows:
            ids = [STOI[ch] for ch in r[:r.index(">") + 1]]
            while len(ids) < L:
                ids.append(model(torch.tensor([ids]))[0, -1].argmax().item())
            ok += "".join(ITOS[i] for i in ids) == r
    return ok / len(rows)

test = {k: sample([k], 200, 100 + i) for i, k in enumerate(MAKERS)}
print(test["max"][0], test["rev"][0], test["sort"][0])

print()
print("training set size against held-out accuracy, sort task")
for n in (25, 50, 100, 200, 400, 800):
    torch.manual_seed(0)
    m = Tiny()
    rows = sample(["sort"], n, 1)
    tl = fit(m, rows, 800)
    print("n=%4d  train loss %.4f  train exact %.2f  held-out exact %.2f"
          % (n, tl, exact(m, rows[:100]), exact(m, test["sort"])))

# m:2772>0007; r:9385>5839; s:2952>2259;
#
# training set size against held-out accuracy, sort task
# n=  25  train loss 0.3076  train exact 1.00  held-out exact 0.06
# n=  50  train loss 0.3698  train exact 1.00  held-out exact 0.19
# n= 100  train loss 0.4286  train exact 1.00  held-out exact 0.40
# n= 200  train loss 0.4954  train exact 1.00  held-out exact 0.57
# n= 400  train loss 0.5599  train exact 1.00  held-out exact 0.77
# n= 800  train loss 0.6400  train exact 1.00  held-out exact 0.94

The column to stare at is the third one. At every training set size, from 25 examples to 800, the model gets every training example exactly right. If that were the number you reported, the 25-example run and the 800-example run would look identical, and the first one has learned nothing: six per cent on held-out data against ninety-four. This is what overfitting on a small dataset looks like from the inside, and it is why a fine-tune that looks perfect on the data you built it from tells you nothing at all.

Notice also that the training loss goes up as the dataset grows, from 0.31 to 0.64, while the thing you care about goes up with it. Loss is measured against a training set that is getting harder to fit, so it is not comparable between the rows and should never be the number you optimise. The held-out exact-match is the only column that means anything, and the shape it traces is the ordinary one: fast improvement, then a knee, then a long expensive tail. The practical reading is to plot this curve for your own task at three sizes before committing to a labelling budget, because the difference between the knee being at 200 examples and at 5,000 is the difference between an afternoon and a quarter.

Forgetting#

Fine-tuning on a narrow task can degrade the model's general abilities. LoRA reduces this because the original weights are untouched, and mixing some general examples into the training data helps further. Always check that general capability survived, not only the target task.

The word catastrophic is not rhetorical, and the effect is easy to produce. Take the same architecture, train it on two tasks until it is good at both, then fine-tune it on a third task alone and re-measure the first two. The fine-tuning set has four hundred examples and runs for six hundred steps, which by the standards of a real fine-tune is a short, gentle run.

print()
print("catastrophic forgetting")
torch.manual_seed(0)
base = Tiny()
fit(base, sample(["max", "rev"], 2000, 2), 2500)
print("base                     max %.2f  rev %.2f  sort %.2f"
      % (exact(base, test["max"]), exact(base, test["rev"]), exact(base, test["sort"])))

narrow = sample(["sort"], 400, 3)
for name, rows, lr in (
        ("sort only,       lr 1e-3", narrow, 1e-3),
        ("sort only,       lr 3e-4", narrow, 3e-4),
        ("+ 20% replay,    lr 1e-3", narrow + sample(["max", "rev"], 100, 4), 1e-3),
        ("+ 50% replay,    lr 1e-3", narrow + sample(["max", "rev"], 400, 4), 1e-3)):
    m = copy.deepcopy(base)
    torch.manual_seed(0)
    fit(m, rows, 600, lr=lr)
    print("%s  max %.2f  rev %.2f  sort %.2f"
          % (name, exact(m, test["max"]), exact(m, test["rev"]), exact(m, test["sort"])))

# catastrophic forgetting
# base                     max 1.00  rev 0.99  sort 0.01
# sort only,       lr 1e-3  max 0.03  rev 0.05  sort 0.39
# sort only,       lr 3e-4  max 0.24  rev 0.14  sort 0.20
# + 20% replay,    lr 1e-3  max 0.94  rev 0.44  sort 0.28
# + 50% replay,    lr 1e-3  max 0.96  rev 0.78  sort 0.32

The base model does both old tasks essentially perfectly. Six hundred steps on a fourth of a thousand examples of something else takes one of them from 1.00 to 0.03 and the other from 0.99 to 0.05. Nothing in the fine-tuning loss curve indicates this, because the fine-tuning loss is computed on the fine-tuning data, where everything is going well. If you do not measure the old capability, you do not find out.

The three mitigation rows are the useful part, and each one costs something. Dropping the learning rate to a third preserved more of the old tasks, 0.24 and 0.14 against 0.03 and 0.05, and learned the new one only half as well, 0.20 against 0.39 — which is the trade in a single line: forgetting and learning are the same process, and you cannot slow one without slowing the other. Replay is the better lever: mixing in old examples at a fifth of the new set restored the first task almost completely, and at an equal share restored most of the second, while costing under a tenth of the new task's score.

Two things transfer from this toy to a real fine-tune. The first is that replay is the cheapest insurance available and almost nobody budgets for it: reserve a fifth to a half of your training set for examples of the general behaviour you want to keep, drawn from the same kind of instruction data the base model was tuned on. The second is that the old-capability check must be in the evaluation from the first run, not added after somebody notices in production. A short fixed set of general instructions, scored the same way every time, is enough.

LoRA genuinely helps here, and it is worth being exact about why. The frozen base weights cannot move, so the only thing that can drift is the low-rank update, and at r=8 or 16 that update has nowhere near the capacity to overwrite general behaviour. It is not immunity — the adapter is applied on every input, including the ones you did not fine-tune for — but it puts a ceiling on the damage that full fine-tuning does not have, and it gives you the one-line rollback of serving without the adapter.

The evaluation that tells you whether it worked#

Three numbers decide whether to ship a fine-tune, and only one of them is about the new task. The target-task score against the best prompt you can write. The general-capability score, to catch what the previous section measured. And the cost per request, because the usual reason to do this at all is to move work onto a smaller model, and a fine-tune that matches the big model at the same price has bought nothing.

The comparison itself is where results get overstated, and the fix is to score both systems on the same examples and keep the pairing. Suppose a 200-example held-out set comes back like the counts below: the two systems agree on 158 of them and disagree on 42.

import math, numpy as np

# One row per held-out example, scored by the same grader.
# These four counts are what a paired eval of 200 examples gives you.
both_right, both_wrong = 128, 30
prompt_only, tuned_only = 14, 28          # b and c: the cases that disagree
n = both_right + both_wrong + prompt_only + tuned_only

acc_prompt = (both_right + prompt_only) / n
acc_tuned = (both_right + tuned_only) / n
print("n = %d   best prompt %.3f   fine-tuned %.3f   difference %+.3f"
      % (n, acc_prompt, acc_tuned, acc_tuned - acc_prompt))

# McNemar, exact: under the null the disagreements split 50/50.
b, c = prompt_only, tuned_only
k, m = min(b, c), b + c
p = 2 * sum(math.comb(m, i) for i in range(k + 1)) / 2 ** m
print("disagreements %d (%d vs %d)   exact McNemar p = %.4f" % (m, b, c, min(p, 1.0)))

# The unpaired version, for comparison: throw the pairing away and bootstrap.
rng = np.random.default_rng(0)
a = np.array([1] * (both_right + prompt_only) + [0] * (both_wrong + tuned_only))
t = np.array([1] * (both_right + tuned_only) + [0] * (both_wrong + prompt_only))
diffs = [t[rng.integers(0, n, n)].mean() - a[rng.integers(0, n, n)].mean()
         for _ in range(10000)]
lo, hi = np.percentile(diffs, [2.5, 97.5])
print("unpaired bootstrap 95%% interval on the difference: %+.3f to %+.3f" % (lo, hi))

# n = 200   best prompt 0.710   fine-tuned 0.780   difference +0.070
# disagreements 42 (14 vs 28)   exact McNemar p = 0.0436
# unpaired bootstrap 95% interval on the difference: -0.015 to +0.155

Same data, two conclusions. The paired test asks only about the 42 examples where the systems differed and finds that 28 to 14 is unlikely under a coin flip, p = 0.044. The unpaired bootstrap throws away which example was which, and its interval on the difference runs from minus 1.5 points to plus 15.5 — it cannot rule out that the fine-tune is worse. The pairing is not a statistical nicety; it is most of your power, because the 158 examples both systems agreed on carry no information about the difference and the unpaired version drowns the signal in their variance.

The practical rules that follow are short. Score both systems on the identical held-out set, with the identical grader, in the same run. Report the disagreement counts, not just the two accuracies, because 78 against 71 sounds the same whether the disagreements are 28 to 14 or 70 to 63 and only one of those is a result. And read the fourteen examples the prompt got right and the fine-tune got wrong before shipping, because a fine-tune that is better on average and worse on a category you care about is a decision, not a number.

Serving it, and the adapter-swapping question#

A fine-tune that only exists as a checkpoint on the machine that trained it is not finished. The serving decision is a single fork: merge the adapter into the base weights and serve one ordinary model, or keep the adapter separate and apply it on top of a shared frozen base at request time. The section on building a LoRA layer from scratch shows why merging is exact and costs no latency; what follows is when to choose which.

                      merged                   adapter kept separate

memory for N tunes    N full copies of the     one base, plus N adapters
                      base weights             at tens of MB each

latency               identical to the base    one extra low-rank matmul
                                               per adapted layer, about
                                               1% of those layers at r=16

batching              one model per batch      requests using different
                                               adapters can share a batch
                                               if the server supports it

switching             load a new model         swap a tensor, milliseconds

rollback              redeploy                 stop passing the adapter

quantized base        needs re-quantizing      no interaction; the base
                      after the merge, which   stays exactly as it is
                      can undo the training

Merge when you serve one adapter at high, steady volume and the last milliseconds matter. Keep them separate in every other case, and particularly whenever there is more than one fine-tune, because the memory column is not a small difference: fifty merged copies of an 8B model is fifteen gigabytes each, and fifty adapters at r=16 is about twenty-six megabytes each on top of one shared base.

Three operational details decide whether this is pleasant or not. Version the adapter with the base model it was trained against, in the filename, because an adapter applied to a different base of the same shape will load without error and produce quiet nonsense. Keep the previous adapter served and reachable behind a flag, so a rollback is a configuration change rather than a training run. And pin the exact prompt template used during training alongside the adapter: a fine-tune learns the template as much as the task, and serving it with a prompt that differs by a system message or a separator token is the most common way a fine-tune that evaluated well behaves badly in production.

You can already say that LoRA trains a small number of extra parameters instead of all of them, and that this is what made fine-tuning cost tens of dollars rather than millions. This section writes the layer. It is about fifteen lines, and writing them is the quickest way to stop treating r and alpha as dials to copy from someone else's configuration file — which matters, because the first question after "what is LoRA" is almost always what those two numbers do.

The claim it rests on#

Full fine-tuning learns an update to every weight matrix: the new weight is the old one plus some delta of the same shape. The observation behind LoRA is that this delta, for a model being adapted to one task, has much lower rank than the matrix it is added to — the change is real but it lives in a small subspace. If that is true, the delta can be written as a product of two thin matrices and only those need to be trained.

full fine-tuning    W_new = W + dW           dW is (d_out, d_in), all of it trained
LoRA                W_new = W + (a/r) B A    B is (d_out, r), A is (r, d_in)

                    r is much smaller than either dimension, typically 8 to 64

The layer#

The base weight is frozen: it is read in the forward pass and never receives a gradient. The adapter runs alongside it on the same input, and the two outputs are added. Nothing is inserted in sequence, which is the difference from the older adapter designs and the reason a merged LoRA costs no extra latency.

import numpy as np

rng = np.random.default_rng(0)
d_in, d_out, r, alpha = 768, 768, 8, 16

W = rng.normal(size=(d_out, d_in)) / np.sqrt(d_in)   # frozen, never updated
A = rng.normal(size=(r, d_in)) * 0.01                # small random
B = np.zeros((d_out, r))                             # exactly zero
scale = alpha / r


def forward(x):                                      # x is (batch, d_in)
    base = x @ W.T                                   # (batch, d_out), no gradient
    delta = (x @ A.T) @ B.T * scale                  # (batch, r) then (batch, d_out)
    return base + delta

The order of the two matmuls is not cosmetic. Going through A then B costs batch times r times (d_in + d_out) operations; forming B @ A first and then multiplying costs batch times d_in times d_out, which is the full-rank cost and throws away the saving. On a small example the difference is invisible; on a real layer it is the difference between a training step that fits and one that does not.

Why B starts at zero#

At initialisation the adapter must contribute nothing, so that training starts from exactly the base model rather than from a randomly perturbed one. Zeroing B achieves that. A is random because if both were zero the product would have no gradient direction to move in at all.

x = rng.normal(size=(4, d_in))
assert np.allclose(forward(x), x @ W.T)              # the adapter is inert at step 0

# g stands in for the gradient of the loss with respect to this layer's output
g = rng.normal(size=(4, d_out))
h = x @ A.T                                          # (batch, r)

grad_B = g.T @ h * scale                             # (d_out, r), nonzero
grad_A = (g @ B).T @ x * scale                       # (r, d_in), exactly zero: B is zero

assert np.abs(grad_B).max() > 0
assert np.abs(grad_A).max() == 0

What r does#

r caps the rank of the update, and that is the entire story. B @ A is a product of a (d_out, r) and an (r, d_in) matrix, so whatever the training does, the resulting delta can have rank at most r. If the change the task requires genuinely needs rank 20, an adapter at r=8 cannot represent it, and no amount of training will find it.

This is demonstrable without training anything. Take a target update that is exactly rank 3, and ask for the best approximation available at each r — which is the truncated singular value decomposition, by Eckart-Young.

target = rng.normal(size=(d_out, 3)) @ rng.normal(size=(3, d_in))   # rank 3 by construction
target /= np.linalg.norm(target)

U, S, Vt = np.linalg.svd(target, full_matrices=False)
for rr in [1, 2, 3, 4, 8]:
    best = (U[:, :rr] * S[:rr]) @ Vt[:rr]            # best rank-rr approximation
    print(rr, round(float(np.linalg.norm(target - best)), 4))

# 1 0.7984
# 2 0.5534
# 3 0.0
# 4 0.0
# 8 0.0

Real updates are not exactly low rank, so the curve is a gentle decline rather than a cliff, which is why r=8 or 16 is usually enough for style, format and domain vocabulary, and why raising r past 64 rarely helps. The useful instinct is that if raising r improves your evaluation substantially, the change you are asking for is large, and full fine-tuning is the honest comparison.

What alpha does#

alpha is a fixed multiplier on the adapter's output, applied as alpha over r. It is not learned and it does not change what the adapter can represent; it changes how large the update is for a given set of trained weights, which in practice behaves like a learning rate for the adapter branch.

Dividing by r is the part people misremember. Left unscaled, the product B @ A grows as r grows — more rank-one terms summed — so an r=32 adapter would arrive with a systematically larger update than an r=8 one and would need a different learning rate. The division removes most of that coupling, so r can be changed without retuning.

print(f"{'r':>3} {'||BA||':>8} {'(16/r)||BA||':>14}")
for rr in [4, 8, 16, 32]:
    Ar = rng.normal(size=(rr, d_in)) * 0.01
    Br = rng.normal(size=(d_out, rr)) * 0.05         # a trained B, roughly
    raw = np.linalg.norm(Br @ Ar)
    print(f"{rr:>3} {raw:>8.3f} {16 / rr * raw:>14.3f}")

#   r   ||BA||   (16/r)||BA||
#   4    0.774          3.097
#   8    1.084          2.169
#  16    1.534          1.534
#  32    2.165          1.083

What it actually saves#

The parameter count is arithmetic worth being able to do at a whiteboard: a full matrix is d_in times d_out, and the adapter is r times the sum of the two.

d_in = d_out = 768
r = 8

base = d_in * d_out                # 589,824
lora = r * (d_in + d_out)          # 12,288
print(round(100 * lora / base, 2)) # 2.08

The parameter saving is not the saving that matters most. Training memory is dominated by the optimiser: Adam holds two momentum terms per trainable parameter, and gradients need another copy, so a trainable parameter costs several times what a frozen one does. Frozen weights need only to be stored — often quantized — and read.

7B model, full fine-tune, fp16 weights with Adam in fp32
  weights          14 GB
  gradients        14 GB
  Adam states      56 GB          two fp32 moments per parameter
  total           ~84 GB before activations

same model, LoRA at r=16 on the attention projections (~0.2% of parameters)
  weights          14 GB          frozen, and quantizable to ~4 GB at 4-bit
  gradients      ~0.03 GB
  Adam states     ~0.1 GB
  total           ~14 GB, or ~5 GB with a quantized base

Merging#

Because the adapter is a parallel branch that is added, its contribution can be folded into the base weight once training is finished. The merged model is an ordinary model of the original shape, with no extra matrices and no extra latency at inference.

W_merged = W + scale * (B @ A)                     # (d_out, d_in), the original shape
assert np.allclose(x @ W_merged.T, forward(x))

Whether to merge is a serving decision rather than a training one. Merging gives one model per adapter and the lowest possible latency. Keeping adapters separate lets one frozen base serve many of them at once, switched per request, at the cost of the small extra matmul — which is how a provider offers thousands of fine-tunes without thousands of model copies. There is one case where merging is not clean: if the base was quantized, merging a full-precision update into it requires re-quantizing, and the rounding that follows can undo part of what was trained.