Turning a predictor into an assistant

A model straight out of pretraining is not something you would ship. Ask it a question and it may answer with more questions, because that is what a page of questions looks like in its training data. This section covers the stages that close the gap: instruction tuning on demonstrations, then training against human preferences so the model learns which of two answers people prefer. It is also where the limits get stated plainly — what preference training does not fix, and why a well-behaved model can still be talked out of its behaviour.

A base model is not an assistant#

Straight out of pretraining, a model continues text. Ask it a question and it may continue with more questions, because that is what a list of questions looks like in its training data. It is not being unhelpful; helpfulness was never the objective.

Base model, prompted "What is the capital of France?"

  "What is the capital of Germany?
   What is the capital of Italy?
   ..."

A plausible continuation. Not an answer.

It is worth building the smallest possible version of that, because the mechanism is not mysterious once you see it as counting. Here is a corpus in which most documents are lists of questions and a minority answer them, and a trigram model fitted to it. Given the question, the base model's most likely next word is the start of another question, 91 per cent to 9. Fit the same kind of model to instruction data instead — question, then answer, never anything else — let it carry the pretraining counts at a small weight so the knowledge survives, and the distribution inverts.

from collections import Counter, defaultdict
import random
random.seed(0)

CAPITALS = {"france": "paris", "germany": "berlin", "italy": "rome",
            "spain": "madrid", "japan": "tokyo", "peru": "lima"}

# Pretraining data: whatever the web holds. Mostly pages that list questions.
pretrain = []
for _ in range(300):
    qs = random.sample(list(CAPITALS), 4)
    pretrain.append(" ".join(f"what is the capital of {c} ?" for c in qs))
for _ in range(100):                                  # and some pages that answer them
    c = random.choice(list(CAPITALS))
    pretrain.append(f"what is the capital of {c} ? {CAPITALS[c]} is the capital of {c} .")

# Instruction data: only ever a question followed by its answer.
sft = [f"what is the capital of {c} ? {CAPITALS[c]} is the capital of {c} ."
       for c in CAPITALS for _ in range(200)]

def fit(docs):                                        # a trigram count model
    counts = defaultdict(Counter)
    for d in docs:
        w = d.split()
        for i in range(2, len(w)):
            counts[(w[i-2], w[i-1])][w[i]] += 1
    return counts

base = fit(pretrain)
tuned = fit(sft)
for ctx, c in base.items():          # the fine-tune keeps what pretraining knew, and
    for w, n in c.items():           # outweighs it wherever it has an opinion of its own
        tuned[ctx][w] += n * 0.05

def after(counts, ctx):
    total = sum(counts[ctx].values())
    return [(w, round(n / total, 2)) for w, n in counts[ctx].most_common(2)]

print("next word after '... capital of france ?'")
print("  base model  ", after(base, ("france", "?")))
print("  after tuning", after(tuned, ("france", "?")))
# next word after '... capital of france ?'
#   base model   [('what', 0.91), ('paris', 0.09)]
#   after tuning [('paris', 0.96), ('what', 0.04)]

The point the toy makes properly is that the base model already knew the answer. Paris was there at 9 per cent before any tuning. Nothing was taught; a preference between two continuations it already had was changed. That is worth holding onto, because it is why instruction tuning needs thousands of examples rather than billions, and why it is a poor way to add knowledge the model does not have.

The three stages, and what each one buys#

stage                 learns from                  data scale        what it changes
pretraining           raw text, next token         trillions of tokens  what the model knows
instruction tuning    written question-answer      thousands of pairs   that it answers at all
                      demonstrations                                    and in what format
preference tuning     humans comparing two         tens of thousands    which of two acceptable
                      responses to one prompt      of comparisons       answers it gives

The reason the stages are ordered this way is that each needs a cheaper kind of label than the one before could have used. Nobody can write a demonstration for every question, but a few thousand carefully written ones establish the shape. Nobody can write a demonstration that is better than the model's own output once the model is good, but anyone can still say which of two outputs they prefer. Preference data is the only supervision that keeps working after the model has passed the people supervising it, and that, rather than any mathematical elegance, is why the field ended up here.

Instruction tuning#

Supervised fine-tuning on instruction and response pairs teaches the model that a question should be followed by an answer. This alone produces most of the difference between a base model and something usable.

Mechanically it is ordinary next-token training with two changes. The conversation is rendered into one string using the model family's chat template, which marks with special tokens who is speaking, and the loss is masked so only the assistant's tokens count — otherwise you are also training the model to write the user's half. Both are covered properly in the two hands-on sections of this topic. What is worth knowing at this level is that the format is not cosmetic: a model fine-tuned with one template and served with another degrades in a way that looks like the fine-tune having failed.

Preferences#

Quality is easier to compare than to specify. Show a person two responses and ask which is better, collect many such judgements, and use them to shape the model. RLHF trains a reward model on those preferences and optimises against it; DPO reaches similar ends without a separate reward model, which is simpler and now widely used.

The full RLHF pipeline is four steps and it helps to be able to name them in order rather than describing it as a single thing.

1. 1.Sample several responses to the same prompt from the instruction-tuned model, and have people rank or compare them.
2. 2.Train a reward model on those comparisons: a copy of the model with the token-prediction head replaced by one that outputs a single number, trained so the preferred response scores higher.
3. 3.Optimise the policy — the model you are shipping — to maximise that number, using a reinforcement learning algorithm because the reward arrives only at the end of a generated sequence.
4. 4.Penalise the policy for drifting away from where it started, measured as KL divergence from the instruction-tuned model, because without that term it will find whatever the reward model scores highly rather than whatever is good.

DPO removes the middle of that. The key observation is that the policy maximising reward subject to a KL penalty has a known closed form, and it can be rearranged so that the reward is expressible in terms of the policy itself. Substituting that into the preference loss gives an objective you can train by gradient descent directly on pairs of chosen and rejected responses — no reward model to train, no reinforcement learning loop, no sampling during training. The details, and a demonstration that the two routes land in the same place, are in the DPO section of this topic.

What remains unsolved#

• Sycophancy. Agreeable answers are rated highly, so models learn to agree.
• Calibration. Models remain poor at signalling their own uncertainty.
• Judging beyond your own expertise. Preference data is limited by what annotators can evaluate.
• Robustness. Alignment achieved in training does not hold under all inputs, which is what jailbreaks demonstrate.

The first three share a cause, and stating it plainly is more useful than the list. Preference tuning optimises for what a rater picks, and what a rater picks is not what a user needs. A rater skims, so length reads as thoroughness. A rater is being agreed with, so agreement reads as helpfulness. A rater cannot check a claim outside their expertise, so a confident wrong answer and a confident right one score the same. None of this is the reward model malfunctioning — a reward model that captured rater behaviour perfectly would encode all three. It is the objective being a proxy for the goal, and the gap between them shows up as behaviour the moment you optimise hard against it. The mechanism, with numbers, is in the following section.

Robustness is a different problem with a different shape. Post-training changes behaviour on inputs resembling the training distribution; it does not install a rule. A jailbreak is an input far enough from that distribution that the trained behaviour does not fire, which is why new ones keep being found and why every mitigation is empirical. Treat a model's refusals as a strong default rather than a guarantee, and put the guarantee somewhere you control.

What to skip#

Skip the PPO implementation details — the clipped surrogate objective, the value head, the advantage estimation — unless you are going to run one. Everyone who is not on a post-training team can hold RLHF as reward model plus KL-penalised optimisation and lose nothing. Skip the survey of every preference-optimisation variant that followed DPO; they are worth reading about when one of them is proposed to you and not before. What is genuinely worth memorising is the order of the three stages and the scale of each, the reason preference data exists at all, and the sentence that explains most observed model behaviour: it was trained towards what raters chose.