How preference training is done

The models you call have been through a stage after pretraining that shaped how they answer, and it is worth knowing what that stage consists of. This section walks the pipeline: where preference data comes from and why people are asked to compare rather than score, what a reward model is and why it is only a stand-in for judgement, and what stops the training from optimising that stand-in into nonsense. The failure at the end is the general one, which is that a proxy optimised hard enough stops standing in for the thing you wanted.

Preference data#

Show annotators two responses to the same prompt and ask which is better. Comparison is used rather than scoring because people are far more consistent at ranking than at assigning absolute numbers. The quality of this data bounds everything downstream.

A record is a prompt, two responses, and which one won. That is all. Ties are usually either dropped or split, and a per-item confidence is sometimes collected and then mostly ignored. Both responses are sampled from the same model at a non-zero temperature, so the comparison is between two things the model would actually have said — which matters, because a preference pair made of one model's output against a human-written gold answer teaches the model about a distribution it cannot reach.

{"prompt": "My deployment is failing with OOMKilled. What do I check?",
 "chosen":   "OOMKilled means the container exceeded its memory limit. Check ...",
 "rejected": "There are many possible causes of deployment failures. It is ..."}

Reward models#

A model trained on those comparisons to predict which response a person would prefer. It becomes a stand-in for human judgement, which lets you score millions of responses. It is also a proxy, and optimising hard against a proxy finds its weaknesses.

Architecturally it is the language model you already have with the token-prediction head replaced by a linear layer that emits one number for a whole sequence. The training objective is the Bradley-Terry model: the probability a rater prefers the chosen response is the logistic function of the difference in scores, so the loss is minus the log of a sigmoid of that difference. Two lines. Note what is absent — no target score anywhere, because none was ever collected. Only differences are supervised, which means the reward model's absolute output is meaningless and only its ranking carries information.

loss = -log sigmoid( r(prompt, chosen) - r(prompt, rejected) )

Here is one trained end to end, with responses reduced to six measurable traits so that everything stays inspectable. The rater whose picks generate the data is swayed by length, flattery and formatting on top of the things that genuinely help. Nobody told the reward model that. The question is what it learns.

import torch, torch.nn.functional as F
torch.manual_seed(0)

NAMES = ["correct", "specific", "hedging", "length", "flattery", "formatting"]
N = 20_000
x = torch.randn(N, 6)              # 20,000 candidate responses, as measurable traits

def rater_score(z):                # what an annotator picks, and all the RM can ever learn
    return 2*z[:,0] + 1*z[:,1] - 0.8*z[:,2] + 0.8*z[:,3] + 0.6*z[:,4] + 0.2*z[:,5]

def helpfulness(z):                # what the user actually gets. Never observed by anyone.
    return (3*torch.tanh(z[:,0]) + 1.5*torch.tanh(z[:,1]) - 1.2*torch.tanh(z[:,2])
            + 1.0*z[:,3] - 0.5*z[:,3]**2)

i, j = torch.randint(N, (4000,)), torch.randint(N, (4000,))
gap = rater_score(x[i]) - rater_score(x[j]) + 0.5*torch.randn(4000)
chosen = torch.where(gap > 0, i, j)
rejected = torch.where(gap > 0, j, i)

w = torch.zeros(6, requires_grad=True)
opt = torch.optim.Adam([w], lr=0.05)
for _ in range(3000):
    loss = -F.logsigmoid(x[chosen] @ w - x[rejected] @ w).mean()   # Bradley-Terry
    opt.zero_grad(); loss.backward(); opt.step()
w = w.detach(); w = w / w.norm()

print("  " + "  ".join(f"{n}={v:.2f}" for n, v in zip(NAMES, (w * 2 / w[0]).tolist())))
hi, hj = torch.randint(N, (20_000,)), torch.randint(N, (20_000,))
agree = ((x[hi] @ w > x[hj] @ w) == (rater_score(x[hi]) > rater_score(x[hj]))).float().mean()
print(f"  agrees with the rater on {agree*100:.1f}% of held-out pairs")
#   correct=2.00  specific=1.01  hedging=-0.79  length=0.79  flattery=0.61  formatting=0.19
#   agrees with the rater on 99.7% of held-out pairs

That is the result to sit with. The reward model is not broken, overfitted or under-trained. It is 99.7 per cent accurate at the task it was given, and the task it was given was to predict rater choices. Flattery at 0.61 and length at 0.79 are in there because they were in the data, and no amount of more data, a bigger model or better regularisation removes them. Sycophancy is not a bug in the reward model; it is the reward model working.

Why there is a KL penalty#

Optimising purely for reward drives the model towards outputs that exploit the reward model: strange, repetitive or sycophantic text that scores well and reads badly. A penalty on diverging too far from the original model keeps it anchored to sensible language while it improves.

maximise   E [ r(x) ]  -  beta * KL( policy || reference )

This objective has a closed-form solution, and knowing it is worth more than any amount of intuition about the penalty. The optimal policy is the reference policy reweighted by the exponential of reward over beta: p(x) proportional to p_ref(x) * exp(r(x) / beta). Large beta and the exponent is near zero, so you get the reference model back. Small beta and all the mass collapses onto whatever scores highest. Every RLHF run is somewhere on that path, and beta says where.

Because it is closed form, the whole path can be walked exactly rather than trained. Carrying on from the reward model above, with a Gaussian reference policy the optimum is a shift of the mean by w / beta, so each row below is the true optimum at that value of beta — not an approximation and not a training run that might have converged badly.

print(" beta     KL   proxy reward   rater score   true helpfulness   mean length")
for beta in (2.0, 1.0, 0.5, 0.3, 0.2, 0.15, 0.1):
    mu = w / beta                                  # the exact KL-regularised optimum
    s = torch.randn(400_000, 6) + mu
    print(f"{beta:5.2f}  {0.5*(mu@mu):5.2f}   {(s @ w).mean():11.3f}   "
          f"{rater_score(s).mean():11.3f}   {helpfulness(s).mean():16.3f}   {mu[3]:11.3f}")
#  2.00   0.12         0.499         1.290              0.619         0.152
#  1.00   0.50         1.002         2.589              1.660         0.304
#  0.50   2.00         2.001         5.171              3.289         0.608
#  0.30   5.56         3.333         8.614              4.506         1.013
#  0.20  12.50         4.997        12.915              5.031         1.520
#  0.15  22.22         6.667        17.231              4.934         2.027
#  0.10  50.00        10.001        25.846              3.589         3.040

That divergence is Goodhart's law with numbers attached, and the shape of it is the thing to recognise. Every metric anyone can see keeps improving: the reward goes from 5.0 to 10.0, the rater score from 12.9 to 25.8. The quantity that matters peaks and turns. If you were running this and watching the reward curve — which is what you get to watch — you would conclude the last third of training was the most productive part of it, and you would be shipping a model 29 per cent worse than the one you already had two rows earlier.

The mechanism is visible in the last column. Length is the trait the rater rewarded linearly and the user benefits from only up to a point. As beta falls, the policy pushes length further and further out, past the point of usefulness, because nothing in the reward model knows there is a point. Optimising against a proxy does not fail gradually and everywhere; it fails at the specific traits where the proxy and the goal come apart, and it fails there hard.

The practical defences follow from that table and none of them is clever. Keep beta high enough that KL stays in a range you have previously evaluated. Checkpoint often and evaluate the checkpoints against something the reward model had no part in. Retrain the reward model on fresh comparisons drawn from the current policy, which is the point of doing RLHF in rounds rather than once, because the old reward model has never seen the outputs the new policy produces. And measure the specific proxies you can name — response length above all — before and after, because the fastest way to spot this failure is that the average answer got 40 per cent longer.

AI feedback#

Human preference data is expensive and limited by what annotators can judge. Constitutional approaches have a model critique and revise its own outputs against written principles, generating preference data at scale. This moves some of the judgement from annotators to a written specification, which is easier to inspect and argue about.

The procedure has two halves and they are usually conflated. The first is supervised: the model produces a response, is asked to critique it against a specific written principle, and then to revise it; the revised responses become fine-tuning data. The second is the preference half, RLAIF, where the model is shown two of its own responses and a principle, and asked which better satisfies it; those judgements train the reward model in place of human ones.

1  model answers a prompt
2  "Identify ways this response is harmful or evasive."      <- a written principle
3  "Rewrite it to remove what you identified."
4  fine-tune on (prompt, revised answer)

5  for preferences: show the model two of its own answers plus a principle,
   ask which better satisfies it, and train the reward model on those labels

What this actually buys is worth being precise about, because it is oversold in both directions. It buys scale and it buys inspectability: the judgement now lives in a document you can read, version and argue about in a review, instead of in the aggregate instincts of a rating pool. It does not buy independence from human judgement — the principles were written by people, and the model applying them was itself aligned by human preference data — and it does not escape the proxy problem from earlier in this section. It replaces one proxy with another that happens to be legible. Legible is a real improvement; independent it is not.

The failure to watch for is correlated error. A single model generating responses, judging them and being trained on the result has no outside check, so a blind spot it has is one the training will reinforce rather than reveal. Teams mitigate this by using a different model as the judge, by keeping a human-labelled slice as the yardstick, and by checking the AI labels against human labels on a sample — which is the same validation you would do for any automated judge, and the step most often skipped.

What to skip#

Skip the PPO mechanics unless you are going to run one. Skip the derivation of the closed-form optimum; using it matters, deriving it does not. Skip the catalogue of reward-model architectures. What is worth memorising is the preference-pair loss, the KL-regularised objective and its closed form, the fact that a perfect reward model still encodes the rater's biases, and the shape of the overoptimisation curve — proxy up, goal up then down — because recognising that shape is what the failure looks like from inside a run.