Preference tuning with DPO, hands on

Supervised fine-tuning teaches a model what a good answer looks like. Preference tuning teaches it which of two answers is better, and DPO does that straight from pairs of a chosen and a rejected response, with no reward model and no reinforcement learning loop. This section runs one, explains what beta controls, which is how far the model may drift from a frozen reference copy of itself, and covers the two failures that make a DPO run look like a success: learning that longer is better, and widening the gap by making both answers less likely.

The data#

{"prompt": "Explain what an API rate limit is.",
 "chosen": "A rate limit caps how many requests you can make in a period, such as 100 per minute...",
 "rejected": "Rate limits are a complex topic with many aspects to consider. First, it is important to..."}

What the loss does#

For each pair, direct preference optimisation compares how much more likely the model makes the chosen answer than a frozen reference copy does, against the same ratio for the rejected answer. The loss pushes the first up relative to the second. The reference keeps the model from drifting too far, playing the role of the KL penalty in RLHF.

loss = -log sigmoid( beta * [ (log pi(chosen)  - log ref(chosen))
                            - (log pi(rejected) - log ref(rejected)) ] )

Where it comes from is worth one paragraph, because it is the only part people find mysterious and it is not. RLHF maximises reward minus a KL penalty, and that problem has a closed-form answer: the optimal policy is the reference reweighted by exp(reward / beta). Rearranging gives the reward in terms of the policy — reward equals beta times the log ratio of policy to reference, plus a constant that depends only on the prompt. Substitute that into the Bradley-Terry preference loss and the constant cancels, because the loss only ever sees a difference between two responses to the same prompt. What is left is the line above. The reward model has not been approximated away; it has been written out algebraically.

Which means the claim is checkable. Take a single prompt with six candidate answers, a known reward, and a reference policy. Compute the RLHF optimum in closed form. Then throw the reward away, generate preference pairs from it, and train the DPO objective on nothing but which answer won. If the derivation is right, the two should agree.

import torch, torch.nn.functional as F
torch.manual_seed(0)

reward = torch.tensor([2.0, 1.4, 0.9, 0.3, -0.5, -1.2])     # how good each answer is
ref_logits = torch.tensor([0.2, 0.9, 0.4, 1.5, 0.7, 0.1])   # the model before tuning
ref_logp = F.log_softmax(ref_logits, 0)
BETA = 0.5

target = F.softmax(ref_logp + reward / BETA, 0)             # the closed form from RLHF
print("reference policy  ", [round(v, 3) for v in ref_logp.exp().tolist()])
print("KL-optimal policy ", [round(v, 3) for v in target.tolist()])

# 40,000 pairs sampled from the reference model and labelled by a Bradley-Terry
# rater on the true reward. From here on, the reward is never used again.
i = torch.multinomial(ref_logp.exp(), 40_000, replacement=True)
j = torch.multinomial(ref_logp.exp(), 40_000, replacement=True)
i, j = i[i != j], j[i != j]
pick_i = torch.rand(len(i)) < torch.sigmoid(reward[i] - reward[j])
chosen, rejected = torch.where(pick_i, i, j), torch.where(pick_i, j, i)
# reference policy   [0.096, 0.193, 0.117, 0.351, 0.158, 0.087]
# KL-optimal policy  [0.533, 0.323, 0.072, 0.065, 0.006, 0.001]

logits = ref_logits.clone().requires_grad_(True)
opt = torch.optim.Adam([logits], lr=0.02)
for step in range(4000):
    logp = F.log_softmax(logits, 0)
    margin = BETA * ((logp[chosen] - ref_logp[chosen])
                     - (logp[rejected] - ref_logp[rejected]))
    loss = -F.logsigmoid(margin).mean()                      # the DPO objective
    opt.zero_grad(); loss.backward(); opt.step()
    if step in (0, 100, 500, 3999):
        p = F.log_softmax(logits, 0).detach()
        print(f"step {step:4d}  loss {loss.item():.4f}  log pi(chosen) {p[chosen].mean():6.3f}"
              f"  log pi(rejected) {p[rejected].mean():6.3f}"
              f"  margin {(p[chosen]-p[rejected]).mean():.3f}")

learned = F.softmax(logits, 0).detach()
print("learned policy    ", [round(v, 3) for v in learned.tolist()])
print("largest gap from the closed form:", round((learned - target).abs().max().item(), 4))
# step    0  loss 0.6931  log pi(chosen) -1.735  log pi(rejected) -1.723  margin -0.012
# step  100  loss 0.5436  log pi(chosen) -1.807  log pi(rejected) -2.786  margin 0.978
# step  500  loss 0.5243  log pi(chosen) -2.205  log pi(rejected) -3.712  margin 1.508
# step 3999  loss 0.5243  log pi(chosen) -2.208  log pi(rejected) -3.718  margin 1.510
# learned policy     [0.526, 0.319, 0.079, 0.069, 0.006, 0.001]
# largest gap from the closed form: 0.0073

Now read the middle of that output, because it contains the failure everyone is warned about and almost nobody sees. The log-probability of the chosen answers goes from -1.735 to -2.208. The model is being trained on preferences and it is making the preferred answers less likely. The margin still grows, from -0.012 to 1.510, because the rejected answers fall faster. That is the objective working exactly as written — it only ever constrains the difference — and on a real model it means the policy is moving probability mass onto responses that appear in neither column of your dataset.

Here it is harmless, because six answers exhaust the space and the result still matches the optimum. On a language model with an unbounded output space it is not harmless, which is the whole reason the trainer logs the chosen log-probability separately. A run where the margin rises and the chosen log-probability falls steeply is not converging on your preferences; it is escaping them.

A run#

from datasets import load_dataset
from trl import DPOConfig, DPOTrainer

trainer = DPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train"),
    args=DPOConfig(output_dir="dpo-out", beta=0.1, learning_rate=5e-7),
)
trainer.train()

Beta controls how strongly the model may move away from the reference. Small beta allows large changes; large beta keeps it close. Learning rates for DPO are far lower than for supervised fine-tuning.

Sweeping it on the six-answer problem makes that concrete. At beta = 2 the model has barely moved and its favourite answer is still the reference's favourite, which is the fourth best of the six. At beta = 0.1 it has collapsed almost entirely onto the single best answer and travelled 2.31 nats from where it started.

beta     KL   top answer   P(top)   P(worst two)
2.00   0.09       3        0.291      0.1233     barely moved; still prefers the reference's pick
1.00   0.32       1        0.324      0.0537
0.50   0.89       0        0.526      0.0074     picks the best answer, keeps the alternatives
0.25   1.70       0        0.829      0.0001
0.10   2.31       0        0.995      0.0000     one answer, always

That is the trade in one table. High beta preserves the reference model, including whatever was wrong with it. Low beta gets the preferences but flattens variety, and on a real model that shows up as answers that are hard to steer with a prompt because the distribution has become sharp. The published default of 0.1 sits at the aggressive end of this and is worth treating as a starting point rather than a setting, particularly on preference data you collected yourself and therefore have less of.

The low learning rate has a reason too. DPO's gradient is largest on pairs the model currently gets wrong, and early in training that is most of them, so a rate that would be modest for supervised fine-tuning moves the policy a very long way in the first few hundred steps. A DPO run at 2e-4 does not diverge visibly; it produces a model that has drifted somewhere unhelpful while the loss curve looks fine. Ranges between 1e-7 and 1e-6 are normal.

How it goes wrong#

• Length: if chosen answers are longer on average, the model learns that longer is better, whatever the content. Check average response length before and after.
• Both likelihoods fall: the model can widen the gap by making both answers less likely, drifting towards text that is neither. Watch the chosen answer's log probability in the training logs.
• Overfitting to the preference data's quirks, such as a favoured phrase.

Evaluate it#

Compare against the starting model on held-out prompts with a judge you have validated, controlling for length. Report the win rate with and without length control; a large difference means the gain was mostly length.

The size of that correction surprises people, so here it is simulated with a judge whose bias is known exactly. The tuned checkpoint is genuinely a little better and also 130 tokens longer, and the judge is swayed by length as judges reliably are.

import numpy as np
rng = np.random.default_rng(0)
n = 4000

# Two checkpoints answering the same prompts. The tuned one is slightly better,
# and much longer.
qual_a, len_a = rng.normal(0, 1, n), rng.normal(300, 90, n)
qual_b, len_b = rng.normal(0.15, 1, n), rng.normal(300, 90, n) + 130

judge = lambda q, L: q + 0.004 * L + rng.normal(0, 0.5, n)   # scores quality, likes length
wins = judge(qual_b, len_b) > judge(qual_a, len_a)
print(f"raw win rate: {wins.mean():.1%}")
print(f"mean length: before {len_a.mean():.0f} tokens, after {len_b.mean():.0f} tokens")

close = np.abs(len_b - len_a) < 25          # keep only the near-equal-length pairs
print(f"length-matched win rate ({close.sum()} of {n} pairs): {wins[close].mean():.1%}")
# raw win rate: 66.3%
# mean length: before 301 tokens, after 430 tokens
# length-matched win rate (365 of 4000 pairs): 55.3%

Matching throws away 91 per cent of the data to get that answer, which is fine at 4,000 pairs and wasteful at 200. The alternative is to regress the outcome on the length difference and read the win rate off at zero, which uses everything.

from sklearn.linear_model import LogisticRegression

d = ((len_b - len_a) / 100).reshape(-1, 1)
fit = LogisticRegression().fit(d, wins)
print(f"win rate at equal length: {fit.predict_proba([[0.0]])[0][1]:.1%}")
print(f"each extra 100 tokens is worth {fit.coef_[0][0]:.2f} in log odds")
# win rate at equal length: 54.6%
# each extra 100 tokens is worth 0.41 in log odds

Report both numbers and the length change alongside them. A win rate quoted without the length delta is not a claim an experienced reader will accept, and quoting it is the fastest way to lose an argument about whether your run helped.

What to skip#

Skip the variants until one is proposed to you — IPO, KTO, ORPO, SimPO and the rest each change one term and each claim to fix one of the failures above, and none of them changes what you need to check. Skip building your own preference-data collection interface before you have run DPO once on a public dataset. What is worth holding: the loss itself, that beta is the KL penalty in disguise, that the chosen log-probability falling is the signal to watch, and that a win rate without a length control is not a result.