Supervised fine-tuning, hands on

Supervised fine-tuning is showing a model a few hundred examples of the conversation you want and training it to produce the assistant's half. This section runs one end to end on a small model: formatting the data with the tokenizer's own chat template, masking the loss so only the assistant tokens count, and using LoRA so it fits on a single GPU. The part that decides the outcome is the data, because fine-tuning copies the habits of its examples faithfully, mistakes included, so a careful thousand beats a scraped hundred thousand.

The data#

{"messages": [
  {"role": "system", "content": "You are a concise support assistant for a billing product."},
  {"role": "user", "content": "How do I download last month's invoice?"},
  {"role": "assistant", "content": "Go to Billing, then Invoices, and choose Download next to the month."}
]}

Each model family has a chat template that turns messages into one string with special tokens marking who is speaking. Always use the template that ships with the model's tokenizer; formatting it yourself slightly differently quietly hurts results.

what the template turns those three messages into, roughly:

  <|im_start|>system
  You are a concise support assistant for a billing product.<|im_end|>
  <|im_start|>user
  How do I download last month's invoice?<|im_end|>
  <|im_start|>assistant
  Go to Billing, then Invoices, and choose Download next to the month.<|im_end|>

Two details in that rendering cause most of the quiet failures. The special markers are single tokens the model was pretrained to treat as boundaries, so writing the literal characters yourself produces a different token sequence that happens to look the same to you. And the template appends the opening assistant marker when you ask it for a generation prompt but not when you are formatting a complete training example; getting that backwards trains the model to emit a marker it should be given, or to expect one it will never receive.

Only learn from the answers#

If the loss includes the user's messages, the model also learns to write user messages. Masking the loss so only assistant tokens count focuses the training on the behaviour you want. Current trainer libraries do this by default for prompt and completion data and have an option for conversations.

The mechanism is one convention: cross-entropy in PyTorch ignores any position whose label is -100, so masking is setting the label to -100 everywhere you do not want a gradient. Seeing it on a single example makes the size of the effect obvious. A support answer is often a quarter of the length of the template, system prompt and question that precede it, so an unmasked run spends three quarters of its gradient budget learning to write the prompt.

import torch, torch.nn.functional as F
torch.manual_seed(0)

# One example, already tokenised: 14 tokens of template, system prompt and
# question, then 6 tokens of the answer you actually want the model to learn.
ids = torch.randint(0, 1000, (20,))
prompt_len = 14

labels = ids.clone()
labels[:prompt_len] = -100                       # -100 is what cross_entropy ignores
print(f"{(labels != -100).sum().item()} of {len(labels)} positions carry loss")

logits = torch.randn(20, 1000)                   # stand-in for a model's output
print(f"loss over everything: {F.cross_entropy(logits[:-1], ids[1:]):.4f}")
print(f"loss over the answer: {F.cross_entropy(logits[:-1], labels[1:], ignore_index=-100):.4f}")
print(f"unmasked spends {(len(ids)-prompt_len)/len(ids):.0%} of its gradient on the answer")
# 6 of 20 positions carry loss
# loss over everything: 7.5243
# loss over the answer: 7.9973
# unmasked spends 30% of its gradient on the answer

In a multi-turn conversation the mask is not one contiguous block: every assistant turn carries loss and every user turn does not, which is why the trainer flag is named for assistant-only loss rather than completion-only. Check the mask on one real example before launching a run. Decode the positions where the label is not -100 and read them; they should be exactly the assistant text and its end-of-turn marker, and nothing else. This costs two minutes and catches the most expensive silent bug in the whole pipeline.

A run#

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("json", data_files="support.jsonl", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=data,
    args=SFTConfig(output_dir="sft-out", num_train_epochs=2,
                   learning_rate=2e-4, assistant_only_loss=True),
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear"),
)
trainer.train()

The settings in that call are not arbitrary and are worth being able to justify. A LoRA rank of 16 with alpha 32 is the common starting point: rank is how much capacity the adapter has, alpha over rank is the scale it is applied at, and the convention of alpha equal to twice the rank means changing the rank does not silently change the effective learning rate. Targeting all linear layers rather than only the attention projections costs a little more memory and reliably works better. The learning rate of 2e-4 is right for LoRA and roughly twenty times too high for a full fine-tune, because you are training a small number of freshly initialised parameters rather than nudging pretrained ones — carrying a LoRA learning rate into a full fine-tune destroys the model, and the reverse makes nothing happen.

symptom                                 likely cause                  fix
loss drops to near zero in one epoch    dataset too small or repeated  fewer epochs, more data
loss barely moves                       learning rate too low for LoRA raise to 1e-4 - 3e-4
good at the task, worse at everything   too many epochs, rate too high fewer epochs, lower rate,
                                                                       mix in general data
output has stray template markers       chat template applied twice    format once, check a sample
answers echo the user's question        loss not masked                assistant-only loss

Two epochs, not ten#

Supervised fine-tuning on a few thousand examples converges fast, and the commonest mistake after the format ones is training too long. One to three epochs is the working range. Past that the model starts reproducing training examples rather than generalising from them, and the damage shows up somewhere you are not measuring: it answers your task well and has lost the ability to do anything else. Evaluate at the end of each epoch and keep the earliest checkpoint that meets the bar, not the last one.

A thousand good examples beat a hundred thousand mediocre ones#

Fine-tuning copies the style and habits of the data very faithfully, including its mistakes. A small set of carefully written, consistent, correct examples usually beats a large scraped set. Read a random hundred before training.

Faithfully is meant literally, and the consequences are specific. If a tenth of your examples hedge before answering, the model will hedge. If the answers were written by four people and two of them use bullet points, the model will use bullet points about half the time and you will not be able to prompt it out of that reliably. If three examples say the model cannot access the internet and your product gives it a search tool, those three will surface as refusals in production. Consistency matters more than volume because the model is learning the distribution of your answers, including its variance.

• Sort by length and read the extremes. The shortest examples are usually incomplete and the longest are usually two examples that were merged.
• Deduplicate on the prompt, and near-duplicate on the response. Repeated examples are silently equivalent to raising the learning rate on that behaviour.
• Check the answer distribution for what you did not intend to teach: average length, how often it opens with the same phrase, how often it refuses.
• Hold out 50 examples before you start and never train on them. A held-out set assembled after the run is not a held-out set.

Show what changed#

• Run the same eval on the base and tuned model: task success, format adherence, refusals.
• Also check general ability on a few unrelated prompts; heavy fine-tuning can damage it.
• Read twenty outputs side by side. Numbers hide style changes.

The general-ability check is the one that gets dropped and the one that catches catastrophic forgetting, which is what it is called when a narrow fine-tune degrades everything outside its narrow target. You do not need a benchmark harness for it: twenty prompts from unrelated domains, run before and after, read by you. If the tuned model has become worse at arithmetic, or at following an instruction about output format, or has started answering every unrelated question in your product's house style, you have over-trained, and the fix is fewer epochs or a lower rank rather than more data.

Report the comparison as a table with the base model in it. A fine-tune that scores 86 per cent is not a result; a fine-tune that took the base model from 61 to 86 on 50 held-out cases while leaving 20 general prompts unchanged is. The second version is also the one that survives the follow-up question about whether a better prompt would have done the same thing, which is worth answering before you are asked: run the base model with your best prompt as a third column.