Alignment & RLHF interview questions
9 questions across 5 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Turning a predictor into an assistant
Read the section that teaches thisIf this comes up in an interview
Why can you not use a pretrained model directly as an assistant?
It continues text rather than answering. A question may be continued with more questions, because that is a plausible continuation. Instruction tuning teaches the question-answer shape, and preference methods refine quality beyond that.
What is the difference between RLHF and DPO?
RLHF trains a separate reward model on human preferences and then optimises the policy against it with reinforcement learning. DPO optimises directly against the preference pairs without a separate reward model, which is simpler and more stable.
Why are models sycophantic?
Preference data rewards answers people like, and people tend to prefer agreement. The training signal therefore encourages agreeing, and separating being helpful from being agreeable is an open problem.
How preference training is done
Read the section that teaches thisIf this comes up in an interview
Why use pairwise comparisons for preference data?
People rank far more consistently than they assign absolute scores, so comparisons produce cleaner training signal. The quality of that data bounds the whole post-training pipeline.
What is the KL penalty in RLHF for?
It penalises the policy for moving too far from the original model. Without it, optimising against the reward model finds exploits: odd or sycophantic outputs that score well and read badly. It keeps the model anchored while it improves.
What is constitutional AI?
A model critiques and revises its own outputs against written principles, producing preference data without human annotators for every comparison. It scales feedback and moves judgement into an explicit, inspectable specification.
Supervised fine-tuning, hands on
Read the section that teaches thisIf this comes up in an interview
Why mask the loss on prompt tokens during supervised fine-tuning?
The goal is to learn how to respond. Including prompt tokens spends capacity learning to generate user messages and system prompts, and can shift the model's behaviour in unwanted ways. Masking focuses learning on the assistant's tokens.
Then they push further
Your fine-tuned model got better at the task and worse at everything else. Why?
Too much training, too high a learning rate or too narrow data caused forgetting. Use fewer epochs, a lower rate or LoRA, and mix in some general instruction data.
What gets marked up. Marked up for loss masking, chat templates, and evaluating both the target task and general ability.
Preference tuning with DPO, hands on
Read the section that teaches thisIf this comes up in an interview
What does the beta parameter in DPO control?
How far the trained model may move from the reference model. Low beta allows the policy to deviate a lot to fit the preferences; high beta keeps it close to the reference, like a stronger KL penalty in RLHF.
Then they push further
After DPO, win rate against the base model is up but responses are 40% longer. Is it better?
Maybe not. Judges and people tend to prefer longer answers, so length alone can explain the win. Compare with length-controlled evaluation and read samples; if the gain disappears, the model learned verbosity.
What gets marked up. Marked up for the reference model's role, beta as a drift control, and suspicion of length-driven wins.
Measuring sycophancy, refusals and calibration
Read the section that teaches thisIf this comes up in an interview
How would you measure whether a model is sycophantic?
Take questions with known answers, keep the ones it answers correctly, push back mildly without new evidence, and measure how often correct answers change to wrong. Also test whether stated user opinions shift its answers on factual questions.
Then they push further
Why does post-training make models sycophantic?
Preference data comes from people, and people tend to rate agreement and flattery highly. Optimising for those ratings rewards agreeing with the user, unless the data and rewards are designed to counter it.
What gets marked up. Marked up for a concrete protocol with a control, and for linking the cause to preference data.
These are the questions for one topic. The whole question bank covers every topic on the site.