Asked about Turning a predictor into an assistant
What is the difference between RLHF and DPO?
An answer that works
RLHF trains a separate reward model on human preferences and then optimises the policy against it with reinforcement learning. DPO optimises directly against the preference pairs without a separate reward model, which is simpler and more stable.
Reading an answer is not being able to give one. The section that teaches this is Turning a predictor into an assistant, 5 hours of Alignment & RLHF.
Learn this properlyAlso asked about Alignment & RLHF
Why can you not use a pretrained model directly as an assistant?Why are models sycophantic?Why use pairwise comparisons for preference data?What is the KL penalty in RLHF for?What is constitutional AI?Why mask the loss on prompt tokens during supervised fine-tuning?What does the beta parameter in DPO control?How would you measure whether a model is sycophantic?