Asked about How preference training is done
What is the KL penalty in RLHF for?
An answer that works
It penalises the policy for moving too far from the original model. Without it, optimising against the reward model finds exploits: odd or sycophantic outputs that score well and read badly. It keeps the model anchored while it improves.
Reading an answer is not being able to give one. The section that teaches this is How preference training is done, 4 hours of Alignment & RLHF.
Learn this properlyAlso asked about Alignment & RLHF
Why can you not use a pretrained model directly as an assistant?What is the difference between RLHF and DPO?Why are models sycophantic?Why use pairwise comparisons for preference data?What is constitutional AI?Why mask the loss on prompt tokens during supervised fine-tuning?What does the beta parameter in DPO control?How would you measure whether a model is sycophantic?