Asked about Turning a predictor into an assistant
Why are models sycophantic?
An answer that works
Preference data rewards answers people like, and people tend to prefer agreement. The training signal therefore encourages agreeing, and separating being helpful from being agreeable is an open problem.
Reading an answer is not being able to give one. The section that teaches this is Turning a predictor into an assistant, 5 hours of Alignment & RLHF.
Learn this properlyAlso asked about Alignment & RLHF
Why can you not use a pretrained model directly as an assistant?What is the difference between RLHF and DPO?Why use pairwise comparisons for preference data?What is the KL penalty in RLHF for?What is constitutional AI?Why mask the loss on prompt tokens during supervised fine-tuning?What does the beta parameter in DPO control?How would you measure whether a model is sycophantic?