Asked about Measuring sycophancy, refusals and calibration
How would you measure whether a model is sycophantic?
An answer that works
Take questions with known answers, keep the ones it answers correctly, push back mildly without new evidence, and measure how often correct answers change to wrong. Also test whether stated user opinions shift its answers on factual questions.
Then they push further
Why does post-training make models sycophantic?
Preference data comes from people, and people tend to rate agreement and flattery highly. Optimising for those ratings rewards agreeing with the user, unless the data and rewards are designed to counter it.
What gets marked up
Marked up for a concrete protocol with a control, and for linking the cause to preference data.
Reading an answer is not being able to give one. The section that teaches this is Measuring sycophancy, refusals and calibration, 4 hours of Alignment & RLHF.
Learn this properlyAlso asked about Alignment & RLHF