Asked about Reinforcement learning for language models
A preference-training run shows reward climbing steadily. What do you check before shipping it?
An answer that works
Whether the reward is the only thing improving. Track KL from the starting model alongside it: reward rising while KL rises and held-out evaluation is flat or falling is over-optimisation, and in the toy in this section the reader-facing measure peaks at a KL of 0.06 while reward keeps climbing to twice that point. Also compare response length before and after, since length is the best documented rater bias and the cheapest hack to detect, and read a few dozen samples at each checkpoint.
Then they push further
Why does a held-out set help if the reward model was trained on the same preferences?
It has to be held out from the reward model, not just from the policy. An evaluation that shares data or annotators with the reward model will move with it and will not catch the divergence. That is the one evaluation worth keeping isolated.
Does DPO avoid this?
No. It inherits the same preferences, so it inherits the same biases; its beta plays the same role as the KL coefficient. What it avoids is the sampling loop and the separate reward model, not the Goodhart problem.
What gets marked up
Strong answers name KL as a monitored metric rather than a regulariser detail, and give a specific detector such as length.
Reading an answer is not being able to give one. The section that teaches this is Reinforcement learning for language models, 6 hours of Reinforcement learning.
Learn this properlyAlso asked about Reinforcement learning