Asked about Reinforcement learning for language models
Explain RLHF and why the KL penalty is there.
An answer that works
Fine-tune on demonstrations, train a reward model on human comparisons, then optimise the model with RL to maximise that reward. The KL penalty keeps the policy close to the starting model, because the reward model is only reliable on text similar to what it was trained on, and far from it the policy finds high-scoring nonsense.
Then they push further
How does DPO differ?
It optimises the preference objective directly on the comparison pairs with a supervised-style loss, with no separate reward model and no sampling during training. Simpler and more stable, but it learns only from the fixed dataset rather than from the current model's own outputs.
Why do verifiable rewards help reasoning?
They are exact: an answer is right or a test passes, so there is no reward model to exploit in the usual way. That allows heavy optimisation with many samples per problem. They still need care, since a checker that can be gamed will be.
What gets marked up
Strong answers cover all three stages, explain the KL term by reward model reliability, and give a concrete reward hacking example.
Reading an answer is not being able to give one. The section that teaches this is Reinforcement learning for language models, 6 hours of Reinforcement learning.
Learn this properlyAlso asked about Reinforcement learning