Asked about Preference tuning with DPO, hands on
What does the beta parameter in DPO control?
An answer that works
How far the trained model may move from the reference model. Low beta allows the policy to deviate a lot to fit the preferences; high beta keeps it close to the reference, like a stronger KL penalty in RLHF.
Then they push further
After DPO, win rate against the base model is up but responses are 40% longer. Is it better?
Maybe not. Judges and people tend to prefer longer answers, so length alone can explain the win. Compare with length-controlled evaluation and read samples; if the gain disappears, the model learned verbosity.
What gets marked up
Marked up for the reference model's role, beta as a drift control, and suspicion of length-driven wins.
Reading an answer is not being able to give one. The section that teaches this is Preference tuning with DPO, hands on, 4 hours of Alignment & RLHF.
Learn this properlyAlso asked about Alignment & RLHF