Asked about Learning by acting

What is reward hacking?

An answer that works

Optimising a proxy hard enough that the model finds unintended ways to score well. With human preference data that shows up as confident, agreeable, well-formatted answers being preferred over accurate ones, so reward model quality caps everything built on it.

Reading an answer is not being able to give one. The section that teaches this is Learning by acting, 4 hours of Reinforcement learning.

Learn this properly

Also asked about Reinforcement learning

What is the credit assignment problem?How does reinforcement learning apply to language models?What is the difference between value-based and policy-based methods?Why does PPO limit how much the policy can change?Why is RL less stable than supervised learning?What is the exploration-exploitation trade-off?Our ranker orders items purely by measured click-through rate. What goes wrong?Explain the Q-learning update.

All 14 questions on Reinforcement learning