Asked about How the algorithms actually work

Why does PPO limit how much the policy can change?

An answer that works

Because the policy determines the data you collect next. A bad update in supervised learning is corrected by the next batch; in RL it changes the distribution and can make recovery impossible. Clipping the step is what makes training survivable.

Reading an answer is not being able to give one. The section that teaches this is How the algorithms actually work, 4 hours of Reinforcement learning.

Learn this properly

Also asked about Reinforcement learning

What is the credit assignment problem?How does reinforcement learning apply to language models?What is reward hacking?What is the difference between value-based and policy-based methods?Why is RL less stable than supervised learning?What is the exploration-exploitation trade-off?Our ranker orders items purely by measured click-through rate. What goes wrong?Explain the Q-learning update.

All 14 questions on Reinforcement learning