Asked about Policy gradients: from REINFORCE to PPO

Every REINFORCE implementation standardises the returns within the episode. What does that actually do?

An answer that works

Mostly it fixes the scale of the update. Measured on a cart-pole policy, standardising cuts the total variance of the gradient estimator by a factor of thousands, but it shrinks the mean gradient by the same factor, so the signal-to-noise ratio barely moves: 0.148 against 0.143 for the raw returns. A state-independent baseline subtracted from the returns does raise it, to 0.222. Standardisation is a step-size fix that is usually described as variance reduction.

Then they push further

Is it even a valid baseline?

Strictly, no. The unbiasedness proof needs a baseline that does not depend on the action taken, and the mean and standard deviation of an episode's own returns do. It is a bias that people accept because the alternative is tuning learning rates against episode length. A critic, which sees only the state, is the unbiased version.

What gets marked up

Strong answers separate variance from signal-to-noise, and know that the popular one-liner is biased.

Reading an answer is not being able to give one. The section that teaches this is Policy gradients: from REINFORCE to PPO, 6 hours of Reinforcement learning.

Learn this properly

Also asked about Reinforcement learning

What is the credit assignment problem?How does reinforcement learning apply to language models?What is reward hacking?What is the difference between value-based and policy-based methods?Why does PPO limit how much the policy can change?Why is RL less stable than supervised learning?What is the exploration-exploitation trade-off?Our ranker orders items purely by measured click-through rate. What goes wrong?
All 14 questions on Reinforcement learning