Asked about Value learning: Q-learning from scratch

Your agent's average reward is rising. How do you know it has actually learned the task?

An answer that works

You do not, from that alone. On a small environment, compute the exact answer with value iteration and compare the learned policy's true value against the optimum. On a large one, evaluate the greedy policy separately from the training curve, over several seeds, and report the spread rather than the best run. A rising training average is consistent with learning, with an easier distribution of episodes, and with the reward being gamed.

Then they push further

The learned policy differs from the optimal one on two squares. Is that a bug?

Check the value gap before assuming so. On the gridworld in this section, the two disagreeing squares are ones with barely-visited states or near-tied actions, 0.666 against 0.656, and the resulting policy is worth 0.475 against an optimum of 0.482. Matching policies square by square is the wrong test; measuring the policy's value is the right one.

What gets marked up

Marked up for naming an external check rather than a training-curve check, and for knowing that seed spread is part of the result.

Reading an answer is not being able to give one. The section that teaches this is Value learning: Q-learning from scratch, 5 hours of Reinforcement learning.

Learn this properly

Also asked about Reinforcement learning

What is the credit assignment problem?How does reinforcement learning apply to language models?What is reward hacking?What is the difference between value-based and policy-based methods?Why does PPO limit how much the policy can change?Why is RL less stable than supervised learning?What is the exploration-exploitation trade-off?Our ranker orders items purely by measured click-through rate. What goes wrong?
All 14 questions on Reinforcement learning