Asked about Value learning: Q-learning from scratch

Explain the Q-learning update.

An answer that works

Q of a state and action moves towards the reward received plus the discounted best Q value of the next state. The gap between that target and the current estimate is the temporal difference error, and the learning rate controls how far to move. Repeated over many steps, reward information propagates back to earlier states.

Then they push further

Why is Q-learning called off-policy?

Its target uses the best next action, not the action the agent actually took next. So it learns the value of acting greedily while behaving with exploration. SARSA, which uses the action actually taken, is on-policy.

What do experience replay and target networks fix in DQN?

Replay breaks the correlation between consecutive samples and reuses data. The target network keeps the regression target fixed for a while so the network is not chasing its own changing predictions. Both reduce divergence.

What gets marked up

Strong answers name each term, the TD error, and the off-policy property. Knowing why DQN needed its two tricks marks a candidate as more than a memoriser.

Reading an answer is not being able to give one. The section that teaches this is Value learning: Q-learning from scratch, 5 hours of Reinforcement learning.

Learn this properly

Also asked about Reinforcement learning

What is the credit assignment problem?How does reinforcement learning apply to language models?What is reward hacking?What is the difference between value-based and policy-based methods?Why does PPO limit how much the policy can change?Why is RL less stable than supervised learning?What is the exploration-exploitation trade-off?Our ranker orders items purely by measured click-through rate. What goes wrong?

All 14 questions on Reinforcement learning