Reinforcement learning interview questions
14 questions across 6 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Learning by acting
Read the section that teaches thisIf this comes up in an interview
What is the credit assignment problem?
A reward arrives after a long sequence of actions and it is unclear which actions caused it. It is the central difficulty in reinforcement learning and why it needs far more experience than supervised learning.
Then they push further
How do the two families of method attack it?
Value methods propagate reward backwards one step at a time through the Bellman recursion, so information about the goal reaches earlier states gradually. Policy methods credit every action in an episode with the return that followed it and rely on averaging over many episodes for the coincidences to cancel.
Where is it hardest?
When the reward is a single number at the end of a long sequence, which is exactly the language model case: one score for a response of a thousand tokens. That is why post-training leans so heavily on variance reduction and on limiting how far each update may move the policy.
What gets marked up. Passable answers define it. Strong ones say what each family does about it and name the sparse end-of-episode reward as the hard case.
How does reinforcement learning apply to language models?
Mainly in post-training. A pretrained model predicts plausible text; preference-based methods shape it towards being helpful and harmless, using human judgement as the reward signal rather than an environment score.
What is reward hacking?
Optimising a proxy hard enough that the model finds unintended ways to score well. With human preference data that shows up as confident, agreeable, well-formatted answers being preferred over accurate ones, so reward model quality caps everything built on it.
How the algorithms actually work
Read the section that teaches thisIf this comes up in an interview
What is the difference between value-based and policy-based methods?
Value methods learn how good states or actions are and act greedily with respect to that, suiting discrete actions. Policy methods adjust behaviour directly, handling continuous actions and forming the basis of language model post-training. Actor-critic uses both.
Why does PPO limit how much the policy can change?
Because the policy determines the data you collect next. A bad update in supervised learning is corrected by the next batch; in RL it changes the distribution and can make recovery impossible. Clipping the step is what makes training survivable.
Why is RL less stable than supervised learning?
The data distribution depends on the current policy so nothing is stationary, rewards are sparse and delayed making credit assignment ambiguous, exploration is unsolved, and variance is high enough that results need many seeds.
Then they push further
Put a number on the seed variance.
On the cart-pole run in this topic, identical code at three seeds finished at 293, 500 and 14 steps per episode. The third had reached the 500-step cap earlier in training and collapsed. Any one of those could have been reported as the result of the method.
And on sample efficiency?
On a thirteen-state gridworld, value iteration with the transition probabilities takes 1,872 backups; Q-learning from samples needs around 54,000 environment steps to get within one and a half per cent of the same policy. Roughly thirty times the work on a problem small enough to print, and the gap grows with the state space.
What gets marked up. Strong answers give numbers rather than adjectives, and treat a single learning curve as not being evidence.
The simplest case: bandits and exploration
Read the section that teaches thisIf this comes up in an interview
What is the exploration-exploitation trade-off?
Choosing between the action that looks best so far and trying others that might be better. Exploit too early and you may lock in a worse option; explore too much and you waste reward on options you already know are poor.
Then they push further
How does Thompson sampling handle it?
It keeps a probability distribution over each action's value and acts on a random draw from each. Uncertain actions sometimes draw high and get tried; as evidence accumulates, the distributions narrow and it naturally exploits.
Where would you use a bandit instead of an A/B test?
When the cost of showing the worse option is high and you care about reward during the experiment, such as choosing among many headlines. An A/B test gives cleaner estimates of each option; a bandit gives more reward while learning.
What gets marked up. Marked up for describing a concrete algorithm and knowing when a bandit beats a fixed split test.
Our ranker orders items purely by measured click-through rate. What goes wrong?
New items have no measurements, so they are never shown, so they never acquire any. The ranker is a greedy agent locked onto whatever was popular when it was first switched on, and the catalogue goes stale from the bottom up. The fix is to rank on an optimistic estimate rather than the measured mean, or to sample from the uncertainty, which gives new items a chance proportional to how little is known about them.
Then they push further
How would you size the exploration?
By what it costs. Simulate against your own rates: a small epsilon looks careful and is the worst option early, because the estimates it commits to are noise. An uncertainty-driven method needs no schedule, which is why it is the usual choice when you cannot afford to tune one.
Click rates drift over the year. What breaks?
Every estimator here assumes a fixed rate, so evidence from six months ago counts as much as evidence from today and the agent stops re-checking. Discount old data: an exponentially weighted mean, or decay the Beta counts towards the prior each day.
What gets marked up. Strong answers name the cold-start feedback loop rather than only saying more exploration is needed, and connect it to the locked-on failure measured in this section.
Value learning: Q-learning from scratch
Read the section that teaches thisIf this comes up in an interview
Explain the Q-learning update.
Q of a state and action moves towards the reward received plus the discounted best Q value of the next state. The gap between that target and the current estimate is the temporal difference error, and the learning rate controls how far to move. Repeated over many steps, reward information propagates back to earlier states.
Then they push further
Why is Q-learning called off-policy?
Its target uses the best next action, not the action the agent actually took next. So it learns the value of acting greedily while behaving with exploration. SARSA, which uses the action actually taken, is on-policy.
What do experience replay and target networks fix in DQN?
Replay breaks the correlation between consecutive samples and reuses data. The target network keeps the regression target fixed for a while so the network is not chasing its own changing predictions. Both reduce divergence.
What gets marked up. Strong answers name each term, the TD error, and the off-policy property. Knowing why DQN needed its two tricks marks a candidate as more than a memoriser.
Your agent's average reward is rising. How do you know it has actually learned the task?
You do not, from that alone. On a small environment, compute the exact answer with value iteration and compare the learned policy's true value against the optimum. On a large one, evaluate the greedy policy separately from the training curve, over several seeds, and report the spread rather than the best run. A rising training average is consistent with learning, with an easier distribution of episodes, and with the reward being gamed.
Then they push further
The learned policy differs from the optimal one on two squares. Is that a bug?
Check the value gap before assuming so. On the gridworld in this section, the two disagreeing squares are ones with barely-visited states or near-tied actions, 0.666 against 0.656, and the resulting policy is worth 0.475 against an optimum of 0.482. Matching policies square by square is the wrong test; measuring the policy's value is the right one.
What gets marked up. Marked up for naming an external check rather than a training-curve check, and for knowing that seed spread is part of the result.
Policy gradients: from REINFORCE to PPO
Read the section that teaches thisIf this comes up in an interview
Walk me through the PPO objective.
The ratio compares the new policy's probability of an action with the old policy's. Multiplied by the advantage, it says how much better the new policy is on that data. Clipping the ratio to a band around 1, and taking the minimum of the clipped and unclipped terms, removes the incentive to move the policy far in a single update.
Then they push further
Why is a large policy update so harmful in RL?
The policy decides what data is collected next. A bad update produces bad data, which produces worse updates, and training can collapse without recovering. In supervised learning the data does not depend on the model, so a bad step is corrected by the next batch.
What is the advantage and why use it instead of the return?
The return minus a baseline of expected return from that state. It keeps the gradient direction while greatly reducing its variance, since actions are judged relative to what was expected.
What gets marked up. Strong answers explain ratio, advantage, clip and min, and connect clipping to data depending on the policy.
Every REINFORCE implementation standardises the returns within the episode. What does that actually do?
Mostly it fixes the scale of the update. Measured on a cart-pole policy, standardising cuts the total variance of the gradient estimator by a factor of thousands, but it shrinks the mean gradient by the same factor, so the signal-to-noise ratio barely moves: 0.148 against 0.143 for the raw returns. A state-independent baseline subtracted from the returns does raise it, to 0.222. Standardisation is a step-size fix that is usually described as variance reduction.
Then they push further
Is it even a valid baseline?
Strictly, no. The unbiasedness proof needs a baseline that does not depend on the action taken, and the mean and standard deviation of an episode's own returns do. It is a bias that people accept because the alternative is tuning learning rates against episode length. A critic, which sees only the state, is the unbiased version.
What gets marked up. Strong answers separate variance from signal-to-noise, and know that the popular one-liner is biased.
Reinforcement learning for language models
Read the section that teaches thisIf this comes up in an interview
Explain RLHF and why the KL penalty is there.
Fine-tune on demonstrations, train a reward model on human comparisons, then optimise the model with RL to maximise that reward. The KL penalty keeps the policy close to the starting model, because the reward model is only reliable on text similar to what it was trained on, and far from it the policy finds high-scoring nonsense.
Then they push further
How does DPO differ?
It optimises the preference objective directly on the comparison pairs with a supervised-style loss, with no separate reward model and no sampling during training. Simpler and more stable, but it learns only from the fixed dataset rather than from the current model's own outputs.
Why do verifiable rewards help reasoning?
They are exact: an answer is right or a test passes, so there is no reward model to exploit in the usual way. That allows heavy optimisation with many samples per problem. They still need care, since a checker that can be gamed will be.
What gets marked up. Strong answers cover all three stages, explain the KL term by reward model reliability, and give a concrete reward hacking example.
A preference-training run shows reward climbing steadily. What do you check before shipping it?
Whether the reward is the only thing improving. Track KL from the starting model alongside it: reward rising while KL rises and held-out evaluation is flat or falling is over-optimisation, and in the toy in this section the reader-facing measure peaks at a KL of 0.06 while reward keeps climbing to twice that point. Also compare response length before and after, since length is the best documented rater bias and the cheapest hack to detect, and read a few dozen samples at each checkpoint.
Then they push further
Why does a held-out set help if the reward model was trained on the same preferences?
It has to be held out from the reward model, not just from the policy. An evaluation that shares data or annotators with the reward model will move with it and will not catch the divergence. That is the one evaluation worth keeping isolated.
Does DPO avoid this?
No. It inherits the same preferences, so it inherits the same biases; its beta plays the same role as the KL coefficient. What it avoids is the sampling loop and the separate reward model, not the Goodhart problem.
What gets marked up. Strong answers name KL as a monitored metric rather than a regulariser detail, and give a specific detector such as length.
These are the questions for one topic. The whole question bank covers every topic on the site.