Learning by acting

Supervised learning needs a correct answer for every example. Reinforcement learning does not: an agent acts, receives a score, and has to work out for itself which of its actions earned it. This section covers the setup and the two hard problems inside it — exploring enough to find good behaviour at all, and assigning credit when the reward arrives long after the action that deserved it. It is here rather than in a robotics course because training a language model from human preferences is reinforcement learning, and the alignment literature assumes you have this vocabulary.

The setup#

An agent observes a state, takes an action, receives a reward and moves to a new state. It learns a policy, a mapping from states to actions, that maximises reward over time. Unlike supervised learning there is no correct answer given, only a score.

Every one of those words is used in a specific way and the papers never stop to define them, so here they are against a concrete example. A shop wants to decide which of three versions of its checkout page to show each visitor.

word          what it means                          in the checkout example

agent         the thing choosing                    your ranking service
environment   everything else, which answers back    the visitor and the shop
state         what the agent can see right now       the visitor's country and device
action        one of the choices available           show page A, B or C
reward        a number scoring what just happened    1 if they bought, 0 if not
policy        the rule from state to action          "on mobile, show B"
episode       one run from start to finish           one visit, from landing to leaving
return        the rewards of an episode added up,    total purchases from that visit,
              with later ones discounted             counting today more than next month
value         the return you expect from a state     what a mobile visitor is worth
              if you follow the policy

The policy is the thing being learned and the only thing you ship. Everything else — values, returns, advantages — exists to work out how the policy should change. If you remember one sentence from this section, make it that.

Why this is not supervised learning with extra steps#

It is tempting to read the setup as classification with a strange loss. It is not, and three differences are the reason the field has its own algorithms.

• There is no label. Nobody says which action was correct, only what score followed the one you took. You never find out what the other actions would have paid.
• Consequences are delayed. The reward that arrives now may have been earned twenty actions ago, and working out which one earned it is the credit assignment problem.
• Your own behaviour produces your data. A supervised dataset sits still. A policy that never turns right never sees what is to the right, so it cannot learn that the right-hand path was better.

The third one is the strangest and the easiest to underrate, so here it is measured. A corridor of nine squares, starting in the middle. Falling off the left end pays 0.1, falling off the right end pays 1.0, and the policy is a single number: the probability of stepping right.

import random

CORRIDOR = 9                       # squares 0 to 8; both ends end the episode
START, SMALL, BIG = 4, 0.1, 1.0

def walk(p_right, rng, cap=500):
    s, visited = START, []
    for _ in range(cap):
        visited.append(s)
        s += 1 if rng.random() < p_right else -1
        if s <= 0:
            return visited, SMALL
        if s >= CORRIDOR - 1:
            return visited, BIG
    return visited, 0.0

for p_right in (0.3, 0.5, 0.7):
    rng = random.Random(0)
    seen, big = [0] * CORRIDOR, 0
    for _ in range(2000):
        visited, r = walk(p_right, rng)
        for s in visited:
            seen[s] += 1
        big += (r == BIG)
    print(f"p(right)={p_right}: found the big reward in {100*big/2000:5.1f}% of episodes  visits {seen}")
# p(right)=0.3: found the big reward in   3.8% of episodes  visits [0, 2808, 4003, 4389, 4523, 1795, 691, 225, 0]
# p(right)=0.5: found the big reward in  48.6% of episodes  visits [0, 2119, 4084, 5961, 7975, 6013, 4031, 1997, 0]
# p(right)=0.7: found the big reward in  96.4% of episodes  visits [0, 229, 736, 1975, 4709, 4416, 3865, 2690, 0]

The agent that leans left is not merely slower. It collects almost no data about the half of the corridor where the good reward lives, and the data it does collect keeps telling it that left pays 0.1, which is better than the zero it has estimated for right. A supervised model with a bad initialisation still sees the whole training set on the next epoch. A policy with a bad initialisation may never see the part of the world that would correct it. This is why exploration is a first-class problem here and barely a topic anywhere else.

Return, and what the discount actually decides#

Rewards are added up over an episode, with later ones multiplied by a discount factor gamma raised to the number of steps away they are. Gamma is usually described as how far-sighted the agent is, which is true but vague. What it concretely does is set the exchange rate between a reward now and a larger reward later, and you can read that rate off directly.

now   = [1.0, 0, 0, 0, 0, 0]        # one point, straight away
later = [0, 0, 0, 0, 0, 10.0]       # ten points, five steps from now

def ret(rewards, gamma):
    return sum(r * gamma ** t for t, r in enumerate(rewards))

for gamma in (0.0, 0.5, 0.631, 0.9, 0.99):
    print(f"gamma {gamma:5.3f}:  take it now {ret(now, gamma):6.3f}   wait five steps {ret(later, gamma):6.3f}")
print("crossover at gamma =", round((1 / 10) ** (1 / 5), 3))
# gamma 0.000:  take it now  1.000   wait five steps  0.000
# gamma 0.500:  take it now  1.000   wait five steps  0.312
# gamma 0.631:  take it now  1.000   wait five steps  1.000
# gamma 0.900:  take it now  1.000   wait five steps  5.905
# gamma 0.990:  take it now  1.000   wait five steps  9.510
# crossover at gamma = 0.631

So gamma is part of the problem specification, not a training detail. Set it low and the agent is provably right to be short-sighted, and no amount of training will fix behaviour you have defined as optimal. Set it near 1 and the horizon is long, values grow towards the sum of everything, and learning gets slower and noisier because a single early action is now credited with thousands of later rewards. 0.99 is the usual default for episodes of a few hundred steps, which values a reward a hundred steps away at about a third of one received now.

The two hard problems#

• Credit assignment: a reward arrives after a long sequence of actions. Which of them was responsible? This is the central difficulty.
• Exploration versus exploitation: repeat what worked, or try something new that might work better. Too much of either fails.

Policy gradients#

Rather than learning the value of each state, directly adjust the policy to make high-reward actions more likely. PPO is the widely used variant, and its key idea is limiting how far the policy can move in one update, because large policy changes destabilise training.

The reason the limit matters is the third difference above. In supervised learning a step that overshoots makes the loss worse and the next batch pulls it back, because the data is the same data either way. In reinforcement learning a step that overshoots changes which states the agent visits, so the next batch is drawn from a worse part of the world, and the correction never arrives. PPO's clip is a crude guard against exactly that, and the policy-gradients section of this topic builds it from REINFORCE upwards with the numbers measured.

Why this matters for language models#

The largest practical impact is post-training. A pretrained model produces plausible text; making it helpful and harmless is done by learning from preferences, which is reinforcement learning with the reward derived from human judgement rather than a game score.

The mapping is exact rather than metaphorical. The state is the prompt plus the tokens generated so far, the action is the next token, the policy is the model, one episode is one response, and the reward is a single score for the finished response — which makes it the sparse, delayed, end-of-episode case, the hardest kind. Sequence-level rewards are why post-training needs the variance machinery at all. How the score itself is produced, and what goes wrong when it is a learned stand-in for human judgement, is the alignment topic's subject; this topic supplies the algorithms it assumes.

What to skip if you are an AI engineer rather than a researcher: the classical control literature. Dynamic programming proofs, convergence conditions, continuous-control benchmarks, the Atari results and the exploration-bonus papers are a large and interesting body of work you will not use. Learn the vocabulary on this page, the bandit section because it is genuinely useful for ranking and experimentation, and the section on language models because it is what alignment work is built from. Q-learning and policy gradients are worth doing once with your own hands so that PPO is not a magic word, and then left alone.