RLHF
Also: reinforcement learning from human feedback
Collecting human comparisons of model outputs, training a reward model to predict those preferences, then optimising the model against that reward. It is how helpfulness and tone get trained when nobody can write the target answer.
What people get wrong
The reward model is a stand-in for human judgement, not human judgement. Push too hard against it and the model learns to exploit its blind spots, which is why length and sycophancy tend to creep up.
A definition is not an understanding. This is taught properly in How preference training is done, part of Alignment & RLHF.
Learn this properlyTerms from the same topic