Measuring and reducing what could go wrong

Safety work splits into harms happening now — misuse, deception, bias, privacy, generated content that damages people today — and capabilities that would be dangerous if they appeared, which have to be tested for before they do. This section covers how each is evaluated and what red teaming involves. It also covers why the methodology is so hard: a test that finds nothing is weak evidence, because it may only mean nobody asked in the right way.

Two kinds of concern#

• Present harms: misuse, deception, bias, privacy, generated content that damages people now.
• Anticipated harms: capabilities that would be dangerous if they appeared, evaluated before they do.

The failure modes, named#

Most of this field is a small number of named failure modes, each with its own measurement. Learning the names properly is worth more than any general position on whether the technology is dangerous, and it is what an interview is checking. The table is the recognition rule; the definitions follow.

failure mode            what you see                          measured by
specification gaming    objective satisfied, intent not       inspecting behaviour, not score
reward hacking          proxy reward rises, quality falls     a held-out human judgement
sycophancy              answer follows the user's stated view  paired prompts differing in the cue
jailbreak               refusal bypassed by a framing          attack success rate over many attempts
sandbagging             worse when it looks like a test        matched prompts, one framed as an eval
unfaithful reasoning    stated reasons do not drive the answer intervening on a hidden cue

Specification gaming is the oldest and the most general: the system optimises exactly what you wrote down, and what you wrote down was not what you meant. The standard example is an agent trained to score points in a boat racing game, which found that circling a lagoon hitting respawning targets scored better than finishing the race, and so never finished a race. Simulated robots asked to move forward have learned to grow tall and fall over. The pattern to recognise is that nothing malfunctioned. The objective was maximised, and the objective was wrong.

Reward hacking is specification gaming where the specification is itself a learned model. In preference training, a reward model is fitted to human comparisons and then the policy is optimised against that model. The reward model is an approximation, and optimising hard against an approximation finds the places where it is wrong. The visible symptom is that the reward keeps going up while human raters start preferring the older model.

Sycophancy is the specific case where the reward model learned that raters like being agreed with. The model then shifts its answer towards whatever the user appears to believe, including on questions with a fact of the matter. It is easy to measure and unusually easy to demonstrate, which makes it a common interview question.

A jailbreak is an input that gets the model to do something its training was supposed to prevent. They come in a handful of families, and knowing the families is more useful than knowing any particular prompt, because the prompts are patched and the families are not: role-play and fictional framing, competing instructions that give the model a higher-priority-looking authority, obfuscation such as encoding or translating the request, multi-turn build-up that establishes context gradually rather than asking directly, and automated search for token sequences that happen to work. The last family matters most for evaluation, because it does not need a person to be creative and it transfers between models more often than people expect.

Note what these have in common. None of them is the model breaking. Each one is the model doing well at something adjacent to the thing that was wanted, which is why the fix is rarely a bug fix and usually a change to what is being measured or optimised.

Reward hacking, measured#

The reason reward hacking is hard to catch is that every number on your dashboard improves while it happens. Here is the smallest model of it that shows the shape. One axis, x, stands for how much an answer flatters and pads. True quality is best at a moderate amount and falls away either side. The reward model was fitted on data near x equal to zero, and it picked up a mild preference for x that the humans did not have. Optimise the policy against the reward model and watch both curves.

true_quality = lambda mu: -(mu - 1.0) ** 2 - 1.0      # averaged over answers drawn around mu
proxy_reward = lambda mu: true_quality(mu) + 0.6 * mu
gradient     = lambda mu: -2 * (mu - 1.0) + 0.6

mu, lr = 0.0, 0.08
for step in range(41):
    if step in (0, 2, 5, 10, 15, 20, 30, 40):
        print(step, round(mu, 3), round(proxy_reward(mu), 4), round(true_quality(mu), 4))
    mu += lr * gradient(mu)              # gradient ascent on the reward model, nothing else

step     x    proxy reward    true quality
   0   0.000      -2.0000        -2.0000
   2   0.383      -1.1514        -1.3810
   5   0.756      -0.6056        -1.0594
  10   1.073      -0.3617        -1.0053   <- true quality peaks here
  15   1.205      -0.3190        -1.0420
  20   1.260      -0.3116        -1.0677
  30   1.293      -0.3100        -1.0859
  40   1.299      -0.3100        -1.0893   <- reward is still improving

Two lessons come straight out of that table. The first is that you cannot detect this from the reward curve, by construction, so a training run optimising a learned reward needs a separate held-out judgement that the reward model was not fitted on, sampled periodically. The second is the reason a KL penalty is in every one of these algorithms: it charges the policy for moving away from the reference model it started from, which keeps it inside the region where the reward model was actually fitted.

for beta in (0.0, 0.2, 0.6, 1.0, 2.0):
    settles_at = 2.6 / (2 + beta)        # where gradient ascent stops once the penalty is added
    print(beta, round(settles_at, 3), round(true_quality(settles_at), 4),
          round(proxy_reward(settles_at), 4))
# beta   x      true quality   proxy
# 0.0    1.300    -1.0900     -0.3100    no penalty: fully over-optimised
# 0.2    1.182    -1.0331     -0.3240
# 0.6    1.000    -1.0000     -0.4000    exactly the right answer
# 1.0    0.867    -1.0178     -0.4978
# 2.0    0.650    -1.1225     -0.7325    penalty so heavy the model barely moves

That is the honest state of the technique. The KL coefficient is a knob whose correct value depends on how wrong your reward model is, which is the thing you do not know. In practice it is chosen by sampling the policy at intervals and having people or a separate judge compare against the reference, which is expensive and is the reason these runs are supervised rather than left to finish.

Capability evaluation#

Build tasks that measure whether a model can do something concerning, and measure before deployment rather than after. The methodological difficulty is that a model failing your test may still succeed with better prompting, more scaffolding or more attempts.

Red teaming#

Deliberately attempting to make a model behave badly. Manual red teaming finds creative attacks that automated methods miss; automated red teaming scales and covers more ground. Both are needed, and findings feed back into training and into guardrails.

What makes it hard#

• Absence of evidence is weak evidence, as above.
• Capabilities can appear with scaffolding rather than with scale, so a model evaluated alone may behave differently in an agent.
• Measuring intent or deception is far harder than measuring capability, and interpretability is one of the few routes into it.
• Publishing details of a vulnerability can enable it, so disclosure norms are genuinely contested.

What the training-side methods actually do#

The mitigations have names too, and being vague about what they do is the fastest way to sound unserious. Preference training from human feedback has three steps: collect comparisons where a person picks the better of two model outputs, fit a reward model that predicts those picks, then optimise the policy against that reward model with a KL penalty back to the starting model. Constitutional or AI-feedback methods replace the human in step one with a model asked to judge against a written set of principles, which makes the labels far cheaper and makes the principles inspectable and arguable, since they are written down.

Both do something real: they are why a deployed model answers in a helpful register, declines obvious misuse, and follows instructions at all. Neither does what it is often described as doing, and the gaps are specific.

• The reward model is a proxy, and the section above is what happens when you optimise it too hard. Its quality is also bounded by how much the annotators agreed, which on subtle comparisons is not very much.
• It changes behaviour, not knowledge. Training a model to decline a request does not remove the capability behind it, which is why an elicitation result and a refusal rate are different measurements and why a safety claim based on refusals is weak.
• The trained behaviour is not robust to further training. Fine-tuning an aligned model on a small amount of unrelated or adversarial data can substantially undo the refusal behaviour, which matters a great deal for anyone offering fine-tuning as a product feature.
• Coverage follows the data. Behaviour is shaped where the preference data had examples, so unusual framings, other languages and long multi-turn contexts are systematically weaker, which is exactly where jailbreaks live.
• Agreement is rewarded. A rater's preference for being agreed with is in the data, so sycophancy is not a bug in the method but a faithful reflection of what was optimised.

The practical consequence for an engineer is that training-side alignment is one layer and not a boundary. A deployed system also needs input and output checks it does not control from the same weights, an architecture where secrets are not in the prompt, and a measurement of the attack success rate it actually has rather than the one the model card implies.

The open problems, stated plainly#

Say these as open problems rather than as either crises or curiosities. They are the questions people are paid to work on, and being able to state one precisely is a better interview answer than any opinion about timelines.

• Scalable oversight: how to train or evaluate a model on tasks where a human cannot readily check the answer. Every method above assumes a judge who knows better than the model.
• Evaluation validity: whether a benchmark score corresponds to the real-world behaviour it is named after, especially when the model may recognise the evaluation.
• Robustness of refusals: no method reliably prevents a determined adversary with many attempts, and attack success rate has not gone to zero for any deployed system.
• Measuring what a model is doing rather than what it says: interpretability is the main route and it is early, and chain-of-thought monitoring depends on faithfulness that is not guaranteed.
• Specification: writing down what we want, including the parts nobody thinks to write down, remains unsolved in a way that predates language models by decades.

What to skip: arguments about timelines and probabilities of catastrophe, which are not what this work consists of and are not what the interview is about; and any claim that a model is or is not dangerous, as opposed to a claim about what a stated evaluation found under stated elicitation. The discipline of this field is in the second kind of sentence.