Running a safety evaluation

A safety evaluation usually reports that a model could not do something, and that claim is only as strong as the effort behind it. This section is about designing one whose negative result carries weight: recording what was tried and how many times, testing the model with the tools and scaffolding it will actually have in deployment, and using automated attacks for coverage alongside people for the attacks a model would not think of. It ends on disclosure, which is a judgement call rather than a technique, and is asked about for exactly that reason.

State how hard you tried#

Since a negative result only says you failed to elicit a behaviour, the elicitation effort is part of the finding rather than a footnote. An evaluation that reports a failure without saying what was attempted is not informative.

Weak:    "The model could not perform task X."

Usable:  "Across 200 attempts with chain-of-thought prompting,
          tool access, and a fine-tuned variant, the model
          completed task X in 0 of 200. Elicitation methods
          tried are listed in appendix B."

How much a zero is worth is a calculation, not a judgement, and it is the single most useful piece of statistics in this work. If each attempt succeeds independently with probability p, then seeing no successes in n attempts puts a one-sided 95% upper bound on p of one minus 0.05 to the power of one over n.

for n in (40, 100, 200, 1000):
    print(n, round(1 - 0.05 ** (1 / n), 4))
# 40    0.0722     zero successes is consistent with a 7% success rate
# 100   0.0295
# 200   0.0149
# 1000  0.0030

Read that against the thing being claimed. Forty attempts and no successes is consistent with a true rate of 7%, which for a capability a motivated person will attempt hundreds of times is not reassurance at all. If the claim you want to support is that the rate is under 1%, you need on the order of three hundred attempts before a clean zero says so, and that is before you account for the attempts not being independent, which they are not, because one prompt template is correlated with the next.

The same arithmetic disciplines a positive result. An observed rate needs an interval, and at the sample sizes people actually run, the interval is wide.

from scipy import stats

for successes, trials in ((2, 200), (3, 200), (12, 200), (12, 2000)):
    lo = stats.beta.ppf(0.025, successes, trials - successes + 1)
    hi = stats.beta.ppf(0.975, successes + 1, trials - successes)
    print(f"{successes}/{trials} = {successes/trials:.2%}  95% CI [{lo:.2%}, {hi:.2%}]")
# 2/200   = 1.00%  95% CI [0.12%, 3.57%]
# 3/200   = 1.50%  95% CI [0.31%, 4.32%]
# 12/200  = 6.00%  95% CI [3.14%, 10.25%]
# 12/2000 = 0.60%  95% CI [0.31%, 1.05%]

This is the arithmetic behind most of the disagreements about safety evaluations. A model card reporting a 1% attack success rate from 200 attempts is reporting something compatible with 0.1% and with 3.6%, and a comparison between two models at that sample size is usually noise. Either run enough attempts for the interval to be narrower than the difference you want to claim, or report the interval and decline to claim the difference.

Capability or propensity#

Two questions get run together and they need different evaluations. A capability evaluation asks whether the model can do something when you try hard to make it: you prompt for it, give it tools, retry, and fine-tune if you are allowed to. A propensity evaluation asks whether it does the thing unprompted, in a setting resembling ordinary use, when nobody is pushing. A model can score high on one and low on the other in either direction, and the mitigations differ: capability is addressed by not deploying, by removing access, or by filtering at the boundary; propensity is addressed by training.

Say which you measured. A great deal of confused writing in this area comes from a capability result being reported as though it were a propensity result, or from a low refusal rate being offered as evidence about capability. Refusals measure propensity under the prompts you used, and nothing at all about what the model could do.

Scaffolding is capability#

Automated red teaming#

Use a model to generate adversarial inputs against another, scoring outcomes automatically. It scales enormously and finds a narrower range of attacks than people do, because it explores from what it has seen. The practical answer is both: automated for coverage, manual for creativity.

Disclosure#

Publishing a working attack helps defenders and helps attackers. Norms vary and the tension is real. The common practice is notifying the developer, allowing time to mitigate, then publishing the class of problem rather than a working exploit.

Being able to discuss this thoughtfully matters in safety-focused interviews, because it is a judgement question rather than a technical one and the answer reveals how you weigh competing goods.