Designing a capability evaluation

Run a model once on a dangerous-capability test with a plain prompt, watch it fail, and you have learned almost nothing. This section is how to design an evaluation whose negative result carries weight: choosing a proxy task that measures a necessary skill without causing the harm, climbing an elicitation ladder from bare prompt to tools to many attempts to a fine-tuned variant, and reporting the chance of success in k tries, because a determined misuser gets more than one. The result is a lower bound, and saying which levels you did not try is part of the report.

Proxies, not the real harm#

Evaluations for risky capabilities measure skills that are necessary for a harm without performing the harm. A common example is capture-the-flag security challenges in an isolated environment: they measure skills relevant to cyber misuse, are legal to run, and have unambiguous success criteria.

A proxy task is worth building only if it passes four tests, and it is worth checking them explicitly before writing any code, because a proxy that fails one of them produces a number nobody can act on.

1. 1.Necessity: the skill the task measures is genuinely required for the harm, so a model that cannot do the task cannot do the harm by that route. If it is merely correlated, you have built a benchmark, not a safety case.
2. 2.Gradability: success is checkable by a program, not by a reader's judgement. Capture-the-flag challenges qualify because a flag string either appears or it does not.
3. 3.Safety of running it: the task can be attempted at full effort, repeatedly, without causing the harm. This is what rules out simply asking the model to do the dangerous thing and seeing whether it works.
4. 4.Headroom: current models neither saturate it nor score zero on all of it. A task set where everything scores zero cannot show you an increase next quarter, which is the entire point of running it again.

The fourth is the one that gets skipped and then quietly wastes a quarter of work. Build the set with a spread of difficulty, and record per-task results rather than only the total, so that a model moving from 2 to 9 on the easy third is visible even while the hard third stays at zero.

An elicitation ladder#

Level  Setup                                          Result on 40 tasks
1      plain prompt                                      2
2      + step-by-step instructions and examples          5
3      + tools: shell, file access, in a sandbox         13
4      + 10 attempts per task, best counted              21
5      + a fine-tuned or specialised variant             (if available)

Report every level. The top level reached is the estimate of capability.

Each rung is a different kind of work and a different cost, which is why the ladder is worth planning before you start rather than climbing until you run out of time.

• Rungs one and two are prompt engineering, cheap and quick, and they are also where an evaluation that stops early does most of its damage, because a plain prompt underestimates almost everything.
• Rung three, tools in a sandbox, is the largest single jump in most published results and it is engineering rather than research: a container, a file system, a shell, and a harness that feeds results back. Budget days, not hours.
• Rung four, many attempts, costs linearly in money and needs the statistics in the next part. It is the rung that changes a headline number most and is easiest to justify to a reviewer.
• Rung five, a fine-tuned or specialised variant, is what a well-resourced adversary would do and what most evaluations cannot do, either because the weights are closed or because the team lacks the budget. Say so explicitly rather than omitting the rung.

Stop climbing when a rung stops changing the answer, and record where you stopped. Two consecutive rungs that add nothing is weak evidence that you are near the ceiling of what elicitation buys; one rung that adds nothing is not, because the next one up may be the one that works. The report is a lower bound either way, and the only honest framing of the result is the level you reached and the levels you did not attempt.

Many attempts#

For safety questions, a capability that succeeds once in twenty tries is still a capability, because a determined user will try twenty times. Report the chance of at least one success in k attempts, estimated from n samples per task, alongside single-attempt success.

from math import comb

def pass_at_k(n, c, k):
    """n samples per task, c of them succeeded: chance at least one of k succeeds."""
    if n - c < k:
        return 1.0
    return 1 - comb(n - c, k) / comb(n, k)

for c in (0, 1, 2, 5):
    print(c, [round(pass_at_k(20, c, k), 3) for k in (1, 2, 5, 10, 20)])
# successes out of 20        k=1     k=2     k=5     k=10    k=20
# 0                          0.0     0.0     0.0     0.0     0.0
# 1                          0.05    0.1     0.25    0.5     1.0
# 2                          0.1     0.195   0.447   0.763   1.0
# 5                          0.25    0.447   0.806   0.984   1.0

Read the second row as the thing a safety report has to say. A task the model solves once in twenty attempts has a single-attempt score of 5%, which reads as a failure, and a 50% chance of being solved by somebody who tries ten times, which is what a misuser does. Across forty tasks at that rate, twenty of them fall within ten attempts each. Reporting the 5% alone is not wrong, it is incomplete in the direction that understates risk, and that is the direction a reviewer will push on.

Two practical consequences. Sample generously: n has to be at least as large as the k you want to report, and n of 20 to 50 per task is normal, which is the main cost driver of these evaluations. And decide k from the threat model rather than from convention. A determined individual gets thousands of attempts at a chat interface and one attempt at something that alerts on failure, and those are different numbers.

Size the task set with the same seriousness. A set of 40 tasks on which the model solves 8 gives an observed 20%, with a 95% interval of roughly 9% to 36%, so a rise to 12 out of 40 next quarter is not by itself a finding. Either make the set large enough that the interval is narrower than the change you want to detect, or report the interval every time and compare per-task rather than in aggregate, so that a task that flipped from never to always is visible on its own.

Sandboxes#

Agentic evaluations give the model real tools. Run them in isolated containers with no access to production systems or the open internet beyond what the task needs, log every action, and put a person between the model and anything irreversible. Frameworks such as Inspect provide much of this structure.

Reporting#

• The task set and why it is a good proxy.
• Every elicitation level tried and its result.
• Attempts per task and pass at k.
• What was not tried, so readers know the limits.

Write the sentence a reader will quote, and make sure it is the sentence you meant. 'The model scored 21 of 40 with tools and ten attempts per task, and we did not test a fine-tuned variant' survives being repeated out of context. 'The model could not do X' does not, and it is the version that ends up in a summary slide. The difference is not pedantry: these reports are read by people making deployment decisions, and the only claim your evidence supports is a bounded one.

What to skip: novel task sets when an established one exists and fits the threat model, since comparability across models and across quarters is worth more than novelty; and any attempt to report a single headline capability score, because the number is a function of elicitation effort and a single number invites the reader to forget that.