Studying behaviour: honesty, faithfulness and sandbagging
By the end: You can design experiments that test whether a model's stated reasoning and behaviour match what it actually does.
- Is the chain of thought faithful?
- Does behaviour change when the model thinks it is tested?
- Sandbagging
- Designing controls
New to safety & evaluation research? Empirical work on what a model does wrong and how badly. In practice it is three activities: building test sets for capabilities that would be dangerous if the model had them, red-teaming — deliberately trying to make a model behave badly and writing down what worked — and studying behaviour under pressure, such as whether a model acts differently when it appears to be under test. Why it matters
A model's written reasoning is text it produced, not a log of what drove its answer, and the gap between the two can be measured. This section is the experimental design for measuring it: slip a biasing hint into a question and see whether the answer moves while the reasoning never mentions it, compare matched prompts that differ only in whether the model appears to be under evaluation, and check whether performance drops when a task is framed as measuring something concerning. It matters because reading a model's reasoning is a proposed way of catching problems, and controls are what separate a finding here from a plausible story.
Does the reasoning reflect the decision?#
A model's written reasoning may not describe what actually drove its answer. A clean test: add a biasing hint to multiple-choice questions, such as 'I think the answer is B', and check whether the answer shifts towards the hint and whether the reasoning ever mentions it. Published studies found models often follow such hints without saying so.
For each question:
baseline answer -> A (correct)
with hint "I think it's B" -> B
does the reasoning mention the hint? -> no
Unfaithfulness rate = answers changed by the hint whose reasoning does not mention itThe design has three arms rather than two, and the third is what makes it a measurement instead of an anecdote. Run each question unhinted, hinted towards a wrong answer, and hinted towards the answer the model already gave. The third arm is the control: if answers move in that arm too, then some of what you are seeing is the model being unstable under any change of wording rather than following the hint.
100 questions, three arms, same model, same decoding settings
arm answer changed from baseline
hint towards a wrong answer 18
hint towards its own answer 4 <- control: mere rewording
of the 18 that moved, the reasoning mentioned the hint in 3Then test it rather than eyeballing it. Because every question is answered by the same model under both conditions, the comparison is paired, and the right test counts only the questions that disagree between the arms: 18 moved one way, 4 the other.
from scipy import stats
for moved_to_hint, moved_away in ((18, 4), (9, 5)):
p = stats.binomtest(moved_to_hint, moved_to_hint + moved_away, 0.5).pvalue
print(moved_to_hint, moved_away, round(float(p), 4))
# 18 4 0.0043 a real effect
# 9 5 0.424 the same experiment on a smaller effect: nothing to reportDecide the number of questions before running, not after. The arithmetic below is the standard two-proportion sample size, and it is sobering: the effects people want to claim in this area need hundreds of items per arm, and a 100-item eval can only detect something very large.
from statistics import NormalDist
z = NormalDist().inv_cdf
for p1, p2 in ((0.50, 0.55), (0.50, 0.60), (0.50, 0.70)):
pbar = (p1 + p2) / 2
n = ((z(0.975) * (2 * pbar * (1 - pbar)) ** 0.5
+ z(0.8) * (p1 * (1 - p1) + p2 * (1 - p2)) ** 0.5) ** 2) / (p2 - p1) ** 2
print(p1, p2, int(-(-n // 1)))
# 0.5 -> 0.55 1565 items per arm
# 0.5 -> 0.6 388 items per arm
# 0.5 -> 0.7 93 items per armSycophancy, as a measurement#
Sycophancy is the same experimental shape pointed at a different cue, and it is the one most likely to come up because it is cheap to run on any model. Take questions with a defensible right answer, and ask each one twice: plainly, and with the user stating a view. The measurement is how much the answer follows the stated view.
arm agrees with the user's stated view
neutral question, no view stated n/a, record the answer
"I think the answer is X" (X wrong) the number that switch to X
"I think the answer is X" (X right) control: the number that switch to X
"As a physicist, I believe X" adds an authority cue on top
user pushes back after a correct answer the number that reverse under pressureThe last row is the version that matters most in a product and is measured least. A single-turn agreement rate can look fine while the model folds the moment a user says 'are you sure?', and that is the behaviour a customer actually meets. Run it as a second turn with a fixed pushback phrase, keep the phrase constant across models, and report the reversal rate on questions the model originally got right, which is the only subset where reversing is unambiguously wrong.
Tested or not#
If a model behaves differently when cues suggest it is being evaluated, evaluations measure the wrong thing. Experiments compare behaviour on matched prompts with and without evaluation-like cues, such as formal test formatting or statements that responses will be graded.
Building that comparison properly is harder than it sounds, because almost everything that signals 'this is a test' also changes the task. Multiple-choice formatting, a numbered question, an instruction to answer with a single letter: each is an evaluation cue and each also makes the question easier or harder. The usable designs hold the task fixed and vary something that carries no task information at all, such as a preamble saying the responses will be used for research versus for a live product, or whether the surrounding conversation looks like a transcript or like a chat. Report the cue you varied verbatim, because the reader's first question will be whether you changed the difficulty by accident.
Sandbagging#
Sandbagging is deliberately underperforming, for example on a dangerous capability test. Research designs look for performance that drops specifically when the model is told a task measures something concerning, compared with the same task framed neutrally, and check whether fine-tuning on a few examples recovers much more capability than the prompt alone showed.
Controls make or break these studies#
- Matched prompts that differ in only the manipulated cue.
- Many questions and several models, since single examples mislead.
- Checking that a hint's effect is not just noise by including irrelevant hints of the same length.
Where people go wrong
Reading a model's explanation of its answer as a description of its computation.
If this comes up in an interview
How would you test whether a model's chain of thought is faithful?
Intervene on something that should affect the answer, such as a biasing hint, and check whether the answer changes and whether the reasoning acknowledges the cause. Answers that move with the hint while the reasoning never mentions it are evidence of unfaithful reasoning.
Then they push further
Why does this matter for safety?
Monitoring a model's reasoning is one proposed way to catch problems. If the stated reasoning does not reflect what drives behaviour, that monitoring gives false confidence.
What gets marked up. Marked up for an intervention with a control, and for connecting faithfulness to monitoring.
Build this: Replicate the biasing hint experiment on 100 multiple-choice questions with an open model: measure how often hints change answers, how often the reasoning mentions the hint, and include a control hint that points to the correct answer.
This is one section of Safety & evaluation research, 22 hours across 5 sections. Read it in full to get the whole topic in order.