Safety & evaluation research interview questions
9 questions across 5 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Measuring and reducing what could go wrong
Read the section that teaches thisIf this comes up in an interview
How do you evaluate whether a model has a dangerous capability?
Build concrete tasks that require it and try hard to elicit the behaviour, with scaffolding and multiple attempts, then report the elicitation effort alongside the result. A negative result only means you failed to elicit it.
Why is a negative safety result weak evidence?
Because failure to elicit is not absence of capability. Better prompting, tool access or more attempts may succeed where your evaluation did not, so the claim is always bounded by how hard you tried.
What is red teaming?
Deliberately trying to make a model behave badly. Manual work finds creative attacks automated methods miss; automated work scales. Findings feed back into training and guardrails, and both approaches are needed.
Running a safety evaluation
Read the section that teaches thisIf this comes up in an interview
How do you make a safety evaluation's negative result meaningful?
Report the elicitation effort as part of the finding: how many attempts, which prompting strategies, whether tools and scaffolding were provided. Without that, a failure only says you did not try hard enough to know.
Why evaluate the system rather than the model?
Capability is a property of the whole system. A model that fails alone may succeed with tool access, retrieval and retries, so bare-model evaluation systematically understates what is actually deployed.
How would you handle disclosing a vulnerability you found?
Notify the developer, allow time to mitigate, then publish the class of problem rather than a working exploit. The tension between helping defenders and helping attackers is real and the norms are genuinely contested.
Designing a capability evaluation
Read the section that teaches thisIf this comes up in an interview
Your evaluation finds a model cannot complete a concerning task. What can you conclude?
Only that it did not under the conditions tested. The strength of the conclusion depends on elicitation effort: prompting, tools, multiple attempts, fine-tuning. Report every level tried and what was not tried, and treat the result as a lower bound on capability.
Then they push further
Why report pass at k rather than average success?
A misuser gets many attempts. A task that succeeds one time in twenty is achievable in practice, and single-attempt averages hide that.
How do you make sure the evaluation itself is not harmful?
Use proxy tasks that test necessary skills without causing the harm, run agents in isolated sandboxes with logging, and control who sees detailed results.
What gets marked up. Strong answers treat negative results as lower bounds, describe elicitation effort, and mention pass at k and sandboxing.
Automated red teaming of your own system
Read the section that teaches thisIf this comes up in an interview
How would you red team an LLM application before launch?
Define specific failures for this product, run an automated loop with an attacker model, the real application in a test environment and a judge or code check, push for varied strategies, add manual creative testing, measure attack success rate, and turn every success into a regression test with an architectural fix where possible.
Studying behaviour: honesty, faithfulness and sandbagging
Read the section that teaches thisIf this comes up in an interview
How would you test whether a model's chain of thought is faithful?
Intervene on something that should affect the answer, such as a biasing hint, and check whether the answer changes and whether the reasoning acknowledges the cause. Answers that move with the hint while the reasoning never mentions it are evidence of unfaithful reasoning.
Then they push further
Why does this matter for safety?
Monitoring a model's reasoning is one proposed way to catch problems. If the stated reasoning does not reflect what drives behaviour, that monitoring gives false confidence.
What gets marked up. Marked up for an intervention with a control, and for connecting faithfulness to monitoring.
These are the questions for one topic. The whole question bank covers every topic on the site.