Asked about Measuring and reducing what could go wrong
What is red teaming?
An answer that works
Deliberately trying to make a model behave badly. Manual work finds creative attacks automated methods miss; automated work scales. Findings feed back into training and guardrails, and both approaches are needed.
Reading an answer is not being able to give one. The section that teaches this is Measuring and reducing what could go wrong, 5 hours of Safety & evaluation research.
Learn this properlyAlso asked about Safety & evaluation research
How do you evaluate whether a model has a dangerous capability?Why is a negative safety result weak evidence?How do you make a safety evaluation's negative result meaningful?Why evaluate the system rather than the model?How would you handle disclosing a vulnerability you found?Your evaluation finds a model cannot complete a concerning task. What can you conclude?How would you red team an LLM application before launch?How would you test whether a model's chain of thought is faithful?