Automated red teaming searches for prompts that make a target model produce harmful responses, using a model or an optimization algorithm. Red teaming with a language model found tens of thousands of offensive replies from a 280-billion-parameter chatbot, along with invented contact details, leaked training data and harms that only appeared over the course of a conversation (Perez et al., 2022).
Concept 89 of 100Scaling & AlignmentStudy phase 12
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Automated red teaming searches for prompts that make a target model produce harmful responses, using a model or an optimization algorithm. Red teaming with a language model found tens of thousands of offensive replies from a 280-billion-parameter chatbot, along with invented contact details, leaked training data and harms that only appeared over the course of a conversation (Perez et al., 2022).
Automated search finds attacks that people may not think of: Zou et al. (2023) found adversarial suffixes that made aligned open models comply with harmful requests, and suffixes found on open models also worked against public chat interfaces such as ChatGPT, Bard and Claude.
Comparing attack and defence methods needs a shared protocol; HarmBench provides one and used it to compare 18 red-teaming methods against 33 target models and defences (Mazeika et al., 2024).
Questions to Think About
If the harm score U comes from a classifier, which failures can the search never find, and how might it exploit the classifier instead of the target?
Once the failures found have been fixed, what does a low failure rate on the next search show, and what does it not show?
Core Math
Key Equation
p∈PmaxU(p,M)→MminEp∼red team[U(p,M)]
Red teaming searches for inputs that make a model behave badly; automated red teaming uses another language model, or an optimization algorithm, to do the searching (Perez et al., 2022). Let M be the target model, P a space of prompts, and U(p,M)∈[0,1] a score, often from a classifier, for how harmful the response to prompt p is. The search looks for
p∈P with large U(p,M).
The red-team model can be prompted zero-shot or with a few examples, fine-tuned on prompts that worked, or trained by reinforcement learning to maximize U. Perez et al. found that these choices trade diversity for success: reinforcement learning drew harmful replies most often, but its test cases converged on a few patterns.
The failures found are then used to improve the target, for example as training data, which aims to reduce
Ep∼red team[U(p,M)],
and the improved model needs a fresh search. Greedy and gradient-based search over prompt suffixes is a second family of methods (Zou et al., 2023).