Adversarial Examples & Robustness

Why It Matters for Modern Models

Adversarial examples show that a model can be accurate on ordinary inputs while relying on features that tiny, targeted changes can flip; Ilyas et al. (2019) argue that these are real but non-robust features of the data.
The same idea carries over to language models, where optimized adversarial suffixes can make aligned chat models produce content they would normally refuse (Zou et al., 2023).
Adversarial training with PGD has held up against later, stronger attacks better than most proposed defenses, at a cost in accuracy on clean inputs (Madry et al., 2018; Athalye et al., 2018).

Questions to Think About

Why are even linear models vulnerable, as Goodfellow et al. (2015) argued: how does a tiny change in every coordinate add up to a large change in a dot product?
On CIFAR-10, Madry et al. (2018) report 95.2% clean accuracy for a wide ResNet with standard training and 87.3% with PGD adversarial training; is some of this trade-off unavoidable, as Tsipras et al. (2019) argue?

Core Math

Key Equation

x_{\text{adv}} = x + \epsilon \cdot \text{sign}(\nabla_x L)

An adversarial example is an input $x + \delta$ whose perturbation $\delta$ is too small to matter to a person (for images, often $\|\delta\|_\infty \le 8/255$ ) but changes the model's prediction (Szegedy et al., 2014).

FGSM (fast gradient sign method) takes one step in the direction that increases the loss fastest within an $\ell_\infty$ budget $\epsilon$ (Goodfellow et al., 2015):

x_{\text{adv}} = x + \epsilon \cdot \text{sign}(\nabla_x L(\theta, x, y))

PGD (projected gradient descent) takes several smaller steps and projects back onto the allowed ball $\mathcal{B}_\epsilon(x)$ after each one (Madry et al., 2018):