Lesson · about 8 minutes · a small model, computed exactly

When an average hides a failing group

A classifier can be right on 94.5% of photos and wrong on most of a rare group. Move the training mix and watch it happen; predict what balancing does if you like, check it, and leave with a question for real data.

Depth

Builds on cross-entropy (the loss being minimised), the dot product (the boundary is where w·x + b = 0) and representations (what a model’s “features” are). Your place in this lesson is kept in this browser while you follow those links.

The model · training mix 95% · shape clarity 1.00 · trained as collected

shape: landbird ← → waterbirdbackground ↑ water · ↓ land
Across: the bird’s shape. Up: the background.landbirdwaterbirdon wateron landshaded: the readout says “waterbird”
landbirds on land97.4%
landbirds on water39.2%
waterbirds on land39.2%
waterbirds on water97.4%

Average, weighted like training: 94.5%Worst group: 39.2%

Accuracy on new photos of each group, exact for this model.

The boundary tilts 53° from vertical (0° ignores the background; 90° uses only it). In what the fit sees, the background goes with the bird 95% of the time, and it is cleaner than the shape, so fitting well rewards it, and the rare groups, where it points the wrong way, pay.

  1. A shortcut that pays in training

    Each dot is a training photo: its shape (across) says which bird it is, its background (up) says land or water. The shape is noisy; the background is clean. In training, most waterbirds are on water and most landbirds on land. The readout is a straight line through this plane that labels each photo landbird or waterbird; it is fitted by minimising cross-entropy plus a small penalty on its weights.

    95%

    Computed for this modelThe readout that best fits this training data is right 94.5% of the time on photos mixed like training, and 39.2% on waterbirds on land. Its boundary tilts 53° from vertical: the more it tilts, the more it uses the background.

Sources. The setting follows the Waterbirds benchmark, in which photos of birds are pasted onto land or water backgrounds and 95% of training birds appear on their usual background: Sagawa et al. (2020). Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. ICLR. arxiv.org/abs/1911.08731. Refitting only the last layer on group-balanced data, which can lift the worst group when the frozen features still carry the bird, is studied in Kirichenko, Izmailov & Wilson (2023). Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. ICLR. arxiv.org/abs/2204.02937. This lesson replaces each photo with two numbers, so its accuracies can be computed exactly.