Label Smoothing & Soft Targets

Label smoothing replaces a one-hot target with a mixture of that target and a uniform distribution over classes, with weight α on the uniform part. It was introduced as a regularizer for the Inception-v3 image classifier (Szegedy et al., 2016), and the original Transformer used it with α = 0.1 for translation, where it made perplexity worse but improved accuracy and BLEU score (Vaswani et al., 2017).

Concept 60 of 100OptimizationStudy phase 3

Why It Matters for Modern Models

Label smoothing replaces a one-hot target with a mixture of that target and a uniform distribution over classes, with weight α on the uniform part. It was introduced as a regularizer for the Inception-v3 image classifier (Szegedy et al., 2016), and the original Transformer used it with α = 0.1 for translation, where it made perplexity worse but improved accuracy and BLEU score (Vaswani et al., 2017).
It usually improves calibration: a smoothed model's predicted confidence tracks its accuracy more closely (Müller et al., 2019).
It interacts with knowledge distillation, which also trains on soft targets: Müller et al. (2019) found that a teacher trained with label smoothing makes a much less effective teacher for distillation.

Questions to Think About

Why does label smoothing pull the penultimate-layer representations of each class into tight clusters, and why does that erase information about which examples of different classes resemble each other (Müller et al., 2019)?
When does label smoothing improve calibration in a way that a single temperature fitted after training could not, and when are the two interchangeable?

Core Math

Key Equation

y_{\text{smooth}} = (1 - \alpha)\, y + \frac{\alpha}{K}

Instead of a one-hot target such as $y = [0, 0, 1, 0]$ , train on a smoothed target that moves a fraction $\alpha$ of the probability to the uniform distribution $u$ over the $K$ classes:

y_{\text{smooth}} = (1 - \alpha)\, y + \frac{\alpha}{K}

For $\alpha = 0.1$ and $K = 4$ classes this gives $[0.025, 0.025, 0.925, 0.025]$ .

Cross-entropy is linear in the target, so the smoothed loss splits into two terms:

\mathcal{L}_{\text{smooth}} = (1-\alpha)\, \mathcal{L}_{\text{CE}}(p, y) + \alpha\, \mathcal{L}_{\text{CE}}(p, u)

With a one-hot target the loss keeps falling as the correct logit grows, so the logits can grow without bound. With a smoothed target the loss is smallest when $p = y_{\text{smooth}}$ , which for a softmax means a finite gap between the correct logit and each of the others: