Batch Normalization

Batch normalization subtracts each feature's mean and divides by its standard deviation, both computed over the current mini-batch during training, then applies a learned scale and shift. It made deep convolutional networks much faster to train: an Inception image classifier reached the same accuracy in 14 times fewer training steps (Ioffe & Szegedy, 2015), and ResNets apply it after every convolution (He et al., 2016).

Concept 61 of 100Core TrainingStudy phase 3

Why It Matters for Modern Models

Batch normalization subtracts each feature's mean and divides by its standard deviation, both computed over the current mini-batch during training, then applies a learned scale and shift. It made deep convolutional networks much faster to train: an Inception image classifier reached the same accuracy in 14 times fewer training steps (Ioffe & Szegedy, 2015), and ResNets apply it after every convolution (He et al., 2016).
It allows much higher learning rates and makes training less sensitive to initialization (Ioffe & Szegedy, 2015).
The switch from batch statistics during training to running averages at test time is a common source of bugs, for example when a model is evaluated in training mode or when the test data differ from the training data.

Questions to Think About

Because each example is normalized with its batch-mates' statistics, its output and gradient depend on which other examples share the batch; what problems can that cause?
Why does the error of batch-normalized networks rise quickly as the batch gets small (Wu & He, 2018), and why do transformers use layer normalization, which needs no batch statistics, instead?

Interactive Demo

Core Math

If you prefer intuition first, try the demo above, then come back for the math.

Key Equation

\hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}

Batch normalization normalizes each feature with the mean and variance over the $m$ examples in the current mini-batch. For a convolutional channel, the statistics also include all spatial positions, so $m$ below counts batch elements times spatial positions:

\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}

where $\mu_B = \frac{1}{m}\sum_{i=1}^m x_i$ and $\sigma_B^2 = \frac{1}{m}\sum_{i=1}^m (x_i - \mu_B)^2$ .

Learned parameters then scale and shift the result, so the layer can still produce whatever mean and variance the next layer needs:

y_i = \gamma\, \hat{x}_i + \beta

At inference there may be no batch, so the layer uses running averages of $\mu_B$ and $\sigma_B^2$ collected during training. The network therefore computes a slightly different function at training and test time.

Key Papers and Books