Dropout regularizes a network by training it with random units removed, so that no unit can rely on particular other units being present; it reduced test error on vision, speech and text benchmarks (Hinton et al., 2012; Srivastava et al., 2014).
Concept 52 of 100DropoutOptimization · Study phase 10: Mathematical foundations & information geometry
Question
With inverted dropout, an activation h becomes a random variable with mean h and variance h²p/(1−p); how does this noise, which grows with the size of the activation, differ from adding noise of a fixed size?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Dropout regularizes a network by training it with random units removed, so that no unit can rely on particular other units being present; it reduced test error on vision, speech and text benchmarks (Hinton et al., 2012; Srivastava et al., 2014).
The same idea applies to larger pieces of a network: stochastic depth drops whole residual blocks at random during training (Huang et al., 2016).
Large language models are often pretrained without it: PaLM used no dropout during pretraining and a rate of 0.1 for most fine-tuning (Chowdhery et al., 2023).
Keeping dropout switched on at test time and averaging several stochastic forward passes gives an estimate of the model's uncertainty (Gal & Ghahramani, 2016).
Questions to Think About
Why is it reasonable to replace the average over 2ⁿ weight-sharing subnetworks with one network whose activations are scaled, and when is that replacement exact?
Why does dropout matter less when a model sees each training example only about once, as in much large-scale language-model pretraining?
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
h~=h⊙m,m∼Bernoulli(1−p)
Dropout sets each unit's activation to zero with probability p during training, independently for every unit and every example:
h~=h⊙m,mi∼Bernoulli(1−p)
so each unit is kept with probability 1−p. At test time every unit is kept, and the activations are scaled by the keep probability so that their expected value matches training:
htest=(1−p)h
Most implementations use inverted dropout instead, which rescales during training and leaves the network unchanged at test time:
h~=1−ph⊙m
A network with n units that can be dropped contains 2n subnetworks that share weights, and each training step trains one of them. Scaling at test time approximates averaging the predictions of all of them. For models without nonlinear hidden units, such as softmax regression with dropout on its inputs, this weight-scaling rule gives exactly the normalized geometric mean of the subnetworks' predictions; for deep nonlinear networks it is an approximation that works well in practice (Goodfellow et al., 2016, §7.12).
Key Papers and Books
Dropout: A Simple Way to Prevent Neural Networks from Overfitting