Weight initialization sets the parameters before training, including the scale of the random starting weights. A poor choice can make activations or gradients vanish or explode across layers, preventing the network from training.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Weight initialization sets the parameters before training, including the scale of the random starting weights. A poor choice can make activations or gradients vanish or explode across layers, preventing the network from training.
µP makes hyperparameters transferable across model widths: Yang et al. (2021) tuned a 40-million-parameter model and, with the transferred settings, outperformed the published results of the 6.7-billion-parameter GPT-3.
Initialization scale decides how very wide networks train: under the standard and NTK parametrizations they behave like kernel methods and stop learning features, while µP keeps feature learning at every width (Yang & Hu, 2021).
Questions to Think About
ReLU sets about half of its inputs to zero; why does He initialization double the variance to compensate?
Why must the learning rate of a hidden layer shrink as the layer gets wider for its updates to stay the same size, as µP prescribes for Adam?
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
W∼N(0,2/nin)
Initialization sets the scale of the random starting weights so that signals neither shrink nor blow up as they pass through many layers.
Xavier/Glorot (for tanh and similar activations) balances the variance of signals in the forward and backward passes (Glorot & Bengio, 2010):
He/Kaiming (for ReLU) doubles the variance to make up for the half of the inputs that ReLU sets to zero (He et al., 2015):
W∼N(0,nin2)
µP (maximal update parametrization) also scales the learning rate with width, so that each layer's updates stay the same size as the model grows. For hidden weight matrices trained with Adam:
W∼N(0,1/nin),ηW=ηbase/nin
up to constant factors, with separate rules for the input and output layers (Yang et al., 2021). Under µP, the best hyperparameters found on a narrow model carry over to a wide one.
Key Papers and Books
Understanding the difficulty of training deep feedforward neural networks