Foundation Lab

Layer Normalization & RMSNorm

LayerNorm is in every transformer—it stabilizes training by controlling activation scales

Concept 54 of 100Core TrainingPhase 2

#54LayerNormCore Training

key equation\text{LN}(x) = \gamma \odot \frac{x - \mu}{\sigma}

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 54 of 100LayerNormCore Training / Phase 2: Architecture fundamentals

Current question

LayerNorm makes networks robust to scale: you can multiply weights by constant without changing output

\text{LN}(x) = \gamma \odot \frac{x - \mu}{\sigma}

PredictionCommit before the demo.

Ask what should change when the equation is manipulated, then let the visualization test that expectation.

EvidenceCompare local witness and source.

Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue in the newer notebook.

This atlas page keeps the working demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.

LayerNorm is in every transformer—it stabilizes training by controlling activation scales
Pre-norm vs post-norm placement affects gradient flow and training stability
RMSNorm saves compute (no mean) with similar quality—used in modern efficient LLMs

What is still poorly explained in textbooks and papers:

LayerNorm makes networks robust to scale: you can multiply weights by constant without changing output
The learned γ, β parameters let the network "undo" normalization where needed
Pre-norm (normalize before attention/MLP) is more stable for deep networks

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

\text{LN}(x) = \gamma \odot \frac{x - \mu}{\sigma}

LayerNorm normalizes across features for each example:

\text{LN}(x) = \gamma \odot \frac{x - \mu}{\sigma + \epsilon} + \beta

where $\mu = \frac{1}{d}\sum_i x_i$ , $\sigma = \sqrt{\frac{1}{d}\sum_i(x_i - \mu)^2}$ .

RMSNorm (used in LLaMA, etc.) skips mean centering:

\text{RMSNorm}(x) = \gamma \odot \frac{x}{\text{RMS}(x) + \epsilon}, \quad \text{RMS}(x) = \sqrt{\frac{1}{d}\sum_i x_i^2}

Ba, Kiros, Hinton2016arXiv

Zhang & Sennrich2019NeurIPS

Choose the next question to carry this object forward.